You say they're viable, but most people don't want to make their own engine to make a game. Most people wanna make a game, not a game engine. Furthermore, while I honestly believe that the FOSS community could easily match commercial projects, doing so requires people to acknowledge the software's shortcomings, talk to the professional users about what kinda UI they want, and so on. That means you can't throw lines at them like "user inexperience" or "just learn a new program". They're professionals, having to learn a new program cuts into their income. Learning new programs means they aren't making things, which means they aren't making money.

I know I'm kinda rambling a bit and may not be super coherent (if so, I apologize, I'm running on low sleep), but I'm not trying to be a doomer or say "that's impossible"; I think what I am trying to say is that I don't think the FOSS community currently has all the necessary software features or ease of access for professionals.