So, here's a few things you're missing. And I'll say this as someone who worked in a AAA game studio.
Correlation!=causation. Want to know why those "innovative" games didn't sell? The same reason why Evil Dead did poorly at the box office 30ish years ago, but is considered a cult classic today. Innovation doesn't mean it "has to be great" now, but it does mean that developers have to move out of the box and try new things for good or bad, and work on the feedback of it. When gamers say "we want something innovative" it means they're tired of the same rehashed mechanics over, and over, and over, and over, and over. Did I mention over again? And one effect that you miss, is the uncanny valley which will actually kill sales faster than a 10 year writing your plot, or having two disconnected writing teams.
With regards to the sales, however you're also forgetting this, we were in the "consoles are going to rule everything" moment. That was of course before they began their screaming decline about 3 years ago, and PC gaming took off at a rate that hadn't been seen since the mid-90's. This has of course been expanded by the current generation of consoles which are watered down PC's from roughly 3-5 years ago. The second thing you've forgotten of course is the increase in disposable income from even a decade ago or even 20 years ago, even with the massive economic slowdown, people can buy more "fun time toys."