There are several reasons. Mostly though it was Blizzard's double whammy of
1) BroodWar becoming a major esport, leaving many RTS franchises to chase the same cash cow (that's how we got C&C 4 as well as Empire Earth 3, both of which buried their franchises).
2) WarCraft III both changing the paradigm by bringing in multiple RPG elements (a formula that spawned imitators such as Heroes of Anhililated Empires, and the still-active SpellForce series) AND spawning the MOBA. The latter not only took a lot of the sports playing people away from the game, but also eventually led Relic to abandon their own squad-based RTS sub-genre in favour of a far more tactics-based formula in Dawn of War 2. See also: proliferation of"real time tower defense games", also owing to that secret level in the Frozen Throne Blood Elves campaign;
Special mentions to Microsoft for managing Age of Empires into near oblivion with 3, followed by the online spin-off, before closing Ensemble alltogether, and Chris Taylor's caveman RTS and the shutdown of GasPoweredGames.
Another interesting aspect of the whole thing is Ubisoft eventual abandonment of Settlers and a general trend to "de-RTSify" city builders, differentiating them into a much more simulation-focused genre.
Total War was, and always will be, a refinement of the hybrid Lords of the Realm formula. It's flexible, offers decent range of depth, but ultimately it requires less tactical depth, because all it is is coordination of multiple formations on a small map. It's the closest we have to a "big budget strategy title", but it doesn't really have the same appeal as something like Age of Kings, Microsoft's " lightning in a bottle" mix of town-building and massive armies for the casual crowd. Thankfully. Microsoft appears to have realized this after seeing the ridiculous success of HD re-release (it sits firmly in the top 200 most played Steam games year after year, which is ridiculous for a 1999 RTS), so we have the Definitive Edition coming up, and that just may end up being Windows 10's first "killer app". I'm still wary of letting Relic do Age of Empires 4, because I don't want squad-based, small-scale warfare in my Age of Empires, and that's been Relic's MO ever since they stopped making Homeworld.
The genre isn't exactly dead- we got SpellForce 3, Iron Harvest, new Age of empires game in the works, as well as a whole number of independent projects. But it certainly isn't as big, because major publishers aren't particularly interested in the niche crowd that want to play casual games against computers, while making a bombastic single-player campaign experience and trying to compete with something like StarCraft II on production values is not really feasible for most indies.
