I mean they were also pushing boundaries and inventing the open world genre back in the ps2 era and that didn't stop them from putting out a lot of other great content.What happened is, Rockstar is constantly pushing the envelope when it comes to immersive 3D game design, and in order to continue to push that edge with each new hardware iteration, it takes more and more manpower, time, and money.
They could make more games if they lowered the fidelity of their game designs, but that's not going to happen because it is antithetical toward what they try to do.
Its the gta online money that has them hooked. All of their manpower is going to that rather than making other games (aside from rdr2 of course)