1. American ISPs have done a wonderful job of holding back fiber to the home and gouging customers for sub-par service. Game streaming is collateral damage.
2. Say it with me: LAY-TENN-SEE. Latency. Latency doesn't affect music or movies, or books, or basically any sort of media...except video games.
Latency was (and is) not a focus of ISPs because people don't buy latency, they buy bandwidth, measured in Mbps (or Gbps if you're lucky). When have you ever seen an ISP touting how few milliseconds it takes them to reach various Data centers? The answer is never.
Want game streaming to be a thing? cool! Make 95% of people's connection a fiber one with 2 or less milliseconds roundtrip to the nearest major datacenter (with microsoft, Google, and Sony servers). Then you have an extra 9-10 milliseconds to encode your video AND decode it client side if your game is 60FPS or less, this will give you a latency of 12 ms, let's say 16 ms once we factor in tiny lag spikes (taking your ping from 2 ms to 6 ms).
This is one frame of delay, and imperceptible to virtually everyone. Once you get into more than 1 frame of delay, game streaming will be worse than the local experience.
And god forbid you want to reach for 100Hz or 144Hz gaming, all of a sudden, the time you have to encode your video drops to 4 ms, and you have virtually no room for lag spikes. <--Currently, only VR requires those kinds of speed, but 144Hz gaming IS a thing on PC and has been for a long time, so eventually, there is no reason why it wouldn't percolate down to consoles.
tl;dr: fiber for all and 2ms ping to data centers, until then, game streaming will suck. Good luck!