Really, looking over the past, it feels more like "How did Sega not mess up Sonic sooner?"
The initial conception of Sonic was already kinda bad with Sonic being in a band and having a human girlfriend named Madona. Sega of America got rid of the band and the girlfriend and came up with their own official Sonic bible where Sonic lived in America and was a normal talking hedgehog that became blue after running too fast in an experiment with his friend Dr Kintobor who became Robotnik after accidentally absorbing the evil energy of the chaos emeralds and having his DNA combined with a rotten egg. Thankfully the Classic games ignored both of those canons and just didn't focus on story much at all.
One of the most important factors in why the early games turned out so well is Hirokazu Yasuhara, who was the lead game designer on Sonic 1, Sonic 2, and Sonic 3 & Knuckles. He was in charge of the level design, gameplay conceptualization, and general world art. His absence in Sonic CD is very noticeable.
Sonic 2 was kind of a miracle. Mark Cerny started a new studio in the US called Sega Technical Institute along with Hirokazu Yasuhara and Yuji Naka. Naka had quit Sega after Sonic 1, but Sega bribed him to come back by buying him a Ferrari. Cerny proposed making a sequel to Sonic 2 and initially Sega wouldn't approve the project until they saw the success of Sonic 1. STI had a mixed American and Japanese team. Yasuhara was reportedly easy to work with but Naka really hated the cultural/language barrier.
Following Sonic 3 they decided to make a 3rd game. The game was still made in America at STI, but Naka demanded that the staff be Japanese-only. Originally they wanted to make a 3D game, but thankfully due to time constraints they chose to make a 2D game.
Following Sonic 3, STI continued to try to make new Sonic games with help from Yasuhara, but development was troubled due to the Saturn being a mess and Yuji Naka refusing to let them use his 3D engine. They had multiple demos for Sonic X-Treme, but the lead designer of their best demo was late to the meeting with Sega of Japan's president. Sega's president was furious over the worse demo he saw and stormed back to Japan before he could see the better demo.
Yasuhara left Sega and went to work at Naughty Dog.
Naka was put in charge of Sonic Team and gave creative control to Takashi Iizuka after liking his Sonic RPG concept. And it was all downhill from there.
Basically it could be boiled down to Yuji Naka's ego and the incompetence of Sega's management.
Check out this interview with Naka and how he talks about Hirokazu Yasuhara:
Interviewer: What do you think of the way the Sonic games have evolved; was it a problem when one of your team members left to work with Sony?
Yuji Naka: Yeah, one of the guys from our team went to work on Jak and Daxter! As for the original titles, I was involved from the beginning, the creation of the game. The character was born in a kind of stream of creating, so I'm involved from the very beginning of the character. I gave the game direction, and I was the main programmer also. So I was involved with every aspect of the original Sonics. Some of the details, like making a map, quite straight-forward stuff, was done by the guy who's working on Jak and Daxter right now. He was involved until Sonic 3, and after that for eight years he didn't do anything in Sega, so he was quite useless in Sega. We really didn't need him. He was really doing nothing with Sonic.
I think that says a lot about what happened. The lead game designer was just 'some guy who was useless in Sega.' Level design is 'straightforward unimportant stuff!'
