The main problem of Steam OS was going out too soon for the work Valve was doing with Linux. Launching a platform were a majority of games was not going to work was not really a great idea, launching once Proton had been a project that is pretty good for most AAA games (and hopefully has better partnerships with anti cheat services to actually work through it in the future), it wouldnt have been as bad.
It would have also benefitted a lot of better Steam Link functionality (which was also announced at the same time), such as the one we have now..
Of course, it was necessary for both of those initiatives to take shape, and that there was no "better" time for them to launch that, but still, room to learn.
There was also the problem of it lacking a clearly unifying position and being quite pricey for what they were. It would be interesting to see if they do a SteamOS 2.0 and what are the main takes.