Graphical differences might be minute, with the PS5 and Series X both having different strengths leading to certain games running better on one or the other.
But the SSD is indisputable, while the Series X will be fast as hell, the PS5's speeds wont be just "6v8 seconds", it'll be pretty considerable and to be honest I'm sure the user experience might have some advantages on Sonys console. Not saying the Series X won't feel fast as well, but, the speed is what the speed is.
The controllers are another interesting aspect. Now normally it can be chalked up to, "oh well it'll drop support like HD rumble" or that it doesn't seem that important but some devs and insiders say it adds to the experience quite well. I actually personally think MS will update their controller in the next 1-2 years to include these features in their standard pack in controllers, as well as their next Elite version.