Well the clock speed on the PS5 makes the specs look closer, but the raw shader counts (52 CUs vs 36) might be something to consider. When looking at a RX 5700 XT vs 5700 (40 CU vs 36; ~11% shader increase), the 5700 XT usually comes out 10-15% ahead, although I found a video of the 5700 @ 2.15 Ghz vs the 5700XT @ 1.75 Ghz (similar to clock speed gap of PS5 and Xbox):
It shows that the heavily OC'd 5700 can roughly make up the performance difference and perform similar to the 40 CU 5700 XT. But the Xbox has 30% more shaders than that, and seemingly higher bandwidth as well, so it seems like a 30%+ difference in performance could be quite possible.
Overclocking seems to only usually increase performance 10-15% at best most of the time, and a 44% shader difference (with a bandwidth advantage) would be a tall order to overcome, imo.
Edit: On topic, I don't think the SSD will generally matter too much for games, but it could make some aspects of the UI / booting smoother than the Xbox, imo.