My hunch here on this is that XSX and XSS have a more complete programmable Front end with mesh shaders (vs. the older primitive shader type), the Hardware VRS, and the technical hw required for the next Version of tiled resources, which is called Sampler Feedback. And I think, without saying positviely for certain, that that is why MS was so keen to make those features so prominent in their pre-release Material in the Xbox Series X. It was their hardware Feature advantage, they knew it, so they advertised it - just like how they advertised stable clocks as a thing even before Sony described the dynamic clocks to the public.
Will these advantages of a complete RDNA2 vs the PS5 GPU half way point matter? Probably.
By a lot? Meh. Let us wait for the games. And for the games over time. I think the raw flop and bandwidth will matter the most at first though as the other Features, should they prove limited to just XSX and XSS, require dedicated peogramming time to take advantage of. And multiplatform games do not always spend the resources to do that even. Unlike a faster GPU, which is just faster.