Are you using the HDMI cable that came with the X? As you should be. Your cables must be certified for HDMI High Speed or HDMI Premium.
Read more at https://news.xbox.com/en-us/2017/11/22/best-4k-hdr-experience-xbox/#hzCROEjY626e8FKO.99
I'm very sure that I'm using the 4K cable that came with the One X. I haven't replaced it at all. Still has issues with flickering and screen sometimes not even showing anything at all.This was exactly my thought too. I have a somewhat complex case where everything is going through a 4K switch box to the TV, then TV ARC back to an AV receiver. The only way to get everything to work without flickering, random black screens, and full HDR was to ensure all HDMI cables in the chain were "certified 4K"cables. I can personally vouch for the excellence of the monoprice brand cables.
Give it a try and let us know how it goes.
Probably doesn't mean much but I have a PS4 Pro with the highest settings available on that one but I have had no issues on that one at all.