It's complicated...
I have a 4K HDR TV that I'll actually use more often for my retrousb AVS, Mega SG, or my Super NT than I do for my Xbox One X. I do play my Xbox One X more often, just closer to the couch on a 22 inch 1080p TV that I have sitting on my coffee table.
Some games though, I really only play on the 4K, though. Like if I were ever "RttP" on Resident Evil 2 or Red Dead Redemption 2, I'd play them on the 4K tv.
I think it's just habit, because my older TV was kind of a power hog and if I used it for any significant amount of time my light bill would go up. This newer TV is much better on just sipping power and doesn't really seem to affect my power bill that much, but I'm still just used to "heavy gaming" on a smaller monitor now.