It's because they are supposed to be the "normal" ones. The whole "Silent Majority" thing they hide behind falls apart completely when you point out how abnormal everything they do is.
I think I might even take that a step further. They do want their beliefs to be the norm. But, the real thing they desire with all their heart is to be cool.
You see this is a major driving force in people like Pool, Shapiro, and Musk. They
HAVE to be fucking cool. Musk is Tony Stark. Shapiro is debate bro Einstein. Tim Pool is a skateboarder punk/rock star. The reality is, of course, that they are anything other than cool, and people seeing that kills them on a mental level that all the money in the world can't overcome. So, you see Pool buying up a skate park that shunned him, Musk spending all day every day dancing his heart out while looking back nervously at his fanbase for approval, and Shapiro literally just falling back on "well, I'm rich!" when he actually comes face to face with someone who knows how to shut down his bullshit in a discussion.
In their world view, you can be racist, sexist, transphobic, etc. and still be cool. You can't be fucking weird and be cool, though. So, calling them that actually hurts.