Jim has a little bit of that....I never know what the best term to use to describe it is....but that "YEAH FUCK IT ALL STICK IT TO EM' LOL WE'RE SMARTER" vibe going for them. That stupid Republican, especially these days, mindset that people may latch onto. Best thing I can compare it to is how some audiences view South Park. They see an episode that might rip on something they agree with ripping on, but in the end the lesson is not to rip on that thing. However, this type of audience doesn't care about the lesson, they care about the rip, and that's the "lesson" they end up taking away from the episode.
Most of that audience is not well-versed in nuance, sensibility, or let's say intelligence in general, and they typically align themselves with intense conservative values. I imagine most of the people leaving Jim's channel right now are among that type of audience. That I'm not too surprised they've picked up a few along the way due to the type of content they put out.
I'm not a big fan of Jim, but they're fine and it's a shame that their self acknowledgement is being "rewarded" in this way. However I'm sure soon enough it will level out, and they'll be better off once it does.