It was a racist joke. I am not saying it wasn't.
You're saying that people can make racist jokes without being racist. News flash: that isn't the case.
If you make a racist joke, you're a racist.
If you consistently provide a far reaching platform for people with racist views, you are at the very least an enabler of racist views.
If you do both those things (like Rogan) you're definitely Ok with racism.
You really liking this dudes content doesn't mean that he's not racist. It just means that you're Ok with his brand of racism because he entertains you.
If you're uncomfortable with that assertion, that doesn't make it Ok for you to go to bat for him and you should absolutely expect to be called out for it because, at the end of the day, you're saying you're fine with racism so long as you can have your lols