I think he used this excuse as a way to basically keep it neutral without coming off like a complete douche, but in reality, I think the real reason is that he just doesn't want these less than spectacular streamers leeching off his success. It amazes me that he even has to make an excuse as to why he won't play with certain streamers.
Like just look at this fucking tweet :
https://twitter.com/skipperr/status/1028390365114462208
"If you exclude yourself from playing video games with women because of "gossip" and "rumors," you're only propagating the idea that playing video games with women only starts drama." Clearly it doesn't matter if he plays with these streamers or not, because they are going to start drama no matter what. Ninja can't win this battle.