Look up the synonyms for residue and note how many of those synonyms were words that were/are used to disparage people of color. There's a reason people took offense to what he said, and it's not that we're all just irrational complainers. He went from talking about white liberals leaving, to "residue" being more diverse. It was a direct contrast between white and non-white, and as such that term 'residue' being used to describe the other side of that contrast is completely inappropriate.
More importantly, more than just the word, conceptually he's off-base. The level of support Bernie has among people of color is not just from supporters from 2016. Many of them are new and are not "left overs". That his support from people of color is on a similar LEVEL to 2016 does not mean that that support is 100% FROM 2016 (i.e he lost some of those supporters and gained some new ones). You don't just call these people residue. On top of the fact that he could have just referred to the voters as voters (like he did with 'white liberal'), his statement also discounts the very real concerns that drove these people to Bernie's campaign and effectively handwaves their support as meaningless and insignificant. It's not a good look.
Uhh, let me help you out here (brackets are mine):
He just flat out called these people residue, as evidenced by his contrast with white liberals. There was no need for him to do that.
