Not just non-white Muslims, but Muslims in general, and brown people who could be mistaken for Muslim by ignorant racists. Islamophobia can sometimes just be another form of racism against Middle Easterners and South Asians (and people who look like them). And hatred towards Islam is not always rooted in disagreements with the teachings, but rather because Muslims are an 'other' and a threat to their 'culture'. You'll also find plenty of Islamophobes who either follow or are fine with the other Abrahamic faiths, despite their holy scriptures having equally problematic stuff.
But yes, that's where the pushback against the "Islam isn't a race" argument comes from. Because Islamophobes use it as an excuse to mask their hatred against Muslims specifically. Because the implication is usually "Islam isn't a race, therefore I cannot be a racist when I decide to incite hatred against Muslims, I'm just attacking the religion, you see." There's a big difference between thoughtful and responsible critique and fear-mongering, generalising etc. The latter hurts innocent Muslims who are a persecuted minority in many parts of the world, despite Islam being the second largest religion overall.
I think a big issue that I see recurring in this thread is that people look at this from a black and white perspective, or look at things literally/simplistically (like the argument you made in your initial post). While technically true that Islam isn't a religion, it's ignoring the context in which this phrase is commonly used, and why people take issue with it. On the other hand, I don't think the solution is to never criticise Islam as a religion either, as there does need to be pressure for it to modernise and become more progressive, even in secular nations. I have personal experience with this, and see what this lack of progress does in my own community.
Which kind of goes back to what I mentioned at the end of my first paragraph. In the context of countries where Muslims are a minority, I feel the most responsible way to critique it is speaking about all of the religions that have these issues, including Christianity. Singling out Islam just bears the risk of making Muslims out to be this evil and barbaric group of people, even though there are plenty of Muslims who follow a progressive and inclusive interpretation of Islam, and there are many non-Muslim majority countries that have similar issues.
TLDR: There's a lot to criticise in the Quran and prophetic sayings, but there are responsible ways to do that without endangering marginalised Muslims.
