Without regulations, there will always be some natural (or artificially induced) imbalance of resources. Maybe it happens for religious reasons (e.g. a self-proclaimed prophet). The greedy always find a way. Of course, the state of the other aspects of said society would also need to be considered. But those details are important, as where we are today arose from much smaller communities.How would you imagine a "ruling family" even establish itself though? Genuinely curious. Do you think that this family of say racists would not be recognized by other neighborhoods and not be placed in rehabilitation settings or just tossed from society? To assume that racism would somehow overcome the will of hundreds or thousands in a neighborhood or surrounding neighborhoods would probably fall into the belief that racism is inherent to humans, rather than socially taught. Since these neighborhood protectors live within the neighborhood, they are themselves subject to the neighborhood itself. Someone could be removed from the position or removed from the neighborhood entirely.
I think people succumb quite easily to tribalism without the education and training to effectively manage those thoughts. Depending on the community, I believe that kind of societal conditioning and programming could be all the strong. If the community becomes a white supremacist pseudostate, the neighborhood protectors would most likely fall along with the majority or be ousted themselves. There's a level of artificial selection and maintenance needed (by people with a common ideology) to prevent something like that from happening given how commonplace things like racism, homophobia, etc. are. I guess it's hard to avoid the circular loop of, "who watches the watchmen?"