Simple reason why I don't like to be around kids- anxiety.
Being around kids sends my anxiety through the roof- I'm always tense, on edge, and cannot relax around them. I'm always worried about doing and saying the right thing around kids. They are big responsibility, and I fear failing them.
It is not fun being endlessly mock by society for not "liking" kids.
This right here and being treated poorly as a child by either your parents or by other children seem to be the biggest reasons why some people don't immediately love all children.
And children can be great, but it does take a special kind of relationship to connect with them. The type of relationship that you would had to have been shown by your own guardians.