At the end of the day, if you see and treat the people as humans and not just colors, I see no reason to feel conflicted. If your intentions are pure and there is no malice or ulterior motive in seeking our presence, stop feeling uncomfortable about it.
My best friend, who I met in culinary school, was a white kid who came from a town with literally one black family. It just turned out that he liked everything I did, we held the same values, had similar issues with our parental upbringing, and the way he talked about spacing out and creating on the guitar was exactly how I entered the zone when I rapped. He was also one to judge a person simply on who they are, he just never looked at me as black first. He ended up gravitating toward me and then by default just hung around myself and the black guys in the dorms. We honestly never thought anything of it, and he felt comfortable because he was genuinely my friend.
As far as dating, he was really attracted to black women but was afraid to approach them lol. Couldn't blame the kid.
So
JonCha I get why you are questioning things, but I feel like you have a lot in common with my old friend. You just shouldn't sweat your wavelength aligning with black people, if your heart is pure just see it as incidental to your life's path. Ignore all external factors and check your heart though. If you're good there theen fuck everything and everyone who has something to say.