Depends. I clean the bottoms of my shoes, so dirt isn't really a concern for me. If I'm dressed or plan on leaving soon, I'll wear shoes. If not, I'm wearing socks.
I hate "house shoes", slippers, and slides, so I'm never wearing those.
That was just my experience, but I lived in CT, NM, AZ, CA, GA, NY and VA when I was in the States and the only place I knew where you had to take shoes off at a rich friend's house because they had cream carpets. I also lived abroad quite a bit as a kid and it wasn't until Japan that I ever encountered the shoes off as a rule. Didn't see what other US people said, but have they been similar to your experience or mine?