>Does it? I was curious about the principle of it, I feel like I should choose who I spend my time with.

Absolutely.

If you refused to meet someone because they were a janitor or something, you'd be a huge unreasonable asshole.

There is no principle here, it needs context.