I wasn’t the reason the guy got kicked out of the friend group. No one else in the group knew about my experience. I simply told my partner I didn’t want to be alone with the guy.

You do you, but you asked for opinions and in mine, I think you’re being kind of silly to not even meet the gal. If you meet her and she’s seriously not your cup of tea and you can’t possibly be in her presence, we’ll then fine. But to me it’s like when my partner’s kid says “I hate that, won’t eat it” when he hasn’t even taken a taste and assumes that something green is an onion vs a granny smith apple. Yeah, they look alike but they aren’t the same.