I didn't want to hurt his feelings and I didn't want to be rude.

This - stop this. Your comfort, sense of safety, and actual safety are far more important. Do not talk to anyone or do anything you don't want to do.

"No" is a complete sentence. No explanations needed.