Your boyfriend is a massive jerk for saying that to you. If he cared about you, why would he say that or feel that way?

Consider why you’re happy to remain in that situation.

Everyone deserves to be with someone who shares their interests and actually likes spending time with them. If your boyfriend is so relieved to not have to spend time with you, maybe it’s time to go.