He does not respect or accept you. If he is joking this is not funny, but call his bluff and break up with him. I’m a little suspicious about the “I was just joking about being racist” thing too.