Been married for 7 years. I also have body issues and anxiety. My spouse and I call out hot people all the time together. Sounds like his issues run pretty deep.

As far as you feeling bad, you made an oops and you've apologized I assume. He should get over it and not hold it against you if he really likes you instead of using it as a crutch.

I hope things get better between you two. Don't beat yourself up over it.