Wow. You nailed the terrible person part.

This isn’t a “next team meeting” kind of thing. As far as I’m concerned, it would make a disgusting situation even worse both professionally and for your family’s sake if you failed to immediately remove yourself from that environment.

You cannot continue to interact with an affair partner (yes, I’m calling him that) on account of BOTH of those spheres of your life. Step down, get out of there, find a new job.

How to address this with your family is above my paygrade as far as I’m concerned, but the longer you continue to interact with that man, the worse handling this will become.