The biblical commandment to not take the lord’s name in vain applies only to the people who follow the Bible’s teachings. It’s their commandment, no one else’s. And it’s not even close to a swear in colloquial terms. It’s just an expression of surprise, and an extremely common one at that.

The OP’s child is not capable of using god’s name in vain because he doesn’t even believe in the god in question. You have to believe in this god for it to be a problem, and even then it’s only a problem for you.

OP, I’d do two things. First, I’d talk to the teacher and tell her, politely, to back off. She’s said her piece, but she’s not allowed to ban speech based on religious grounds. Your child is not calling anyone names or harassing his classmates. The teacher is not supposed to give a religious lesson in public school. I’d let her know that I’m still asking my child, out of respect for his classmates, to say “oh my gosh” instead, but that he will not be perfect and she is not to speak to him again about this subject.

And then I’d ask my kid to try to switch to “oh my gosh,” but I’d emphasize it’s not a swear and that we’re asking him to try to say “gosh” because some of his classmates might feel better if he says that instead of “god.”

And then I’d drop it unless the teacher keeps harassing him, at which point I’d go over her head.

Edited to add: I would also document everything with date of discussion and what you and the teacher both said. You’ll need it if you have to go to the principal.