Indeed. The presence of the impure could act like a gateway of sorts and the vengeful spirits/demons could enter the world and cause all sorts of problems. I think it’s supporting evidence of the belief that normal people in general were capable of derailing magic by their mere presence.

Much like someone who has the power of invisibility, but it only works when no one is looking. Which I think is what Palaephatus is getting at.