According to Hindu belief the soul choses it's parents before birth. If for whatever reason the baby is not born the soul gets attached to the person responsible due to the anger of not getting to live the life that it wanted to. The soul even follows them to their next lives.

OP might have killed the foetus and that's why the soul is attached.