Definitely remove all the old caulk, doesn't sound like it's is the right product. Like others have said silicon is the way to go. You can even get mould resistant ones especially for bathrooms. I have heard you are supposed to fill the tub with water to weigh it down whilst doing the silicon.