This phenomenon is called diffusion. When you use a waterproof marker you can't wipe it off with water because the solvent and the dye of your marker are apolar substances. Water a polar and doesn't mix (well) with apolar substances. Polymers however are apolar as well. So even though vinyl isn't a liquid it's made of long apolar chains (which I always imagine like spaghetti) and those polymer chains have the same apolar properties like the dye and solvent in your waterproof marker, so they mix well. Over time the dye is simply absorbed into the vinyl. If you would take a water based dye this will not happen.