I also thought of this, but perhaps what the old man feared was the three would follow her when she got off at a stop and thus fulfil the condition of being alone with her.

It's sad that the bus driver had to die, I wonder whether he could have been saved too if all three of them had managed to get off together. But I get that the old man was probably too scared and didn't think the bus driver would believe him anyway. Also probably didn't want to test the 'alone' thing enough to risk the men realising he was on to them.