I think that potentially the answer is that time is not linear. It is perceived by humans as linear. It might be thought of more as interdimensional. “Your past can’t be your future,” and so on. You end up in a new timeline when you travel. It’s difficult to describe without spherical Venn diagrams.