Space is vast. Like so big that you cant comprehend the distances involved. The stars we see in the night sky are very far away hence we dont have a revolving door of guest stars in our sight line. Think of it like this: if you see a mountain in the distance and you run 5 miles in any direction, the mountain doesnt appear to have moved at all, even though you know you moved 5 miles. As the distance between objects increases, the angle of the triangle formed from the observed object, your previous position, and the current position shrinks to imperceptible.