> Wouldn't a 50 meter change compound over 120million years?

The effect on the size and shape of Earth's orbit would not "compound", it would still be small. The effect on Earth's position around its orbit does add up though.

Even without large impacts the solar system is chaotic, in the mathematical sense. We can make good predictions for the next few million years. But we cannot say, for example, what season it will be in the northern hemisphere exactly 100 million current-day years from now.