It's not a guess, we can see it! For example, here's an image from the Hubble telescope of one galaxy pulling material from a smaller nearby galaxy. So now we know that's a thing that happens - and we're not just guessing, it's something we're actually observing happening.

BTW those two galaxies are 200 million light-years away. For reference, our own galaxy is about 100K light years across. So that's an event we're directly seeing that's about 2000 galaxy-widths away from our own galaxy.