Imagine you’re traveling at NLS speeds, which would be necessary for travel between solar systems. Just to escape our solar system you have the Oort Cloud to deal with, and without some kind of deflection system would rip any vessel to shreds (really it would blow up but that’s a more fun way to imagine it).

Even barring that, there are probably countless small bodies we haven’t detected floating freely that would pierce the hull when traveling at those speeds.