That's true. You will continue to accelerate until hitting the surface. And the speed you'll be accelerated to is the escape velocity. And it's because your starting and end states are the same, just in reverse.

When calculating escape velocity you're saying "how fast do I need to launch myself from the surface of the planet, so that I don't come to rest until infinity?" But for this question you're saying "if I'm at rest at infinity, what speed will I have when I hit the surface of the planet?" Since the start and end points are the same, the speeds are also the same.

If the force that accelerated you were constant, you could indeed reach a speed arbitrarily close to the speed of light just by starting arbitrarily far away – and escape velocity wouldn't be a thing, either.

But gravity is inversely proportional to distance squared, so if you're very far away (in a toy universe where there are only you plus the object you're falling towards), your initial acceleration will also be very slow and almost all of the speed gain will happen when you're already very close to the object.

If you're starting from rest and falling towards a planet, then you will accelerate faster until you hit. Remember that gravity gets lower the further away you are and eventually you will be far enough away that the planet's pull on you isn't relevant anymore - escape velocity is how fast you would accelerate to if you started at the point where you are only just falling towards the planet (simplifying a lot but close enough)