That's true. You will continue to accelerate until hitting the surface. And the speed you'll be accelerated to is the escape velocity. And it's because your starting and end states are the same, just in reverse.

When calculating escape velocity you're saying "how fast do I need to launch myself from the surface of the planet, so that I don't come to rest until infinity?" But for this question you're saying "if I'm at rest at infinity, what speed will I have when I hit the surface of the planet?" Since the start and end points are the same, the speeds are also the same.