The answer is time dilation, essentially. Even though our spacetime is globally curved, it's more intuitive to think about a flat spacetime, since that removes ambiguity about the definitions of distances and relative velocities. In this scenario, the most distant objects are receding at velocities arbitrarily close to the speed of light. That means that even though they might have traveled a great distance from their origin point, arbitrarily little time has passed for them, due to time dilation. So we can receive light from these objects that tells us the state of the universe at arbitrarily early times, even though they could be quite distant from the origin point for the universe.

(I'll also note that a common answer to this question is that the universe didn't begin at a point. While that would also resolve the problem, it's not something we can say for certain.)