You are assuming that the universal speed limit, aka. the speed of light, applies to space itself. But this is not the case. Nothing can move faster then the univerals speed limit compared to the fabric of spacetime but the fabric itself does move faster then this. As far as we can tell the space that is infinatly far away from us also have infinate speed away from us. It could therefore have moved from a single point to where it is now.