But the space between objects in the universe is expanding too. They're not all just moving in an "outward" direction from some central point.

Space is expanding in all directions.