If you think of the world as being made of dots, each dot emits light in a bunch of different directions. With the light from all the dots pouring into a big hole, it's just a bunch of light. The information for color is kind of lost to the melting pot of all the beams of light

A tiny hole only let's one beam through of each "dot", which retains the color information on the wall because it's not melding with a bunch of other rays.

It's inverted because of how the beams work. You can't have a dot higher than the hole somehow appear higher than the hole in the final image. It travels on a straight line, and thus ends up passing through the hole and landing at the bottom of the final image.