To be more specific, the brain has an entire region dedicated to recognizing faces in the fusiform gyrus. When upside-down that region is not fully activated, as the orientation doesn't correspond to the brains idea of what a face is (two eyes above a nose and mouth). Because of this, the brain fills in some gaps. When inverted the face region activates and basically tells the rest of the brain something is wrong.

Fun fact on this topic: Humans aren't born with the ability to discern specifically human faces, but are born with a tendency to look at faces so as to develop this skill better. And yet in the same breath, early infants can discern between similar faces on other species (tested with pictures of a type of monkey) that even individuals who spend their lives researching the species may not be able to detect.