As everyone else has said, I've heard it's mostly, if not only the result of social conditioning.

I'd love someone with some background to confirm or deny this, but I think in African drumming music (which I'm sure is incredibly varied), being much more rhythmic than classical Western music, it's the speed of a piece that determines the emotional impact. Fast pieces are happy, slow pieces are sad.