Here is my answer: First, let’s ask why not an exponential or logarithmic function instead of a quadratic or a higher order polynomial? Or maybe a sinusoid function? The thing is, we might be needing one of such nonlinearities, or maybe another kind of nonlinearity, based on the problem, and, we don’t know it. The idea with neural networks is that it combines many simple neurons that can learn any of these nonlinearities inside it during training. If such a quadratic relationship is a relevant feature for your problem, it will learn it, meaning that some of the neurons will end up with simulating that quadratic relationship. This is a much more flexible way than hard coding the nonlinearity right away from the beginning.