Non-linear doesn’t mean polynomial. There are many non-linear functions polynomials cannot approximate well, such as softmax() or even just sin(). Ultimately, it’s enough to approximate complex functions with simple functions without having to worry about picking the correct complex function.
reddisaurus t1_iqz48i8 wrote
Reply to [D] Why restrict to using a linear function to represent neurons? by MLNoober
Non-linear doesn’t mean polynomial. There are many non-linear functions polynomials cannot approximate well, such as softmax() or even just sin(). Ultimately, it’s enough to approximate complex functions with simple functions without having to worry about picking the correct complex function.