Viewing a single comment thread. View all comments

maybelator t1_ivxgacq wrote

Reply to comment by jrkirby in [R] ZerO Initialization: Initializing Neural Networks with only Zeros and Ones by hardmaru

> Is the derivative of ReLU at 0.0 equal to NaN, 0 or 1?

The derivative of ReLu is not defined at 0, but its subderivative is and is the set [0,1].

You can pick any value in this set, and you end up with (stochastic) subgradient descent, which converges for small enough learning rates (to a critical point).

For ReLu, the discontinuity are of mass 0 and are not "attractive", ie there is no reason for the iterate to end up exactly at 0, so it can be safely ignored. This is not the case for the L1 norm for example, whose subgradient at 0 is [-1,1]. It present a "kink" at 0 as the subderivative contains a neighborhood of 0, and hence is attractive: your iterate will get stuck there. In these cases, it is recommended to use proximal algorithms, typically forward-backward schemes.

91

robbsc t1_ivypqg0 wrote

Thanks for taking the time to type this out

3

samloveshummus t1_iw1o1jg wrote

This has to be one of the most useful comments I've read in nearly ten years on Reddit! You must be a gifted teacher.

3

zimonitrome t1_iwbmzoq wrote

Huber loss let's go.

1

maybelator t1_iwbpkjo wrote

Not if you want true sparsity !

1

zimonitrome t1_iwbst8p wrote

Can you elaborate?

1

maybelator t1_iwbxutj wrote

The Huber loss encourages the regularized variable to be close to 0. However, this loss is also smooth: the amplitude of the gradient decreases as the variable nears its stationary point. In consequence, it will have many coordinates close to 0 but not exactly. Achieving true sparsity requires thresholding which adds a a lot of other complications.

In contrast the amplitude of the gradient of the L1 norm (absolute value in dim 1) remain the same no matter how close it gets to 0. The functional has a kink (the subgradient contains a neighborhood of 0). In consequence, if you used a well-suited optimization algorithm, the variable will have true sparsity, i.e. a lot of exact 0.

2

zimonitrome t1_iwc14i5 wrote

Wow thanks for the explanation, it does make sense.

I had a pre-conception that all optimizers dealing with any linear functions (kinda like L1 norm) still produce values close to 0.

I can see someone disregarding tiny values when using said sparsity (pruning, quantization) but didn't think that it would be exactly 0.

1