Viewing a single comment thread. View all comments

029187 OP t1_iqp6s79 wrote

Reply to comment by dasayan05 in [Discussion] If we had enough memory to always do full batch gradient descent, would we still need rmsprop/momentum/adam? by 029187

what if, as another poster said, we did full batch but also injected noise into it?

1

dasayan05 t1_iqp8hnf wrote

possible. but what is the advantage with that ? even if we did find a way to explicitly noise the data/gradient, we are still better off with mini-batches as they offer less memory consumption

2

029187 OP t1_iqrinm2 wrote

If its only as good, then it has no benefit. But if it ends up being better, then it is useful for situations where we have enough memory.

​

https://arxiv.org/abs/2103.17182

​

This paper here is claiming they might have found interesting ways to potentially make it better.

1

Red-Portal t1_iqpczjb wrote

People have tried it, and so far no one has been able to achieve the same effect. It's still somewhat of an open research problem.

1