currentscurrents t1_j7q8q5v wrote on February 8, 2023 at 5:18 PM

Reply to comment by wintermute93 in [Discussion] Cognitive science inspired AI research by theanswerisnt42

So far nobody's figured out a good way to train them.

You can't easily do backprop, but you wouldn't want to anyway - the goal of SNNs is to run on ultra-low-power analog computers. For this you need local learning, where neurons can learn by communicating only with adjacent neurons. There's some ideas (forward-forward learning, predictive coding, etc) but so far nothing is as good as backprop.

There's a bit of a chicken-and-egg problem too. Without a good way to train SNNs, there's little interest in the specialized hardware - and without the hardware, there's little interest in good ways to train them. You can emulate them on regular computers but that removes all their benefits.

katadh t1_j7s73hw wrote on February 9, 2023 at 12:54 AM

SNN - ANN conversion and surrogate gradient methods can both get good results these days, so training has become a lot more comparable to ANNs than it was in the past. I would agree though that there is a disconnect between the hardware and software still which is preventing SNNs from reaching the dream of super low power models.

currentscurrents t1_j7sri62 wrote on February 9, 2023 at 3:26 AM

SNN-ANN conversion is kludge - not only do you have to train an ANN first, it means your SNN is incapable of learning anything new.

Surrogate gradients are better! But they're still non-local and require backwards passes, which means you're missing out on the massive parallelization you could achieve with local learning rules on the right hardware.

Local learning is the dream, and would have benefits for ANNs too: you could train a single giant model distributed across an entire datacenter or even multiple datacenters over the internet. Quadrillion-parameter models would be technically feasible - I don't know what happens at that scale, but I'd sure love to find out.