Viewing a single comment thread. View all comments

katadh t1_j7pv6f1 wrote

Look into spiking neural networks if you're not aware of them already

3

wintermute93 t1_j7pxlsj wrote

Have spiking networks actually produced any meaningful results? Granted, the last time I looked into the field was like 5 years ago, but back then the answer was definitely "no, these are just a toy".

2

currentscurrents t1_j7q8q5v wrote

So far nobody's figured out a good way to train them.

You can't easily do backprop, but you wouldn't want to anyway - the goal of SNNs is to run on ultra-low-power analog computers. For this you need local learning, where neurons can learn by communicating only with adjacent neurons. There's some ideas (forward-forward learning, predictive coding, etc) but so far nothing is as good as backprop.

There's a bit of a chicken-and-egg problem too. Without a good way to train SNNs, there's little interest in the specialized hardware - and without the hardware, there's little interest in good ways to train them. You can emulate them on regular computers but that removes all their benefits.

3

katadh t1_j7s73hw wrote

SNN - ANN conversion and surrogate gradient methods can both get good results these days, so training has become a lot more comparable to ANNs than it was in the past. I would agree though that there is a disconnect between the hardware and software still which is preventing SNNs from reaching the dream of super low power models.

1

currentscurrents t1_j7sri62 wrote

SNN-ANN conversion is kludge - not only do you have to train an ANN first, it means your SNN is incapable of learning anything new.

Surrogate gradients are better! But they're still non-local and require backwards passes, which means you're missing out on the massive parallelization you could achieve with local learning rules on the right hardware.

Local learning is the dream, and would have benefits for ANNs too: you could train a single giant model distributed across an entire datacenter or even multiple datacenters over the internet. Quadrillion-parameter models would be technically feasible - I don't know what happens at that scale, but I'd sure love to find out.

2

katadh t1_j7s68c6 wrote

There has been a lot of progress in the last 2 - 3 years. They're still not quite at the level of ANNs in general but have been gaining ground quickly and do outperform ANNs on some specific tasks -- usually things with a temporal component but low data dimensionality per time-step. Another area with comparable results to ANNs would be object detection.

1

theanswerisnt42 OP t1_j7ssq9y wrote

Thanks for the suggestion! Out of curiosity, has there been any theoretical work comparing SNNs and ANNs to explore if there are any advantages of using them?

1

katadh t1_j7x7924 wrote

There's been a decent amount of work showing that they should be much more energy efficient. There is some empirical work showing other potential advantages (like robustness) but most of that work is still too nascent to be definitive.

1