Viewing a single comment thread. View all comments

SoylentRox t1_j8e7bvb wrote

Reply to comment by genericrich in Anthropic's Jack Clark on AI progress by Impressive-Injury-91

This is false. None of the algorithms we use now existed. They were not understood. Prior versions of the algorithms that were much simpler did exist. It is chicken egg - we needed immense amounts of compute to find the algorithms needed to take advantage of immense amounts of compute.

8

FusionRocketsPlease t1_j8eeegf wrote

What do you mean computation is needed to discover algorithms?

5

SoylentRox t1_j8efx92 wrote

Many algorithms don't show a benefit unless used at large scales. Maybe "discover" is the wrong word, if your ml researcher pool has 10,000 ideas but only 3 are good, you need a lot of compute to benchmark all the ideas to find the good ones. A LOT of compute.

Arguably you "knew" about the 3 good ideas years ago but couldn't distinguish them from the rest. So no, you really didn't know.

Also transformers are a recent discovery (2017), it required compute and software frameworks to support complex nn graphs to even develop the idea.

7

genericrich t1_j8ebipv wrote

Gradient Descent was well understood in the early 20th century for fluid dynamics I believe.

So, not false. :)

−1

SoylentRox t1_j8ecpwg wrote

But yes false? Your argument is like saying people in 1850 knew about aerodynamics and combustion engines.

Which, yes, some did. Doesn't negate the first powered flight 50 years later, it was still a significant accomplishment.

7

genericrich t1_j8edlrk wrote

<eyeroll> Nobody is saying there haven't been major changes in AI in the last few years. I certainly am not saying that.

But many of the underlying algorithms were well understood in different disciplines and the industry knew they would have application for AI, but the data and infrastructure just weren't there in the 60s or 1980s.

−1

SoylentRox t1_j8eh4tu wrote

My point is that scale matters. A 3d multiplayer game was "known" to be possible in the 1950s. They had mostly offline rendered graphics. They had computer networks. There was nothing in the idea that couldn't be done, but in practice it was nearly completely impossible. The only thing remotely similar cost more than the entire manhattan project and they were playing that 3d game in real life. https://en.wikipedia.org/wiki/Semi-Automatic_Ground_Environment

If you enthused about future game consoles in the 1950s, you'd get blown off. Similarly, we have heard about the possibility of AI about that long - and suddenly boom, the dialogue of HAL 9000 for instance is actually quite straightforward and we could duplicate EXACTLY the functions of that AI right now, no problem. Just take a transformer network, add some stream control characters to send commands to ship systems, add a summary of the ship's system status to the memory it sees each frame. Easy. (note this would be dangerous and unreliable...just like the movie)

Also note that in the the 1950s there was no guarantee the number of vacuum tubes you would need to support a 3d game (hundreds of millions) would EVER be cheap enough to allow ordinary consumers to play them. The transistor had not been invented.

Humans for decades thought an AGI might take centuries of programming effort.

11