Viewing a single comment thread. View all comments

currentscurrents t1_j6btqta wrote

Reply to comment by visarga in [N] OpenAI has 1000s of contractors to fine-tune codex by yazriel0

Frankly though, there's got to be a way to do with less data. The typical human brain has heard maybe a million words of english and about 8000 hrs of video per year of life. (and that's assuming dreams are generative training data somehow - halve that if you only get to count the waking world)

We need something beyond transformers. They were a great breakthrough in 2018, but we're not going to get to AGI just by scaling them up.

6

visarga t1_j6c1rmo wrote

Humans are harder to scale, and it took billions of years for evolution to get here, with enormous resource and energy usage. A brain trained by evolution is already fit for the environment niche it has to inhabit. But an AI model has none of that, no evolution selecting the internal structure to be optimal. So they have to compensate by learning these things from tons of raw data. We are great at some tasks that relate to our survival, but bad at other tasks, even worse than other animals or AIs - we are not generally intelligent either.

Also, most AIs don't have real time interaction with the world. They only have restricted text interfaces or APIs, no robotic bodies, no way to do interventions to distinguish causal relations from correlations. When an AI has feedback loop from the environment it gets much better at solving tasks.

3

vivehelpme t1_j6cno58 wrote

22 hours of video content per day?

1

currentscurrents t1_j6e4get wrote

I rounded. Data collection is like astronomy, it's the order of magnitude that matters.

1

MysteryInc152 t1_j6jkmus wrote

The human brain has trillions of synapses (the closest biological equivalent to parameters), is multimodal and evolution fine-tuned.

1

currentscurrents t1_j6m3ik5 wrote

We could make models with trillions of parameters, but we wouldn't have enough data to train them. Multimodality definitely allows some interesting things but all existing multimodal models still require billions of training examples.

More efficient architectures must be possible - evolution has probably discovered one of them.

1