Sure, you can do it if you have enough data, and a powerful enough computer.

Idk how you're going to do reinforcement learning to update the transformer weights though (I assume you want to use a transformer?). That's a lot of computation. The bigger your model is, the slower this update step will be.

Are you separating hearing and speaking/moving in time? Like are they separate steps that can't happen at the same time? My question then is why not make them simultaneous?

My (non-expert) take :

The problem is that there are many black boxes with that.

LLMs work well when we have a huge amount of data to train the model with. In an oversimplified way, LLMs predict the next word, based on the previous data they've seen. But how to "predict the next action you'll take" ? If we had a massive amount of "sensation --> action" data (probably just like the human brain accumulates during life ?) then that would be possible. I haven't heard of a way to achieve that today, and I think it's more complicated than that anyways.

I think what's your suggesting is kinda like what they try to do with Google's SayCan : but as you can see, for the moment there's no easy way to link LLMs with physical action. LLMs manage to create plausible scenarios of what's happening, or what could be some consequences of action X, but practically it's not usable yet.

There's also the fact that, as someone pointed earlier, there are issues with continuous learning, such as catastrophic forgetting. I think many brilliant minds are actively trying to surpass those issues, but it's no easy feat.

Catastrophic interference

>Catastrophic interference, also known as catastrophic forgetting, is the tendency of an artificial neural network to abruptly and drastically forget previously learned information upon learning new information. Neural networks are an important part of the network approach and connectionist approach to cognitive science. With these networks, human capabilities such as memory and learning can be modeled using computer simulations. Catastrophic interference is an important issue to consider when creating connectionist models of memory.

^([ )^(F.A.Q)^( | )^(Opt Out)^( | )^(Opt Out Of Subreddit)^( | )^(GitHub)^( ] Downvote to remove | v1.5)

I don't think it has to be AGI for it to kick off the Singularity.

A recent approach saves past experience data and loads it back for in-context-learning. The model itself can be task generic. So it learns by collecting new data.

We can have a model trained on a large video dataset, and then fine-tuned for various tasks like GPT3.

Using YouTube as training data we'd get video + audio which decompose in image, movement, body pose, intonation, text transcript, metadata all in parallel. This dataset could dwarf the text datasets we have now, and it will have lots of information that doesn't get captured in text, such as physical movements for achieving a specific task.

I think the OP was almost right. The multi-modal AI will be a good base for the next step, but it needs instruction tuning and RLHF. Just pre-training is not enough.

One immediate application I see - automating desktop activities. After watching many hours of screen casting from YT, the model will learn how to use apps and solve tasks at first sight like GPT-3.5, but not limited to just text.

Yes probably. You don't need to learn anything to be generally intelligent if you've already been trained on the entirety of human knowledge.

How does it generalize across tasks, concepts, etc?

It should have persistence. That is hard to achieve when the model is big and slow. Especially if the model gets slower because of adding persistence and multimodality

I wouldn’t say it’s AGI, but I would also say you don’t need AGI to kick off the Singularity. 😉

If it can learn new skills almost on the fly, proto agi but I that that may take a bit of time

First off... Very interesting... But just so you know that wouldnt be a language model anymore

They dont really have a term for that other than multimodal... Multimodal world model???

Models cant speak or hear when they want to Its just not part of their programming

They respond to input

So if they are receiving continuous input... Theoretically they should be continuously outputting...

The whole conversation history could be saved into a database

Reward models are currently given texts with scores made by humans its called RLHF, or Reinforcement Learning from Human Feedback... AI doesnt do the scoring... That's for language models though...

How could they know what is good and what is bad???

Now for world models reinforcement learning works differently... Which is probably what youre referring to... I wont go into it because its pretty complex...

Updating its weights continuously is currently impossible due to an energy inefficiency problem with the von Neumann hardware architecture... Basically traditional cpus and gpus... More basically, it requires too many computations and too much electricity to continuously "backpropagate" (data science word) data input...

Conversations shouldn't be encoded into a language model either imo... because of "hallucinations" they may make some things up that didn't happen

Querying a database of old conversations is better and will always be more accurate

In order for an AGI to truly be AGI by definition it needs to be able to learn any task... This is currently possible manually server side through manual backpropagation... But this is not possible continuously like how human brains work...

Humans continuously learn...

An AI neural network manually learns by being fed data through a command line interface... This is called "training"... Data science terminology

An AI neural network model is then "deployed" aka opened and ran on a single gpu or multiple depending on model size... When a language model is running it is said to be in "inference mode"... More terminology

We need an entirely different hardware architecture in order to run AI Neural Networks in training and Inference mode simultaneously...

Photonics or Neuromorphic computing, perhaps a combination of both... These seem like the way forward in my opinion

I agree with all of this, but just to be a bit over-pedantic on one bit:

> Models cant speak or hear when they want to Its just not part of their programming.

As you said it's not part of their programming, in today's models. In general though, it wouldn't be too difficult to construct a new model that judges at each timestep based on both external stimuli and internal hidden states when to speak/interrupt or listen intently. Actually at first glance such a thing sounds trivial.

Where would you get one trillion tokens for touch and actuator state (proprioception) from?

As another commenter said, you also forgot reward/goal/feelings.