MinaKovacs t1_j9ref87 wrote
We are so far away from anything you can really call "AI" it is not on my mind at all. What we have today is simply algorithmic pattern recognition and it is actually really disappointing. The scale of ChatGPT is impressive, but the performance is not. Many many thousands of man-hours were needed to manually tag training datasets. The only place "AI" exists is in the marketing department.
Small-Fall-6500 t1_j9ro4tl wrote
About a year or two ago, we were so far away from having an AI model that could reliably and easily produce high quality artwork that almost no one was thinking about AI art generators.
Then diffusion models became a thing.
AGI could easily be very similar; it could take decades to discover what is required to make an AGI, or just a few more years. But AGI is not quite like diffusion models, because a diffusion model can’t create and carry out a plan to convert every single living thing into computronium or whatever helps maximize its utility function.
arg_max t1_j9rt2ew wrote
The thing is that the theory behind diffusion models is at least 40-50 years old. Forward diffusion is a discretization of a stochastic differential equations that transforms the data distribution into a normal distribution. People figured out that it is possible to reverse this process, so to go from the normal distribution back to the data distribution using another sde In the 1970s. The thing is that this reverse SDE contains the score function, so the gradient of the log density of the data and people just didn't really know how to get that from data. Then some smart guys came along, found the ideas about denoising score matching from the 2000s and did the necessary engineering to make it work with deep nets.
The point I am making is that this problem was theoretically well understood a long time ago, it just took humanity lots of years to actually be able to compute it. But for AGI, we don't have such a recipe. There's not one equation hidden in some old math book that will suddenly get us AGI. Reinforcement learning really is the only approach I could think of but even there I just don't see how we would get there with the algorithms we are currently using.
SchmidhuberDidIt OP t1_j9rwh3i wrote
What about current architectures makes you think they won’t continue to improve with scale and multimodality, provided a good way of tokenizing? Is it the context length? What about models like S4/RWKV?
[deleted] t1_j9ruj0m wrote
[deleted]
Veedrac t1_j9tnubd wrote
Ah, yes, those well-understood equations for aesthetic beauty from the 1970s.
sticky_symbols t1_j9rf56v wrote
Many thousands of human hours are cheap to buy, and cycles get cheaper every year. So those things aren't really constraints except currently for small businesses.
MinaKovacs t1_j9rfnej wrote
True, but it doesn't matter - it is still just algorithmic. There is no "intelligence" of any kind yet. We are not even remotely close to anything like actual brain functions.
gettheflyoffmycock t1_j9rqd5w wrote
Lol, downvotes. this subreddit has been completely overran by non engineers. I guarantee no one here has ever custom trained and inferred with a model outside of API calls. Crazy. Since ChatGPT, open enrollment ML communities are so cringe
Langdon_St_Ives t1_j9rsn1f wrote
Or maybe downvotes because they’re stating the obvious. I didn’t downvote for that or any other reason. Just stating it as another possibility. I haven’t seen anyone here claim language models are actual AI, let alone AGI.
royalemate357 t1_j9rsqd3 wrote
>We are not even remotely close to anything like actual brain functions. Intelligence need not look anything remotely close to actual brain functions though, right? Like a plane's wings don't function anything like a bird's wings, yet it can still fly. In the same sense, why must intelligence not be algorithmic?
At any rate I feel like saying that probabilistic machine learning approaches like GPT3 are nowhere near intelligence is a bit of a stretch. If you continue scaling up these approaches, you get closer and closer to the entropy of natural language/whatever other domain, and if youve learned the exact distribution of language, imo that would be "understanding"
wind_dude t1_j9rvmbb wrote
When they scale they hallucinate more, produce more wrong information, thus arguably getting further from intelligence.
royalemate357 t1_j9rzbbc wrote
>When they scale they hallucinate more, produce more wrong information
Any papers/literature on this? AFAIK they do better and better on fact/trivia benchmarks and whatnot as you scale them up. It's not like smaller (GPT-like) language models are factually more correct ...
wind_dude t1_j9s1cr4 wrote
I'll see if I can find the benchmarks, I believe there are a few papers from IBM and deepmind talking about it. And a benchmark study in relation to flan.
MinaKovacs t1_j9s04eh wrote
It's just matrix multiplication and derivatives. The only real advance in machine learning over the last 20yrs is scale. Nvida was very clever and made a math processor that can do matrix multiplication 100x faster than general purpose CPUs. As a result, the $1bil data center, required to make something like GPT-3, now only costs $100mil. It's still just a text bot.
sticky_symbols t1_j9w9b6e wrote
There's obviously intelligence under some definitions. It meets a weak definition of AGI since it reasons about a lot of things almost as well as the average human.
And yes, I know how it works and what its limitations are. It's not that useful yet, but discounting it entirely is as silly as thinking it's the AGI we're looking for.
Marcapiel t1_j9rjpbi wrote
The definition of intelligence is quite simple, we definitely have AI.
Viewing a single comment thread. View all comments