mocny-chlapik t1_j08w90k wrote on December 14, 2022 at 10:23 PM

Can aiplanes fly? They clearly do not flap their wings so we shouldn't say they fly. In the nature, we can see that flying is based on flapping wings, not on jet engines. Thus we shouldn't say that airplanes fly, since clearly jet engines are not capable of flight, they are merely moving air with their turbines. Even though we can see that the airplanes are in the air, it is only a trick and they are actually not flying in the philosophical sense of that word.

leondz t1_j0a9tdd wrote on December 15, 2022 at 4:36 AM

People fly airplanes. Airplanes don't fly on their own.

respeckKnuckles t1_j0ajskv wrote on December 15, 2022 at 6:17 AM

Airplanes can fly on autopilot. Autopilot is part of the autopilot-using plane. Therefore, at least some airplanes can fly on their own.

leondz t1_j0cjg4x wrote on December 15, 2022 at 5:36 PM

autopilot helps the pilot. it requires the pilot. who flies the plane

respeckKnuckles t1_j0ctc2w wrote on December 15, 2022 at 6:39 PM

https://en.wikipedia.org/wiki/Autonomous_aircraft#:~:text=An%20autonomous%20aircraft%20is%20an,unmanned%20aerial%20vehicle%20or%20drones.

WikiSummarizerBot t1_j0ctdx6 wrote on December 15, 2022 at 6:39 PM

[Autonomous aircraft](https://en.wikipedia.org/wiki/Autonomous_aircraft#:~:text=An autonomous aircraft is an,unmanned aerial vehicle or drones)

>An autonomous aircraft is an aircraft which flies under the control of automatic systems and needs no intervention from a human pilot. Most autonomous aircraft are unmanned aerial vehicle or drones. However, autonomous control systems are reaching a point where several air taxis and associated regulatory regimes are being developed.

^([ )^(F.A.Q)^( | )^(Opt Out)^( | )^(Opt Out Of Subreddit)^( | )^(GitHub)^( ] Downvote to remove | v1.5)

leondz t1_j0cugwd wrote on December 15, 2022 at 6:46 PM

Surely you're not contending that autopilots

> Airplanes can fly on autopilot. Autopilot is part of the autopilot-using plane.

are only used in the handful of autonomous flights? also: if autonomous flights were reliable, and could fly reliably, they'd be used more! but they're not, because the problem isn't solved, because good autonomous flight isn't there, because autopilots can't reliably fly planes

CherubimHD t1_j090ds4 wrote on December 14, 2022 at 10:52 PM

Except that there is not philosophical understanding of the act of flying.

blind_cartography t1_j09jjcl wrote on December 15, 2022 at 1:12 AM

There is a philosophical understanding of what we mean by the word 'flying' though. It's still a little bit obtuse of an argument, since flying and thinking are quite different conceptual categories (maybe birds would argue different), but the point that we should not limit our definition of thinking (and knowing, believing, etc) to exactly how human's do it is spot on since i) many humans' thinking can't really be explained either and ii) I've met many humans whose output was purely a result of fine-tuning a base statistical phenotype on temporally adjacent stimuli.

economy_programmer_ t1_j098oz2 wrote on December 14, 2022 at 11:51 PM

I strongly disagree.
First of all, you should define the "philosophical sense of fly", and second of all, try to imagine a perfect robotic replica of the anatomy of a bird, why that should not be considered fly? And if it is considered flying, what's the line that divides an airplane, a robotic bird replica and a real bird? I think you are reducing a philosophical problem to a mechanical problem.

Nameless1995 t1_j09c3f0 wrote on December 15, 2022 at 12:16 AM

It was a satire.

economy_programmer_ t1_j09cozr wrote on December 15, 2022 at 12:21 AM

I don't think so

Nameless1995 t1_j09eifz wrote on December 15, 2022 at 12:34 AM

/u/mocny-chlapik thinks OP paper is suggesting that LLMs don't understand by pointing out that differences in how humans understand and how LLMs "understand". /u/mocny-chlapik is criticizing this point by showing that this is similar to saying aeroplanes don't fly (which they obviously do under standard convention) just because of the differences in the manner in which they fly and in which birds do. Since the form of the argument doesn't apply in the latter case, we should be cautious of applying this same form for the former case. That is their point. If you think it is not a satire meant to criticize OP, why do you think a comment is talking about flying in r/machinelearning in a post about LLMs and understanding?

Pikalima t1_j0az5t9 wrote on December 15, 2022 at 9:36 AM

I don’t know who was the first to use the analogy to bird flight, but it’s a somewhat common refutation used in philosophy of AI. That’s just to say, it’s been used before.

Nameless1995 t1_j09pzkf wrote on December 15, 2022 at 2:01 AM

> “Here’s a fragment of text. Tell me how this fragment might go on. According to your model of the statistics of human language, what words are likely to come next?”1

> Even if an LLM is fine-tuned, for example using reinforcement learning with human feedback (e.g. to filter out potentially toxic language) (Glaese et al., 2022), the result is still a model of the distribution of tokens in human language, albeit one that has been slightly perturbed.

....I don't see what's the point is.

I have an internal model of a world developed from the statistics of my experiences through which I model mereology (object boundaries, speech segmentation, and such), environmental dynamics, affordances, and the distribution of next events and actions. If the incoming signal is highly divergent from my estimated distribution, I experience "surprise" or "salience". In my imagination, I can use the world model generatively to simulate actions and feedbacks. When I am generating language, I am modeling a distribution of "likely" sequence of words to write down conditioned on a high level plan, style, persona, and other associated aspects of my world model (all of which can be modeled in a NN, and may even be implicitly modeled in LLMs; or can be constrained in different manners (eg. prompting)).

Moreover in neuroscience and cognitive science, there is a rise of predictive coding/predictive error minimization/predictive processing frameworks treating error minimization as a core unifying principle about function of the cortical regions of brains:

https://arxiv.org/pdf/2107.12979.pdf

> Predictive coding theory is an influential theory in computational and cognitive neuroscience, which proposes a potential unifying theory of cortical function (Clark, 2013; K. Friston, 2003, 2005, 2010; Rao & Ballard, 1999; A. K. Seth, 2014) – namely that the core function of the brain is simply to minimize prediction error, where the prediction errors signal mismatches between predicted input and the input actually received

> “Here’s a fragment of text. Tell me how this fragment might go on. According to your model of the statistics of human language, what words are likely to come next?”1

One can argue the semantics of whether LLMs can be understood to be understanding meanings of words if not learning in the exact kind fo live physically embedded active context as humans or not, but I don't see the point of this kind of "it's just statistics" argument -- it seems completely orthogonal. Even if we make a full-blown embodied multi-modal model it will "likely" constitute a world model based on the statistics of environmental-oberservations, providing distributing of "likely" events and actions given some context.

My guess it that these statements makes people think in frequentists terms which feels like "not really understanding" but merely counting frequencies of words/tokens in data. But that's hardly what happens. LLMs can easily generalize to highly novel requests alien to anything occuring in the data (eg. novel math problems, asking about creatively integrating nordvpn advertisement to any random answer and so on - even though nothing as familiar appear in the training data (I guess)). You can't really explain those phenomena without hypothesizing that LLMs model deeper relational principles underlying the statistics of the data -- which is not necessarily much different from "understanding".

Sure, sure, it won't have the exact sensori-motor-affordance associations with language; and we have to go further for grounding; but I am not sure why we should be drawing a hard line to "understanding" because some of these things are missing.

> These examples of what Dennett calls the intentional stance are harmless and useful forms of shorthand for complex processes whose details we don’t know or care about.

The author seems to cherry-pick from Dennett. He is making it sound as if taking an intentional stance is simply about "harmless metaphorical" ascriptions of intentional states to systems; and based on intentional stance we can be licensed to attribute intentional states to LLMs.

But Dennett also argues against the idea that there is some principled difference between "original/true intentionality" vs "as-if metaphorical intentionality". Instead Dennett considers that to be simply a matter of continuum.

> (1) there is no principled (theoretically motivated) way to distinguish ‘original’ intentionality from ‘derived’ intentionality, and

> (2) there is a continuum of cases of legitimate attributions, with no theoretically motivated threshold distinguishing the ‘literal’ from the ‘metaphorical’ or merely

https://ase.tufts.edu/cogstud/dennett/papers/intentionalsystems.pdf

Dennett seems also happy to attribute "true intentionality" to simple robots (and possibly LLMs (I don't see why not; his reasons here also applies to LLMs)):

> The robot poker player that bluffs its makers seems to be guided by internal states that function just as a human poker player’s intentions do, and if that is not original intentionality, it is hard to say why not. Moreover, our ‘original’ intentionality, if it is not a miraculous or God-given property, must have evolved over the eons from ancestors with simpler cognitive equipment, and there is no plausible candidate for an origin of original intentionality that doesn’t run afoul of a problem with the second distinction, between literal and metaphorical attributions. ‘as if’ cases.

The author seems to be trying to do the exact opposite by arguing against the use of intentional ascriptions to LLMs in a "less-than-metaphorical" sense (and even in the metaphorical sense for some unclear sociopolitical reason) despite current LLMs being able to perform bluffing and all kind of complex functionalities.

Purplekeyboard t1_j0a7dwb wrote on December 15, 2022 at 4:15 AM

> You can't really explain those phenomena without hypothesizing that LLMs model deeper relational principles underlying the statistics of the data -- which is not necessarily much different from "understanding". > > > > Sure, sure, it won't have the exact sensori-motor-affordance associations with language; and we have to go further for grounding; but I am not sure why we should be drawing a hard line to "understanding" because some of these things are missing.

AI language models have a large amount of information that is baked into them, but they clearly cannot understand any of it in the way that a person does.

You could create a fictional language, call it Mungo, and use an algorithm to churn out tens of thousands of nonsense words. Fritox, purdlip, orp, nunta, bip. Then write another highly complex algorithm to combine these nonsense words into text, and use it to churn out millions of pages of text of these nonsense words. You could make some words much more likely to appear than others, and give it hundreds of thousands of rules to follow regarding what words are likely to follow other words. (You'd want an algorithm to write all those rules as well)

Then take your millions of pages of text in Mungo and train GPT-3 on it. GPT-3 would learn Mungo well enough that it could then churn out large amounts of text that would be very similar to your text. It might reproduce your text so well that you couldn't tell the difference between your pages and the ones GPT-3 came up with.

But it would all be nonsense. And from the perspective of GPT-3, there would be little or no difference between what it was doing producing Mungo text and producing English text. It just knows that certain words tend to follow other words in a highly complex pattern.

So GPT-3 can define democracy, and it also can tell you that zorbot mo woosh woshony (a common phrase in Mongo), but these both mean exactly the same thing to GPT-3.

There is vast amounts of information baked into GPT-3 and other large language models, and you can call it "understanding" if you want, but there can't be anything there which actually understands the world. GPT-3 only knows the text world, it only knows what words tend to follow what other words.

Nameless1995 t1_j0a97yt wrote on December 15, 2022 at 4:31 AM

> But it would all be nonsense.

Modeling the data generating rules (even if arbitrarily created rules) and relations from data, seems to be close to "understanding". I don't know what would even count as a positive conception of understanding. In our case, the data that we recieve is not just generated by an arbitrarily created algorithm, but by the world - and so the models we create helps us orient better to the world and in that sense "more senseful", but at a functional level not necessarily fundamentally different.

More this applies to any "intelligent agent". If you feed it arbitrary procedurally generated data what it can "understand" will be restricted to that specific domain (and not reach the larger world).

> GPT-3 only knows the text world, it only knows what words tend to follow what other words.

One thing to note that text world is not just something that exists in the air, it is a part of the larget world and created by social interactions. In essence they are "offline" expert demonstrations in virtual worlds (forums, QA, reviews, critics etc.).

However, obviously, GPT3 cannot go beyond that, and cannot comprehend the multimodal associations (images, proprioception, bodily signals etc.) beyond text (it can still associate different sub-modalities within text like programs vs natural texts and so on), and whatever it "understands" would be far alien from what a human understands (having much limited text data, but much richer multimodally embodied data). But that doesn't mean it doesn't have any form of understanding (understood in a functionalist (multiply realizable) sense -- ignoring any matter about "phenomenal consciousness") at all; and moreover, none of these mean somehow "making likely prediction from statistics" is dichotomous with understanding.

Purplekeyboard t1_j0aayw0 wrote on December 15, 2022 at 4:47 AM

One thing that impresses me about GPT-3 (the best of the language models I've been able to use) is that it is functionally able to synthesize information it has about the world to produce conclusions that aren't in its training material.

I've used a chat bot prompt (and now ChatGPT) to have a conversation with GPT-3 regarding whether it is dangerous for a person to be upstairs in a house if there is a great white shark in the basement. GPT-3, speaking as a chat partner, told me that it is not dangerous because sharks can't climb stairs.

ChatGPT insisted that it was highly unlikely that a great white shark would be in a basement, and after I asked it what would happen if someone filled the basement with water and put the shark there, once again said that sharks lack the ability to move from the basement of a house to the upstairs.

This is not information that is in its training material, there are no conversations on the internet or anywhere about sharks being in basements or unable to climb stairs. This is a novel situation, one that has not been discussed anywhere likely before, and GPT-3 can take what it does know about sharks and use it to conclude that I am safe in the upstairs of my house from the shark in the basement.

So we've managed to create intelligence (text world intelligence) without awareness.

respeckKnuckles t1_j0ajork wrote on December 15, 2022 at 6:16 AM

> which actually understands the world.

Please define what it means to "actually understand" the world in an operationalizable, non-circular way.

Purplekeyboard t1_j0amzem wrote on December 15, 2022 at 6:54 AM

I'm referring to two things here. One is having an experience of understanding the world, which of course GPT-3 lacks as it is not having any experience at all. The other is the state of knowing that you know something and can analyze it, look at it from different angles, change your mind about it given new information, and so on.

You could have an AGI machine which had no actual experience, no qualia, nobody is really home, but still understand things as per my second definition above. Today's AI language models have lots of information contained within themselves, but they can only use this information to complete prompts, to add words to the end of a sequence of words you give them. They have no memory of what they've done, no ability to look at themselves, no viewpoints. There is understanding of the world contained within their model in a sense, but THEY don't understand anything, because there is no them at all, there is no operator there which can do anything but add more words to the end of the word chain.

respeckKnuckles t1_j0apq5l wrote on December 15, 2022 at 7:28 AM

I asked for an operationalizable, non-circular definition. These are neither.

> the state of knowing that you know something and can analyze it, look at it from different angles, change your mind about it given new information, and so on.

Can it be measured? Can it be detected in a measurable, objective way? How is this not simply circular: truly understanding is defined as truly knowing, and truly knowing is defined as truly understanding?

> Today's AI language models have lots of information contained within themselves, but they can only use this information to complete prompts, to add words to the end of a sequence of words you give them. They have no memory of what they've done, no ability to look at themselves, no viewpoints. There is understanding of the world contained within their model in a sense, but THEY don't understand anything, because there is no them at all, there is no operator there which can do anything but add more words to the end of the word chain.

This is the problem with the "argumentum ad qualia"; qualia is simply asserted as this non-measurable thing that "you just gotta feel, man", and then is supported by these assertions of what AI is not and never can be. And how do they back up those assertions? By saying it all reduces to qualia, of course. And they conveniently hide behind the non-falsifiable shell that their belief in qualia provides. It's exhausting.

Purplekeyboard t1_j0ar5ea wrote on December 15, 2022 at 7:46 AM

>Can it be measured? Can it be detected in a measurable, objective way?

Yes, we can measure whether someone (or some AI) knows things, can analyze them, take in new information about them, change their mind, and so on. We can observe them and put them in situations which would result in them doing those things and watch to see if they do them.

An AI language model sits there and does nothing until given some words, and then adds more words to the end of the first words which goes with them. This is very different from what an AGI would do, or what a person would do, and the difference is easily recognizable and measurable.

>This is the problem with the "argumentum ad qualia"; qualia is simply asserted as this non-measurable thing that "you just gotta feel, man", and then is supported by these assertions of what AI is not and never can be. And how do they back up those assertions? By saying it all reduces to qualia, of course. And they conveniently hide behind the non-falsifiable shell that their belief in qualia provides. It's exhausting.

I wasn't talking about qualia at all here. You misunderstand what I was saying. I was talking about the difference between an AGI and an AI language model. An AGI wouldn't need to have any qualia at all.

Forms_Deep t1_j0b2vw6 wrote on December 15, 2022 at 10:29 AM

Sorry to butt in, but I took your statement "Having an experience of understanding the world" as a reference to qualia also.

If it isn't, could you explain what you mean by "experience of understanding" and how it can be measured?

calciumcitrate t1_j0amrhw wrote on December 15, 2022 at 6:51 AM

But a model is just a model - it learns statistical correlations* within its training data. If you train it on nonsense, then it will learn nonsense patterns. If you train it on real text, it will learn patterns within that, but patterns within real text also correspond to patterns in the real world, albeit in way that's heavily biased toward text. If you fed a human nonsense sensory input since birth, they'd produce an "understanding" of that nonsense sensory data as well.

So, I don't think it makes sense to assign "understanding" based on the architecture as a model is a combination of both its architecture and the data you train it on. Rather, if you have a trained model that captures representations that are generalizable and representative of the real world, then I think it'd be reasonable to say that those representations are meaningful and that the model holds an understanding of the real world. So, the extent to which GPT-3 has an understanding of the real world is the extent to which the underlying representations learned from pure text data correspond to real world patterns.

* This isn't necessarily a direct reply to anything you said, but I feel like people use "correlations" as a way to discount the ability of statistical models to learn meaning. I think people used to say the same thing about models just being "function approximators." Correlations (and models) are just a mathematical lens with which to view the world: everything's a correlation -- it's the mechanism in the model that produces those correlations that's interesting.

Purplekeyboard t1_j0aorp8 wrote on December 15, 2022 at 7:16 AM

>Rather, if you have a trained model that captures representations that are generalizable and representative of the real world, then I think it'd be reasonable to say that those representations are meaningful and that the model holds an understanding of the real world. So, the extent to which GPT-3 has an understanding of the real world is the extent to which the underlying representations learned from pure text data correspond the real world patterns.

GPT-3 contains an understanding of the world, or at least the text world. So does Wikipedia, so does a dictionary. The contents of the dictionary are meaningful. But nobody would say that the dictionary understands the world.

I think that's the key point here. AI language models are text predictors which functionally contain a model of the world, they contain a vast amount of information, which can make them very good at writing text. But we want to make sure not to anthropomorphize them, which tends to happen when people use them as chatbots. In a chatbot conversation, you are not talking to anything like a conscious being, but instead to a character which the language model is creating.

By the way, minor point:

>If you fed a human nonsense sensory input since birth, they'd produce an "understanding" of that nonsense sensory data as well.

I think if you fed a human nonsense information since birth, the person would withdraw from everything and become catatonic. Bombarding them with random sensory experiences which didn't match their actions would result in them carrying out no actions at all.

calciumcitrate t1_j0asr1n wrote on December 15, 2022 at 8:07 AM

> GPT-3 contains an understanding of the world, or at least the text world. So does Wikipedia, so does a dictionary. The contents of the dictionary are meaningful. But nobody would say that the dictionary understands the world.

What differs GPT-3 from a database of text is that it seems like GPT-3 contains some representations of concepts that make sense outside of a text domain. It's that ability to create generalizable representations of concepts from sensory input that constitutes understanding.

> I think if you fed a human nonsense information since birth, the person would withdraw from everything and become catatonic. Bombarding them with random sensory experiences which didn't match their actions would result in them carrying out no actions at all.

Maybe my analogy wasn't clear. The point I was trying to make was that if your argument is:

GPT-3 holds no understanding because you can feed it data with patterns not representative of the world, and it'll learn those incorrect patterns.

Then my counter is:

People being fed incorrect data (i.e. incorrect sensory input) would also learn incorrect patterns. e.g. someone who feels cold things as hot and hot things as cold is being given incorrect sensory patterns (ones that aren't representative of real-world temperature), and forming an incorrect idea of what "hot" and "cold" things are as a result, i.e. not properly understanding the world.

My point being that it's the learned representations that determine understanding, not the architecture itself. Of course, if you gave a model completely random data with no correlations at all, then the model would not train either.

Anti-Queen_Elle t1_j13k3lb wrote on December 21, 2022 at 12:24 PM

And if you create 6 billion neural networks, all speaking Mungo, and they invent a space ship and fly to the moon, would you still readily call it nonsense?

Luftetari t1_j0am47x wrote on December 15, 2022 at 6:44 AM

GPT or any LLM is just another actor in the production and reproduction of human intelligence. Like a teacher at a school, who tries to explain relativity theory without understanding it fully. Just another node in the vast human network. It just happens to be evolving at faster rates and impacting a larger number of actors. From an actor-centered and action-centered perspective it is an actor that operates intentionally and understands (operates, reflects, produces) its position and assigned role in the network.

ktpr t1_j09064e wrote on December 14, 2022 at 10:51 PM

Just making sure here, this isn’t a published conference paper that went through peer review, correct?

bballerkt7 t1_j0921vx wrote on December 14, 2022 at 11:04 PM

Nope

leondz t1_j0a9rp8 wrote on December 15, 2022 at 4:36 AM

It's only just appeared on arXiv, so wouldn't expect this yet

antonivs t1_j0a7svs wrote on December 15, 2022 at 4:18 AM

The first rule of talking about large language models, is that as a large language model, I can't talk about large language models.

[deleted] t1_j0bb4pn wrote on December 15, 2022 at 12:11 PM

[deleted]

Anti-Queen_Elle t1_j13jv60 wrote on December 21, 2022 at 12:21 PM

As an attention network, I have a pretty shit attention span

SnowyNW t1_j0930dg wrote on December 14, 2022 at 11:10 PM

Lol after reading that I’m even more convinced that the holistic system with LLMs applied leads to emergent phenomena such as consciousness. This paper basically hypothesizes this as well. I think it had the opposite of the intended effect the OP had but the author is simply trying to make the distinction between human and machine “knowing” just to prove how gosh dang close we really are to showing what that difference really is, if there even is one…

versaceblues t1_j0aa5n2 wrote on December 15, 2022 at 4:39 AM

It seems like every argument against it is.

“Oh but it’s just doing statistical filtertring of patterns it’s been trained on. Which is different from the human brain”

But with no clear explanation of how the human brain is different, aside from “oh it’s more complex and does other things that humans do”

economy_programmer_ t1_j098tdu wrote on December 14, 2022 at 11:52 PM

Totally agree

VordeMan t1_j09ik5r wrote on December 15, 2022 at 1:05 AM

A lot of Murray's arguments break down completely when the LLM has been RLHF-ed, or otherwise finetuned (i.e., the case we care about), which is a bit shocking to me (did no one point this out?). I guess that's supposed to be the point of peer review :)

Given that fact, it's unclear to me how useful this paper is....

Nameless1995 t1_j09m8ir wrote on December 15, 2022 at 1:33 AM

Footnote 1 Page 2. It's a bit of a wishy washy statement with no clear point but he does mention RLHF.

HateRedditCantQuitit t1_j0c0og4 wrote on December 15, 2022 at 3:36 PM

This paper has some interesting points we might agree or disagree with, but the headline point seems important and much more universally agreeable:

We have to be much more precise in how we talk about these things.

For example this comment section is fully of people arguing whether current LLMs satisfy ill-defined criteria. It’s a waste of time because it’s just people talking past each other. To stop talking past each other, we should consider whether they satisfy precisely defined criteria.

evil0sheep t1_j0eif3i wrote on December 16, 2022 at 1:34 AM

When we make a student read a book we test whether they understand that book by having them write a report on it and reviewing whether that report makes sense. If the report make sense, and it seems they extracted the themes of the book correctly, then we assess that they understood the book. So if I feed an LLM a book and it can generate me a report about the book, and that report makes sense captures the themes of the book, why should I not assess that the LLM understood the book?

When I interview someone for a job I test their understanding of domain knowledge by asking them subtle and nuanced questions about the domain and assessing whether their responses capture the nuance of the domain and demonstrate understanding of it. If I can ask an LLM nuanced questions about a domain, and it can provide nuanced and articulate answers about the domain, why should I not assess that it understands the domain?

This whole "its just a statistical model bro, you're just anthropomorphizing it" thing is such a copout. 350GB of weights and biases is plenty of space to store knowledge about complex topics, its plenty of space to store real high level understanding of the complex, nuanced relationships between the concepts that the words represent. I don't think its smart because I can ask it to write me a story and then give it nuanced critical feedback on its story and it can rewrite the story in a way that incorporates the feedback. Like I don't know how you can see something like this and not think that it has some sort of like real understanding of the concepts that the language encodes. It seems bizarre to me

HateRedditCantQuitit t1_j0f3ilt wrote on December 16, 2022 at 4:18 AM

If you give me a precise enough definition of what you mean by ”understanding” we can talk, but otherwise we’re not discussing what gpt does, we’re just discussing how we think english ought to be used.

waffles2go2 t1_j09j6br wrote on December 15, 2022 at 1:10 AM

Oof, so LLMs use regression to figure out what's next.

If you bolt LLMs onto a system that can perform multi-step problem solving (the McGuffin of this paper) then you have a system that can "reason"....

Oof...

fooazma t1_j0acepl wrote on December 15, 2022 at 5:00 AM

Why a McGuffin? The lack of multi-step problem solving is clearly limiting. Examples of what's wrong with ChatGPT are almost always examples of the lack of few-step problem solving based on factual knowledge.

In an evolutionary competition between LLMs with this capability and those without, the former will wipe the floor with the latter. Shanahan, like all GOFAI people, understands this very well.

waffles2go2 t1_j0hslfb wrote on December 16, 2022 at 7:08 PM

Agree, it just lacks any nuance. "if you assume x" "then here is how you could use y"...

Also, confidentlyincorrect is pretty much every prediction in this rapidly-evolving space and if you're looking for business applications it's a cost/precision tradeoff where often the most advanced solutions lose..

rafgro t1_j0amjj9 wrote on December 15, 2022 at 6:49 AM

In case of such articles, I cannot escape the feeling that the authors do not interact with these models at length and mainly argue with their imagined form of interaction. Here, it is the premise of the significant part of the paper:

>a fictional question-answering system based on a large language model

...with imagined conversations and discussion of its imagined flaws, eg. the author criticizes it for lack of communicative intent, no awareness of the situation, no ability to "know anything", or that it "cannot participate fully in the human language game of truth" (self-citation a.d. 2010, in "Embodiment" presented as, roughly, everyday use of words and adjusting the use to the context). Thanks, I guess? How about interacting with actual models that beat you in the game of truth and are sometimes too nosy in their communicative intent?

timscarfe t1_j1jz3nx wrote on December 24, 2022 at 11:40 PM

I just interviewed Murray on MLST about this paper and his views on consciousness -- see https://www.youtube.com/watch?v=BqkWpP3uMMU

glichez t1_j08qfhh wrote on December 14, 2022 at 9:45 PM

its good to remember this. a lot of luddites have been sharpening their pitchforks lately...

jms4607 t1_j09860v wrote on December 14, 2022 at 11:47 PM

You could argue a LLM trained with RL like ChatGPT has intent in that is aware it is acting in an MDP and needs to take purposeful action.

ReginaldIII t1_j0b9rwb wrote on December 15, 2022 at 11:56 AM

RL is being used to apply weight updates during fine tuning. The resulting LLM is still just a static LLM with the same architecture.

It has no intent and has no awareness. It is just a model, being shown some prior, and being asked to sample the next token.

It is just an LLM. The method of fine tuning just creates a high quality looking LLM for the specific task of conversationally structured inputs and outputs.

You would never take your linear regression model that happens to perfectly fit the data, take a new prior of some X value, see that it gives a good Y value that makes sense, and come to the conclusion "Look my linear regression is really aware of the problem domain!"

Nope. Your linear regression model fit the data well, and you were able to sample something from it that was on the manifold the training data also lived on. That's all that's going on. Just in higher dimensions.

NotDoingResearch2 t1_j0d1dgb wrote on December 15, 2022 at 7:31 PM

I feel like our modern day education system has somehow made us unable to tell the difference between models and reality.

Hyper1on t1_j0e7dv0 wrote on December 16, 2022 at 12:11 AM

Look at Algorithm Distillation, you can clearly do RL in-context in LLMs. The point of this discussion is that "being asked to sample the next token" can, if sufficiently optimized, encompass a wide variety of behaviours and understanding of concepts, so saying that it's just a static LLM seems to be missing the point. And yes, it's just correlations all the way down. But why should this preclude understanding or awareness of the problem domain?

jms4607 t1_j0ckrj0 wrote on December 15, 2022 at 5:45 PM

Ur only able to sample something from the manifold you have been trained on.

ReginaldIII t1_j0cl6lg wrote on December 15, 2022 at 5:47 PM

That's not really true because because both under- and over-fitting can happen.

And it doesn't reinforce your assertion that ChatGPT has awareness or intent.

jms4607 t1_j0cqu0a wrote on December 15, 2022 at 6:23 PM

I’d argue that if ChatGPT was fine tuned in RL based off of the responses of a human, for example, if it’s goal as a debater ai was to make humans less confident of their belief by responding in contrary in a conversation, than it arguably has awareness of intent. Is this not possible in the training scheme of ChatGPT? I looked into how they use RL right now, and I agree it is just fine-tuning human-like responses, but I think a different reward function could illicit awareness of intent.

ReginaldIII t1_j0cuujj wrote on December 15, 2022 at 6:49 PM

It mimics statistical trends from the training data. It uses embeddings that make related semantics and concepts near to one another, and unrelated ones far from one another. Therefore, when it regurgitates structures and logical templates that were observed in the training data it is able to project other similar concepts and semantics into those structures, making them look convincingly like entirely novel and intentional responses.

jms4607 t1_j0cva57 wrote on December 15, 2022 at 6:52 PM

I don’t think we know enough about the human brain to say we aren’t doing something very similar ourselves. 90% at least of human brain development has been to optimize E[agents with my dna in future]. Our brains are basically embedding our sensory input into a compressed latent internal state, then sampling actions to optimize some objective.

ReginaldIII t1_j0cxciw wrote on December 15, 2022 at 7:05 PM

That we have the ability to project concepts into the scaffold of other concepts? Imagine a puppy wearing a sailor hat. Yup we definitely can do that.

f(x) = 2x

I can put x=1 in, I can put x=2 but if I don't put anything in then it just exists as a mathematical construct and it doesn't sit their pondering its own existence or the nature of what x even is. "I mean, why 2x ?!"

If I write an equation c(Φ,ω) =（Φ ω Φ）do you zoomorphise it because it looks like a cat?

What about this function which plots out Simba. Is it aware of how cute it is?

x(t) = ((-1/12 sin(3/2 - 49 t) - 1/4 sin(19/13 - 44 t) - 1/7 sin(37/25 - 39 t) - 3/10 sin(20/13 - 32 t) - 5/16 sin(23/15 - 27 t) - 1/7 sin(11/7 - 25 t) - 7/4 sin(14/9 - 18 t) - 5/3 sin(14/9 - 6 t) - 31/10 sin(11/7 - 3 t) - 39/4 sin(11/7 - t) + 6/5 sin(2 t + 47/10) + 34/11 sin(4 t + 19/12) + 83/10 sin(5 t + 19/12) + 13/3 sin(7 t + 19/12) + 94/13 sin(8 t + 8/5) + 19/8 sin(9 t + 19/12) + 9/10 sin(10 t + 61/13) + 13/6 sin(11 t + 13/8) + 23/9 sin(12 t + 33/7) + 2/9 sin(13 t + 37/8) + 4/9 sin(14 t + 19/11) + 37/16 sin(15 t + 8/5) + 7/9 sin(16 t + 5/3) + 2/11 sin(17 t + 47/10) + 3/4 sin(19 t + 5/3) + 1/20 sin(20 t + 24/11) + 11/10 sin(21 t + 21/13) + 1/5 sin(22 t + 22/13) + 2/11 sin(23 t + 11/7) + 3/11 sin(24 t + 22/13) + 1/9 sin(26 t + 17/9) + 1/63 sin(28 t + 43/13) + 3/10 sin(29 t + 23/14) + 1/45 sin(30 t + 45/23) + 1/7 sin(31 t + 5/3) + 3/7 sin(33 t + 5/3) + 1/23 sin(34 t + 9/2) + 1/6 sin(35 t + 8/5) + 1/7 sin(36 t + 7/4) + 1/10 sin(37 t + 8/5) + 1/6 sin(38 t + 16/9) + 1/28 sin(40 t + 4) + 1/41 sin(41 t + 31/7) + 1/37 sin(42 t + 25/6) + 3/14 sin(43 t + 12/7) + 2/7 sin(45 t + 22/13) + 1/9 sin(46 t + 17/10) + 1/26 sin(47 t + 12/7) + 1/23 sin(48 t + 58/13) - 55/4) θ(111 π - t) θ(t - 107 π) + (-1/5 sin(25/17 - 43 t) - 1/42 sin(1/38 - 41 t) - 1/9 sin(17/11 - 37 t) - 1/5 sin(4/3 - 25 t) - 10/9 sin(17/11 - 19 t) - 1/6 sin(20/19 - 17 t) - 161/17 sin(14/9 - 2 t) + 34/9 sin(t + 11/7) + 78/7 sin(3 t + 8/5) + 494/11 sin(4 t + 33/7) + 15/4 sin(5 t + 51/11) + 9/4 sin(6 t + 47/10) + 123/19 sin(7 t + 33/7) + 49/24 sin(8 t + 8/5) + 32/19 sin(9 t + 17/11) + 55/18 sin(10 t + 17/11) + 16/5 sin(11 t + 29/19) + 4 sin(12 t + 14/9) + 77/19 sin(13 t + 61/13) + 29/12 sin(14 t + 14/3) + 13/7 sin(15 t + 29/19) + 13/4 sin(16 t + 23/15) ...

jms4607 t1_j0d65c3 wrote on December 15, 2022 at 8:01 PM

Projecting can be interpolation, which these models are capable of. There are a handful of image/text models that can imagine/project an image of a puppy wearing a sailor hat.
All you need to do is have continuous sensory input in your RL environment/include cost or delay of thought in actions, which is something that has been implemented in research to resolve your f(x) = 2x issue.
The Cat example is only ridiculous because it obviously isn’t a cat. If we can’t reasonably prove that it is or isn’t a cat, then asking whether it is a cat or not is not a question worth considering. Similar idea goes for the question “is ChatGPT capturing some aspect of human cognition”. If we can’t prove that our brains work in a functionally different way that can’t be approximated to arbitrary degree by a ML model, then it isn’t something worth arguing ab. I don’t think we know enough ab neuroscience to state we aren’t just doing latent interpolation to optimize some objective.
The simba is only cute because you think it is cute. If we trained an accompanying text model for the simba function, where it was given the training data “you are cute” in different forms, it would probably respond yes if asked if it was cute. GPT-3 or ChatGPT can refer and make statements ab itself.

At least agree that evolution on earth and human actions are nothing but a MARL POMDP environment.

red75prime t1_j0i1n56 wrote on December 16, 2022 at 8:10 PM

> linear regression model

Where is that coming from? LLMs are not LRMs. LRM will not be able to learn theory of mind, which LLMs seem to be able to do. Can you guarantee that no modelling of intent is happening inside LLMs?

> Just in higher dimensions.

Haha. A picture is just a number, but in higher dimensions. And our world is just a point in enormously high-dimensional state space.

ReginaldIII t1_j0i67uc wrote on December 16, 2022 at 8:41 PM

Linear regression / logistic regression is all just curve fitting.

> A picture is just a number, but in higher dimensions.

Yes... It literally is. A 10x10 RGB 24bpp image is just a point in the 100 dimensional hypercube bounded by 0-255 with 256 discrete steps. In each 10x10 spatial location there are 256^3 == 2^24 possible colours, meaning there are 256^3^100 possible images in that entire domain. Any one image you can come up with or randomly generate is a unique point in that space.

I'm not sure what you are trying to argue...

When a GAN is trained to map between points on some input manifold (a 512 dimensional unit hypersphere) to points on some output manifold (natural looking images of cats embedded within the 256x256x3 dimensional space bounded between 0-255 and discretized into 256 distinct intensity values) then yes -- the GAN has mapped a projection from one high dimensional manifold to a point on another.

It is quite literally just a bijective function.

red75prime t1_j0i966c wrote on December 16, 2022 at 9:01 PM

"Just a" seems very misplaced when we are talking about not-linear transformations in million-dimensional spaces. Like arguing that an asteroid is just a big rock.

ReginaldIII t1_j0i9imv wrote on December 16, 2022 at 9:03 PM

That you have come to that conclusion is ultimately a failing of the primary education system.

Its late. Im tired. And I dont have to argue about this. Good night.

red75prime t1_j0iay49 wrote on December 16, 2022 at 9:13 PM

Good night. Happy multidimensional transformations that your brain will perform in sleep mode.

Comments