Good post. I do have responses to a few of your points.

You argue that the systems we're building will fail to be genuine intelligences because, at bottom, they are blindly manipulating symbols without true understanding. That's a good objection, just as valid in the ChatGPT era as it was when John Searle presented it as a thought experiment that has become known as "The Chinese Room argument":

> Searle imagines himself alone in a room following a computer program for responding to Chinese characters slipped under the door. Searle understands nothing of Chinese, and yet, by following the program for manipulating symbols and numerals just as a computer does, he sends appropriate strings of Chinese characters back out under the door, and this leads those outside to mistakenly suppose there is a Chinese speaker in the room.

There's plenty of evidence to show that modern "AIs," which are just language models, are essentially the same as Searle's box (worse, even, because their instructions are noticeably imperfect). So, I think you're on solid ground to say that ChatGPT and other language models are not real intelligences, and furthermore that nothing which is just a language model could ever qualify.

But it's one thing to say "a language model will never achieve understanding," and quite another to say "it is impossible to create an artificial construct which has real understanding." And you do make that second, stronger claim.

Your argument is that the foundation that works for humans is not available to computers. I think the story you tell here is problematic.

You talk a bit about the detailed chain of physical processes that occur as sensory input reaches the human body, travels through the perceptual apparatus, and ultimately modifies the physical structure of the brain.

But, computers also undergo complex physical processes when stimulated, so "having a complex process occur" is not a categorical differentiator between humans and computers. I suspect that the processes which occur in humans are currently much more complex than those in computers, but we can and will be making our computers more complex, and presumably we will not stop until we succeed.

And, notably, a lot of the story you tell about physical processes is irrelevant.

What happens in my mind when I see something has very little to do with the rods and cones in my eyes, which is plain when we consider any of these things:

• When I think about something I saw earlier, that process of reflection does not involve my eyeballs.
• Color-blind people can learn, understand, and think about all the same things as someone with color-vision.
• A person with normal sight who becomes blind later does not lose all their visual memories, the knowledge derived from those memories, or their ability to reflect on those things.

Knowledge and understanding occur in the brain and not in the perceptual apparatus. (I don't know much about muscle memory, but I'd wager that the hand muscles of a practiced pianist don't play a real part in understanding Rachmaninoff work. If any real pianists disagree on that point, PM me with your thoughts.)

So, turning our attention to just what happens in the brain, you say:

> The physical activity and shifting state ARE the result, no further interpretation necessary

I get what you're saying here: the adjustment that occurs within the physical brain is the learning. But you're overlooking the fact that this adjustment is itself an encoding of information, and is not the information itself.

It's important to note that there is no resemblance between the physical state of the brain and the knowledge content of the mind. This is a pretty famous topic in philosophy, where it's known as "the mind-body problem."

To put it crudely: we are quite certain that the mind depends on the brain, and so doing stuff to the brain will have effects on the mind, but we also know from experiment that the brain doesn't "hold" information the way a backpack "holds" books. The connection is not straightforward enough that we can inspect the content of a mind by inspecting the brain.

I understand the word "horse." But if you cut my brain open, you would not find a picture of a horse, or the word "horse" written on my gray matter. We can't "teach" somebody my email password by using surgery to reshape their brain like mine.

And that cuts both ways: when I think about horses, I have no access to whatever physical brain state underlies my understanding. In fact, since there aren't any nerve endings in the brain, and my brain is encased in my skull (which I have not sawed open), I have no direct access to my brain at all, despite being quite aware of at least some of the content of my mind.

So, yes, granted: AI based on real-world computing hardware would have to store information in a way that doesn't resemble the actual knowledge, but so do our brains. And not only is there no reason to suppose that intelligence resides in just one particular encoding mechanism, even if it did, there's no reason to suppose that we couldn't construct a "brain" device that uses that same special encoding: an organic brain-thing, but with five lobes, arranged differently to suit our purposes.

The underpinnings you highlight are also problematic.

I think this quote is representative:

> The base case, the MEANING comes from visceral experience.

One real objection to this is that lots of learning is not visceral at all. For example: I understand the term "genocide," but not because I experienced it first-hand.

Another objection is that the viscera of many learning experiences are essentially indistinguishable from each other. As an example: I learned different stuff in my Philosophy of Art class than I learned in my Classical Philosophy class, but the viscera of both consisted of listening to the exact same instructor lecturing, pointing at slides that were visually all-but-identical from each other, and reading texts that were printed on paper of the same color and with the same typeface, all in the exact same classroom.

If the viscera were the knowledge, then because the information in these two classes was so different, I would expect there to be at least some perceptible difference in the viscera.

And, a Spanish student who took the same class in Spain would gain the same understanding as I did, even though the specific sounds and slides and texts were different.

I think all of this undermines the argument that knowledge or understanding are inextricably bound up in the specifics of the sensory experience or the resulting chain reaction of microscopic events that occurs within an intelligent creature.

TO BE CONTINUED...