https://en.m.wikipedia.org/wiki/Chinese_room

This is a problem or theory that addresses the issue at possibly better lengths.

Additionally, I believe recommend to most people who are interested in AI to read the Fourth Age, which is a philosophy book targeted at ai. It explains it in a nice and easier to read concept about what it is to be truly AGI and the steps we’ve made so far and will need to make.

Quick Edit: I also don’t think youre wrong, what this AI is saying it wouldn’t be able to explain but it’s learned to take the code behind it and spit out something akin to human language, no matter how garbled or incoherent that is to the machine behind it doesn’t care, as long as it suits it’s learning.