Humans integrate indirect context, LLMS are essentially "story finishers", finishing a story begun by the prompt, base on statistical probability. the AI gives the illusion of comprehending its own output, but it doesn't comprehend its own output. It;s simply telling you what it thinks you want to hear.
To put it another way. An AI chatbot is analogue to an actor play a doctor on a long-running TV show,. the actor can present themselves as a doctor, they can say lots of medical things, but they are not a doctor. They're repeating a script. The difference is that the AI can generate its own script by combining information it has sourced. But it does not inherently comprehend the content of that self-generated script.
Quite. few decades ago, I was in Morocco for a bit. There are no public bars in Morocco, but the workaround at the time was to have Karaoke clubs, which did sell alcohol. No-one went to these places to perform karaoke, so the Karaoke clubs fired professional singers, to give the impression that Karaoke, rather then just drinking alcohol, was happening.
The singers were really good, word-perfect for any classic song they sung. standard pop hits, classics, etc. Sounded great. Speaking with one of the singers afterwards, I suddenly realized that he couldn't speak a word of English, but could sing English language songs perfectly without understand what any of the words mean.
That's essentially AI. An appearance of comprehension.
You are making some fair points, but I think your analogy breaks down a bit. Note that I'm not saying you're incorrect, I'm simply suggesting some things to think about that might refine what you're stating. This response is long so feel free to skip.
I agree that current LLMs do not comprehend in the full human sense -- they do not have subjective awareness, lived experience, etc. However, saying an LLM is “just finishing a story...based on statistical probability” already defines its output as non-comprehending before establishing what comprehension is. You are building your conclusion into your premise: LLMs don't comprehend therefore what they do is not comprehension.
We know that LLMs can produce language about pain, grief, or morality without experiencing any of those things as humans do. However, to be fair, people can do the same thing -- rationally talk about things they have not personally experienced, but such comments come from a place without phenomenological understanding. In other words, we can have knowledge of things but not understanding of the experience of those things. In that sense there is comprehension but maybe no empathy and compassion. (As an aside, the topic of whether LLMs have emotions gets into some fascinating areas -- it depends on what emotions are and how they are experienced).
Also, distilling the functioning of LLMs down to an equivalence to someone singing English lyrics phonetically without understanding the words is going a little far in the reduction. The singer cannot flexibly explain the song, translate it, critique it, apply its themes to another context, or revise it for a different audience. LLMs can do many of those things. That does not prove consciousness or true understanding, but it does suggest something more complex than rote mimicry. If LLMs are
only mimicking, we can argue that is also true for people who are writing, speaking, or creating anything.
This will continue to be philosophical. The better distinction about what's happening may be between
phenomenological understanding (going back to my third paragraph) and
functional language-based modeling. Humans integrate language into lived experience, emotion, goals, social behavior, and self-awareness. LLMs do not (there's an assumption there, but it's a safe one at this stage of their development). However, the LLMs can still model contextual semantics well enough to generate useful responses. If they are merely simulating comprehension and intelligence are those simulations not real? If not, why? Those are rhetorical questions here.
So I agree that we should not mistake fluent output for human comprehension, wisdom, or authority. But I would not reduce LLMs to mere karaoke. They are not minds as we typically think of minds, but they are also not simply just repeating things. They are sophisticated systems that can simulate aspects of reasoning through language, while lacking the consciousness that human understanding depends on (again, another assumption in there!). That's part of the difficulty people have in deciding whether they are intelligent.
