You are making some fair points, but I think your analogy breaks down a bit. Note that I'm not saying you're incorrect, I'm simply suggesting some things to think about.
I agree that current LLMs do not comprehend in the human sense. They do not have subjective awareness, lived experience, etc. They can produce language about pain, grief, or morality without experiencing any of those things as humans do. To be fair, people can do the same thing -- rationally talk about things they have not personally experienced, but such comments come from a place without phenomenological understanding. However, saying an LLM is “just finishing a story...based on statistical probability” already defines its output as non-comprehending before establishing what comprehension is. You are building your conclusion into your premise.
True, but it’s a forum, and for a technical forum, not that technical or formal. This is entertainment, not writing an abstract for a journal. Honestly, it would be a ware of time putting the effort into developing a formal argument on here. There are more appropriate places for that.
Also, distilling this down to an equivalence to someone singing English lyrics phonetically without understanding the words is going a little far in the reduction. The singer cannot flexibly explain the song, translate it, critique it, apply its themes to another context, or revise it for a different audience. LLMs can do many of those things. That does not prove consciousness or true understanding, but it does suggest something more complex than rote mimicry.
That’s closer to where I’m going, except saying “something more complex than rote mimicry” implies I think it is “just rote mimicry”. It is. Don’t underestimate the power of taking in large amounts of data, then using that data to develop both a reasoned output, but more importantly, developing a method of reasoning itself.
Reasoning does don’t develop in a vacuum and then is applied to data. Reasoning is developed through a huge amount of data input. The input precedes the development of reason8ng.
There’s the old trope of people talking about their education as a huge amount of information that was not in any way relevant to their adult life. That’s not completely true, but a lot of that educational input is to enable the development if critical thinking and reasoning.
Yes LLM based models can reason. I’m not underplaying that. But that reasoning is a result of inputting and training based on statically probabilities. Reason is the result of analysis Ing data, it isn’t separate to it.
Reason, human or otherwise, itself could be described as a „story completer”: examine data, deduce patterns, extrapolate. It is possible to use formal logical reasoning to reach a conclusion, while not necessarily comprehending the content of the problem.
Reason is not comprehension. When we think of „human intelligence”, it is not simply reasoning.
This will continue to be philosophical. The better distinction about what's happening may be between phenomenological understanding (going back to my second paragraph) and functional language-based modeling. Humans integrate language into lived experience, emotion, goals, social behavior, and self-awareness. LLMs do not (there's an assumption there, but it's a safe one at this stage of development). However, the LLMs can still model contextual semantics well enough to generate useful responses.
So I agree that we should not mistake fluent output for human comprehension, wisdom, or authority. But I would not reduce LLMs to mere karaoke. They are not minds as we typically think of minds, but they are also not simply just repeating things. They are sophisticated systems that can simulate aspects of reasoning through language, while lacking the consciousness that human understanding depends on (again, another assumption in there!).
Again, I’m NOT saying “mere karaoke”. In my anecdote, I wasn’t belittling those singers. Was it fake. No. Did they give excellent renditions of the songs. Yes so did they achieve their aim. Yes. But they achieved that aim through a process very different to the process people would except. They achieved that process without comprehension ( u derstanding of the greater context behind a query / instruction ).
The kanged used in the original reticle “honesty” is unhelpful. An AI simply gives the prompter what he/she asked for. If this means fabricating / hallucinating non-facts that appear convincing, then it has achieved its aim. In a sense, you can’t criticise it. It is not doing this to be edgy or lazy, it is doing what it was asked.
It’s not a person, it’s a nomreflexive genie. It fulfills requests, and it will fulfil that request with fabrication, if it needs to.
It’s not that the karaoke singers are lesser, it’s that a listener would assume they knew the some through comprehension, but this was not th case.
In the same way, people as making misassumptions about „intelligence” of AIs, because the output may have the appearance of comprehension (I.e. consciousness and self-awareness), when they AI has been able to price this output without any self-awareness or comprehension.
To try and sum it up, humans inevitably ask “cui bono?”, AI does not.
If is is “intelligence”, it is not analogous to “human intelligence”, even if it reaches the same conclusion as “human intelligence” in answer to a specific query.