Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Apple Study Reveals Critical Flaws in AI's Logical Reasoning Abilities
- Thread starter MacRumors
- Start date
- Sort by reaction score
The general model design is fundamentally sound but to get the level of logical reasoning possible in a human being you’d need a much higher resolution pattern recognition and preferably based on much smaller portions of quantized data. That’s generally the distinction…. Not accounting for potential quantum computing that may occur in carbon nano tubules in the brain. For that we simply cannot do with anything less than quantum computing on silicon. Explaining that would require a bit more text. LOL
Some assumptions have to be made. There is a duty on the questioner.
Specifics, Bob.
Turns out emulating the human brain is more complicated than originally thought. Who’d’ve thunk it.
If you take the time to give me details about the size of some kiwis, I might also discard them from the total. And the reason for this is that I’m reasoning!
I’d be really curious to see what the model would reply if it had been asked why it subtracted them from the total.
Last edited:
Now ask it "are you sure about this answer?" and it will likely change it's response. That's what it often does for me.
That's exactly what I was referencing. If you've ever played with that program or one similar and you read the transcript in the NY Times article from a couple of years ago, you'd see the latter as an updated version of the former. The only ghost in the machine is the person who wrote the program that sets the bot's parameters.
The innovations are the sheer size of the source database and the ability to receive queries and respond to them using realistic language. The rest is a search engine and the use of probability calculations to generate a seemingly plausible response to the query.
I think you are missing the thing here.
The models produced a correct answer through statistical means. They did not solve it.
I mean... none of this is news. LLMs have always been and will always be highly sophisticated, power-intensive autocompletes. The only people who think otherwise are the uninformed and the people who stand to gain from duping the public.
Lying to you? No it’s not. That suggests intent. It has no intent other than to respond to your prompt. It might be wrong. It might hallucinate. It might make connections that aren’t there or conflate two concepts with similar words. But it’s not lying. It can’t.
I asked ChatGPT...
"If an AI model removed the 5 smaller kiwis from the total, it might have misinterpreted the phrase "five of them were a bit smaller than average" as a condition to exclude them. The model could have inferred that since the kiwis were described differently, they should be treated separately or excluded, even though the problem doesn’t explicitly state that the smaller kiwis should be removed from the total count."
Exactly what I could have done!
Last edited:
I think the best approach to start reasoning about this about is with Arthur C Clarke: "any sufficiently advanced technology is indistinguishable from magic". That is not adequate for explaining human behaviour so I pose an alternative quote: "any poorly educated society is incapable of reasoning about the claims that arrive with technology".
Alas through some irony, people who believe the claims probably can be replaced by this technology.
What a quagmire! Perhaps we should use it to generate content for education so we can improve society to a point where it doesn't think this is credible magic. Oh no, wait, it fails at that too! This fell out the other day...
The clever people rode NVDA up to the sky, laughing at the investors who are going to pay for their gains later.
Alas through some irony, people who believe the claims probably can be replaced by this technology.
What a quagmire! Perhaps we should use it to generate content for education so we can improve society to a point where it doesn't think this is credible magic. Oh no, wait, it fails at that too! This fell out the other day...
The clever people rode NVDA up to the sky, laughing at the investors who are going to pay for their gains later.
I guess that implies something about the true utility of “executive team”
Obviously, in this case ChatGPT used better reasoning than you. By default, everyone uses base 10. If you want to get correct unswer, you need to use correct (properly specified) prompt which you did not.
OMG, Apple!!! Research? Study?
I would consider this common knowledge. There is no reasoning in LLMs - the latest ChatGpt o1-preview tries to work in this direction. But an LLM constructs sentences from statistical data (very rough explanation). Neither does an AI/LLM understand you, nor does it think about what you said - it just generates text. Btw an LLM doesn‘t wake up in the middle of the night, repeating the questions and answers of the day, asking itself if the given answers were correct.
With the insane amount of training data the LLMs create surprisingly correct output. But depending on the model, no one guarantees that the generated stuff is correct.
Btw - if you ask a human, the output can be garbage as well.
I would consider this common knowledge. There is no reasoning in LLMs - the latest ChatGpt o1-preview tries to work in this direction. But an LLM constructs sentences from statistical data (very rough explanation). Neither does an AI/LLM understand you, nor does it think about what you said - it just generates text. Btw an LLM doesn‘t wake up in the middle of the night, repeating the questions and answers of the day, asking itself if the given answers were correct.
With the insane amount of training data the LLMs create surprisingly correct output. But depending on the model, no one guarantees that the generated stuff is correct.
Btw - if you ask a human, the output can be garbage as well.
I also asked it 'When does 9x6 = 42' - it's reply started "Nine times six equals fifty-four, not forty-two"
Not a good answer... Correct would be in Base 13
It then referenced the Hitchhikers Guide, so it clearly knows much more about literature than arithmetic...
Apple is bringing this to your attention to lower expectations for these models, but to also mention the solution. Neural network integration. Guess who has 2 billion devices powered by neural networks? Hint hint, they will have an event talking about it in June of next year and it ain’t Google.
I feel like Apple just threw together Apple Intelligence as they needed to say they were competing. iPhones still don’t have the whole feature they were supposedly “designed for Apple Intelligence.” Marketing at its worst.
Reality is Apple should have been on top of AI for years, should have more advanced models rather than outsource the data to ChatGPT. Who actually believes Apple cares about your privacy can realize they don’t when using third parties for screening your calls and AI.
Maybe if they hadn’t been spending Billions on cars and Vision Pros they could actually compete with their own solution.
Reality is Apple should have been on top of AI for years, should have more advanced models rather than outsource the data to ChatGPT. Who actually believes Apple cares about your privacy can realize they don’t when using third parties for screening your calls and AI.
Maybe if they hadn’t been spending Billions on cars and Vision Pros they could actually compete with their own solution.
Just for money, that’s all. AI has been hyped as the next big thing by tech corporations. We’ve been “educated” that without AI we will fall behind. The push is on for everybody to use AI and it means more profit. Just look at how Apple was vilified and dismissed for not deploying AI immediately with every other tech Tom/Dick/Harry who jumped on the band wagon. The technology industry is always on the hunt for the next big thing to keep the coin rolling in.
Last edited:
To be fair, Siri has no reasoning abilities whatsoever...
