Always available, always private is a good feature. Also, that is free. Intelligence would not be comparable to frontier, but if Siri can call to their smarter colleague in cloud, that might solve some of it.
Pivotal question: How smart real time voice model one can actually host on a phone? If we can distillate GPT-4o level model there, this might be ok enough strategy. But that is a big if.
Trouble is that smartest models are multimodal and real time. One cannot match that experience by having a dumber model that needs to say "I do not know about that, just a bit, I'll call my colleague to check if he knows".
Run Llama, Gemma, Qwen, DeepSeek, and more on your iPhone, iPad, and Mac. Optimized for Apple Silicon. Offline. Private.
locallyai.app
I have it hooked up to the action button on my iPhone, so I press that button (kinda the equivalent of pressing power to call up Siri) and LocallyAI pops up.
Model sizes tend to be around 4B. Bonsai is 8B, but that's a ternary model.
Multimodality exists, in the sense that a few of the models can "see/analyze" images you give them. (AFM cannot, even though it's supposedly a VLM model, ie a combined vision/language model; this is probably a limitation of the iOS26 APIs and I expect it to be fixed as of WWDC)
LocallyAI comes with its own speech recognizer which, honestly, I find substantially inferior to Apple's keyboard-built-in speech recognition. So that kinda sorta gets you some multi-modality.
What's very much missing (and this goes for pretty much every model everywhere) is a harness that understands that you're on a device with a screen, and that you're a human with a face and hands. What I want from multimodality is things like being able to say "You see that PDF I have open with Thai writing, can you translate it to English". That's not a "model capability" issue, it's a UI issue. Similarly while talking to the model I want to be able to open the camera and say something like "What's that building I'm pointing to"; again not so much a tech issue as a "putting the pieces together" issue.
My GUESS is that Apple understands this, and this is the sort wrapper "around" the model that we'll get in OS27 (well, we'll get the first part of it; like building the GUI in the 1980s, or building iOS in the late 2000s, for a few years each year will deliver a big new set of important features).
To see how limited things are (and most of this is not malice, it's just everything takes time and evolution) for example right now you can get Grok (or a few other large LLMs) as a CarPlay app. This can be very useful BUT there's apparently no entitlement available to allow access to GPS. So you can't make a query like "Hey Grok, tell me about this town I'm about to drive through" which could be really helpful! This is just oversight, and the sort of thing that tells you nothing about limitations, just that it takes a few years to polish anything!
As for
1. speed, I find that on my iPhone 15 Pro Max all the local LLMs generate at slower than I can read, but not unbearably slower. Prefill is also not a terrible wait, a few seconds or so. Generally the actual answer I want is within the first few sentences, then the LLM goes off on its usual trying to be helpful blather giving you more and more, so I just kill it at that point.
For comparison:
- Grok, which is obviously, like all the hyperscalers, serving many queries at once, does not generate a response much faster. CarPlay Grok, which "talks" its response is, IMHO, horribly slow; all the voices irritate the heck out of me because they are so slow. Presumably this will improve with time, but honestly I find a locallyAI response to be generated at about the same speed, but with less irritation (because of the difference in scanning text vs waiting for slow speech).
- at the other end, Google in basic "giving a pseudo-AI response on the google request web page" mode is absurdly fast. That's obviously some kinda hybrid, if you go into legit AI mode you drop down to Grok-like speeds. It's obviously where we'd all like to be, but no-one gets that yet [unless you have your own personal Cerebras maybe?!]
2. accuracy. CURRENT small models intrinsically hallucinate more, it goes with the territory. I don't think that's inevitable; my understanding is there are schemes for the net, as it generates a response, to track a "confidence level", and if those sorts of ideas work out, we could have a small model switch to RAG at that point rather than hallucinate a response. I've found that every "factual type" question I asked got a correct response (usually just a factoid I'd forgotten and couldn't immediately retrieve -- one of the hells of growing older :-( ) except for one technical question where all the small models clearly went off the rails. (It's unsurprising a small model wouldn't know a technical chemistry fact; problem, like I said, is that *current* models don't know that they don't know.)
A second element of accuracy is something like a translation task, and here I was very impressed. All the models did a much better job of translating technical Chinese than did both Apple's built-in translation and Google translate. Of course they were also a lot slower! Apple/Google translate are instantaneous, a local model takes 30 sec or a minute depending on the length of the passage. Essentially Apple/Google translate is like a 12 yr old, knows something about the languages but no domain expertise. The LLMs were like a somewhat knowledgeable 18-yr old. By comparison Claude was close to a student domain expert, not an actual chip designer, but say a 2nd year EE student.
So one way, for example, that local LLMs might sneak into our daily use is as translation apps move to more and more of an LLM-like design. (I assume both Apple and Google translate are built around something like BERT, but BERT is what, 2017 technology? So if they move to something like 2023 technology that would be nice bump in capabilities.)
