Aren't the on device models quite limited in capabilities? What can they do? In any case even access to a limited model could be huge.
While Apple’s OpenELM LLM is no ChatGPT, it is a very capable resource for incorporating conversational interface, on-device RAG and LoRA fine-tuning into an app.
We used it to incorporate a fine-tuned model into one of our apps and was very pleased with the results. We held off on shipping the updated app because at the time it would require the app to download a 3.8GB fine-tuned OpenELM model from huggingface — which we didn’t want to require users to do.
We also tried implementing the same AI feature set by incorporating ChatGPT and Perplexity via API. It worked functionally but the latency and API costs were prohibitive for our use case.
Need to see the details but this announcement could possibly eliminate all of these problems — assuming the foundation models include OpenELM equivalents and the API supports fine-tuning via Adapters as was announced at last year’s WWDC. I can’t wait to see the details and start working with the beta!
Follow-up: After watching the WWDC Platforms State of Union keynote .. AFM Framework is a Bonafide Game Changer!
The AFM framework and APIs exceed my expectations by delivering utility that goes well beyond providing the desired functionality — which for me was simply (a) a capable on-device LLM that eliminates the need for costly, high-latency, off-board 3rd-party LLMs, (b) adapters to allow model fine-tuning, (c) user data privacy, and (d) off-line operation.
The unexpected benefits include tool calling, response streaming and model macros that eliminate complex and error-prone LLM response parsing and mapping to app data structures.
I took the last item (complex, imprecise and time-consuming parsing and mapping) as a given and something that developers should just expect to do when incorporating LLMs into an app with structured data — and was completely surprised to hear that Apple completely eliminated this issue for Apple platform developers! This is a really big deal because this single issue is actually a limiting factor for app development use cases. Prior to the Apple specialized macro utility the solution was either complex and brittle regular expressions that were guaranteed to fail (because LLM output is non-deterministic), or ballooning LLM API cost and latency to add guardrails to constrain LLM output to behave more deterministically.
The final word will depend on the stability of the AFM implementation and how well it aligns with what was demonstrated, but this developer is very pleased. The AFM API is a year late, but definitely way better than what was expected.
Bravo Apple. Thank you! 🙏🏽
