I think Apple is being delusional at this point. It’s clear they don’t have the skills, the people or the vision to deliver something half as good as ChatGPT, let alone ‘on par’. Cook has to go, his leadership has been appalling.
A lot of the "Apple cannot make a modem" and "Apple cannot make a GPU" vibe here...
It's easy to make these sorts of strong claims when you know nothing about what's involved in making an LLM (or modem, or GPU) and nothing about what's going on inside Apple.
Meanwhile in the real world we have things like this
which has led to things like Mamba and similar linear or sub-linear attention schemes or attention replacements.
Apple looks like it's behind for the same reason they looked like they were behind in modem – they have somewhat different goals from the mainstream, and they won't release until those goals are met.
In the case of the modem, the goals were not just to implement the spec (like any other modem) but to hit certain power levels.
In the case of the LLM, the additional goals appear to include
- lower power (always ...)
- the usual security/privacy stuff
- a deep set of APIs that both expose the LLM and give the LLM agentic power
You can see this if you track the Apple papers. It's not that they are behind others, it's that they're continually looking at different
types of things from others. For example, if you quantize an LLM, sure performance goes down a little acrossa range of tasks; that's expected. But exactly WHERE does the performance go down? What's the TYPE of functionality that's hurt most by quantization? That's the sort of question Apple is investigating and that I don't see anyone else investigating.