Interesting approach. I dug deeper into this. Other models are trained with 16-bit weights, and then shrunk (quantized) by throwing data away to reduce the weights to 8-bit, or 4-bit, or 2-bit, etc. Once you get below 4-bit, most models hallucinate and make mistakes to the point of being unreliable or even unusable. PrismML flips this and trains the model to 1-bit weights from the start, which does yield much better results. PrismML is due to release their 27B model shortly, which supposedly can run on an iPhone 17 Pro (I would read between the lines and assume that means it was still too big to run on an iPhone 15 Pro/iPhone 16). We will have to wait and see how good this new model is. If they can prove that their training methodology scales with larger models, this would be great news.
Running a model on your phone requires more than just the core model weights - it also needs a KV cache that tracks what questions/information you sent to the model. For a long conversation, the KV cache may require as much memory as the model weights themselves. Fortunately, companies like Google have developed algorithms (TurboQuant) to compress the KV cache. So PrismML + TurboQuant drastically reduce the memory required to run a model.
However, to make a proper comparison, you must realize that frontier models (ChatGPT, Claude) are absolutely massive, using over 1T parameters (estimated, as they don't actually publish this info). Current open source models with 35B to 120B parameters are comparable, but definitely not as good.
So the real question that hasn't been answered yet is that even with all of these improvements in memory usage, will the model that you can run on your phone be good enough? My honest guess is "no" until phones start shipping with 16 to 32 GB of RAM.
