No. A TPUv4 rack requires more water flow than a fire truck pump can deliver to keep the system cool enough. A TPUv5 rack likely requires as much or more power and water cooling. These are racks and racks of massively hot supercomputer chips that Apple is using for ML training. The electric power bill for training just one of the largest LLM models can run into many millions of (USD) dollars.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Apple Used Google Tensor Chips to Develop Apple Intelligence
- Thread starter MacRumors
- Start date
- Sort by reaction score
No. A TPUv4 rack requires more water flow than a fire truck pump can deliver to keep the system cool enough. A TPUv5 rack likely requires as much or more power and water cooling. These are racks and racks of massively hot supercomputer chips that Apple is using for ML training. The electric power bill for training just one of the largest LLM models can run into many millions of (USD) dollars.
It matters because there is a time component. The competition isn't standing still. We are no longer impressed with GPT-3.5. More performance means less training time, faster inference, better tuning, and ultimately a more refined model.
No, it's not like making the "best music" at all.
It's about building a better LLM, computer vision, image generation, decision tree model, etc.
If performance didn't matter, Google wouldn't be buying Nvidia Blackwell clusters. Microsoft and Elon Musk wouldn't be building 100,000 GPU clusters.
Do you think companies are idiots for paying $40,000 for a single H200 GPU when they could settle for Google? It's irrelevant what hardware they use, right?
This is true for frontier ML models, those that are racing OpenAI to get to the edge of what's possible first. Those models have on the order of a trillion or more parameters. That size requires the latest exotic hardware.
However the Apple paper is about creating cost effective models the use "merely" billions of parameters, so that they can be run on device for privacy, or profitable in the cloud. That size of ML model training does not require the latest exotic hardware.
Maybe you should read about the limitations of the Tensor chip vs Qualcomm Snapdragon 8gen 2/3 & vs Samsungs Exynos.
Do youevenrealizeSamsu ghas theie on language models/LLMs? Hmm
Where do you think those processing units are within?They are using Tensor Processing Units (below), not the mobile SoC named Tensor.
The SoCs youlitersry just pictured 4 of them!
What AMD hardware was used in Mac's?
The bad blood must've happened after the 2008-2010 MacBook ans MBPs that used NVidia Southbridge hardware which both Apple & NVidia collaborated on. As I recall that was a great partnership that leap-frogged over the Intel and AMD laptops back thenbut not for long
Misleading headline.
How the hell does the writer not know the difference between the Tensor SoC and Google TPU (custom server hardware) !? Those are two VERY DIFFERENT things.
There are so many comments talking about Google Tensor SoC.. sigh! Why are people so easily misled?
How the hell does the writer not know the difference between the Tensor SoC and Google TPU (custom server hardware) !? Those are two VERY DIFFERENT things.
There are so many comments talking about Google Tensor SoC.. sigh! Why are people so easily misled?
EXACTLY THIS.They are using Tensor Processing Units (below), not the mobile SoC named Tensor.
Misleading headline, followed by misinformed comments.
Which had nothing to do with TPUv5, which are much larger high power water cooled server chips for ML mainframes.
These Google TPU chips don't just use wires, but use laser optical circuit switching to interconnect with each other. Much more exotic than any Apple M series chip interconnect.
Last edited:
But Apple doesn't need the best and biggest LLM model, they're not competing with ChatGPT 5 or Claude 4 etc.
They've specifically built small models that run on device needing 2gb of ram so as to not need the cloud, keep privacy and work with zero signal. They don't need more powerful processes to speed up training as they've already created the LLMs months ago. They're now working on ensuring it works everywhere in the system it needs to. It doesn't and isn't designed to know everything that man has ever done like a general LLM - Claude has roughly 1 trillion parameters, which translates to approximately 400 GB of data. That's obviously designed to run in the cloud not on your local device. It's just supposed to understand natural language and reply with natural language whilst processing what it needs to on device - hence why it uses 3rd party massive models for general knowledge questions.
AMD were the graphics option for over a decade in all Macs until Apple Silicon came along.
Why are people suddenly so confidently wrong?
Apple’s small on-device machine learning cores are designed for running ML projects on the device itself, such as removing the background from an image, identifying faces in photos, recognising voices, etc.
They are NOT intended for creating large language models, which require entire data centres’ worth of processing power and millions of pounds in electricity to learn billions of parameters. Once the LLM model is created on this hardware, THEN it can be put on the device to run. We expect Apple to have a small LLM that runs with a billion parameters, is less than 2GB in size, and works for all on-phone actions without needing a phone signal or data connection. This is exactly why it will need an internet connection to connect services like ChatGPT for general knowledge questions, as these require a database in the region of 400GB with a trillion parameters, trained on everything ever written by humans (again with enormous amounts of server hardware costing hundreds of millions of pounds).
Training small ML models on M series chips is possible (I use my M1 MacBook for student level ML training). But the largest ML models are many thousands of times too big to train on an M3 Ultra. Consumer M series chips aren't designed for projects requiring data centers full of water cooled supercomputer size systems.
There is actually a subsection of the field that is devoted to producing generative training data. So in theory, you could train on both types of responses (not that I think this is happening here, I don't).
Bingo. Apple did not choose an "inferior" AI solution. They almost assuredly chose the best one for their purposes.
Nvidia is simply not cost effective, and their GPUs, while serendipitously well suited for AI type computations, were not designed specifically for AI calculations, but for graphics output, and their advantage is and will be whittled down by competing products from AMD, Google, Meta, and perhaps soon Apple.
When you need to perform X number of computations in Y amount of time at a cost of Z, there are many ways to get there. Just because one Nvidia chip performs double the computations of a Google TPU chip, doesn't make Nvidia the superior choice. Particularly if it costs more than twice the Google chip or burns more than twice its power. Cost, efficiency, and rent vs own analysis comes into play.
As others have said here, if Apple is developing its own chips to use in the future, it could be cheaper to rent an interim solution than to purchase a bunch of expensive Nvidia chips that will likely have a low resale value in a few years as more competitor products are released.
All this armchair quarterbacking... I agree Apple is playing catchup here and wasted time on cars and subscription services and goggles but its decision with the TPUs makes sense. The big thing is going to be whether its AI product is a success or a failure. And whether it is able to create *functional* and *beneficial* uses for AI. To date, I really have no use for AI.
Number of parameters for som LLMs:
GPT-2: 1.5 billions
Apple Intelligence on-device: 3 billions
Apple Intelligence server: >3 billions
Apple Intelligence on-device before pruning: 6.4 billions
GPT-3: 175 billions
GPT 4: 1,760 billions*
*OpenAI hasn't released info about how big GPT-4 is
When it comes to compute, Apple is more like Driving Miss Daisy. Apple doesn't have a time problem when it comes to the language models being used for Apple Intelligence.
Both Facebook and Google has said they might have over invested in AI: https://www.cnbc.com/2024/07/25/techs-splurge-on-ai-chips-has-meta-alphabet-tesla-in-arms-race.html
Microsoft, Google and Amazon also runs datacenters and are renting out compute to customers. They aren't buying this hardware only for their own internal usage. In fact most of Microsofts investment in AI is based on demand from their customers.
When it comes to Elon Musk, he is late to the game, so he has a time issue. FSD is late, Grok is lagging behind OpenAI and Google for general LLMs.
Last edited:
Google uses 'tensor' for two types of hardware:
Google Tensor for mobile hone: https://en.wikipedia.org/wiki/Google_Tensor
Tensor Processing Unit: https://en.wikipedia.org/wiki/Tensor_Processing_Unit
They are very different.
Just show me anything that says Google does not have visibility. Have you read the terms for any modern cloud agreement. If not, I suggest that you do. Since the computing power is Google's, they have authority to do whatever they want behind the scenes. And with Google's moral record, I'll bet they what is best for Google, not Apple.
Apple is publishing what they are doing with these ML training runs, so everybody, including you, can see what Apple is doing on Google's TPUs.
The results are also likely targeted to Apple's specific NPU hardware, so likely only useful for apps that run on Apple products.
It's not cloud computing in the traditional sense though. You're not running a web server on there, they're just renting the processing power in a cluster.
Regardless even Google's standard cloud computing terms don't allow them to see what you're doing. You don't think Apple has negotiated an even stricter deal for their own work. They're only training an LLM anyway - an LLM that is then made open source, it's not like Apple has got anything to hide or even needs to - Google knows how to train LLMs - customer data isn't going anywhere near these CPUs.
Really, you think Google, with its amoral management will live up to contract terms. You have much more faith than me.
