Let's keep the conversation straight. Someone was talking about the Mac Studio for LLM work, and you replied to them saying that the Mac Pro can have up to 1.5TB of user upgradeable RAM.
Agree I was thinking after I last posted that I had gone a bit off thread. I get a little frustrated that everything is seen through the lens of AI and nothing else matters. The Mac Pro -- and its capabilities -- were suited to high-performance and similar computing needs that are still important even if not AI. However, this thread started as a lament that Apple is not aligned to the AI boon so let's stay focused on that.
They don't offer Studios with 512GB unified memory because of supply issues. But I'll still bite.
I don't think an old Mac Pro with 1.5TB memory is going to be better for an AI than a 512GB Mac Studio, even if it has 3 times as much memory.
Once the VRAM need is met then of course more VRAM is not much more help. Apple's 512GB Studio provided cost effective access to run all but the largest flagship models, which was great.
The issue as noted is that the 512GB is no longer available and the Apple solution is a sole-source solution. Ditto now for 256GB. Or even 128GB unless you want a MacBook Pro. And then if you can fit within 96GB, it's not even an M4 but an M3. Not a bad processor by any standard but of course 2 generations old.
Apple's current approach is cost-effective at the low-end but it boxes one in. And in this case it boxes Apple in as seen by their need to drop all their large RAM configurations.
And there is no competing solution that has user replaceable RAM in the first place where that RAM is going to meaningfully impact AI. AND as was pointed out, GPUs with similar levels of memory are in the tens of thousands of dollars anyway.
Can we even have a serious discussion here? If you truly have a pressing need to run AI models, you are not using a Mac Pro or any other system and adding 1.5TB of RAM on to a motherboard to do it. Why do we have to talk around in circles?
The point of bringing up the Mac Pro is not that people should go back to it but rather it can do things that current systems can't. Which is not great when said system is 7 years old. Again not that the current Apple designs don't have advantages but we also have to admit the trade-offs. The Mac Pro's flexibility though it has costs gives it advantages over current Apple options for many situations.
For models that fit within ~ 90GB VRAM, Apple has competitive options but doesn't cover a lot of price/performance tradeoffs within that range and then that's it. An M4 Max with 128GB might have been faster than an AMD AI Max 395+ with 128GB but it's no longer available in the Studio. The closest in the Studio is the 96GB and that's with the M3 Ultra -- which definitely can't handle quite as big models and is it even faster than the AMD?
Then for smaller models there's a range of options for tiny/small (e.g. within 8-12GB VRAM) and medium (e.g. up to 2x24GB) models that are probably faster than Apple's though probably more expensive as a solution.
And then once you want to go to giant models, maybe you can cluster 4xStudio with 96GB of RAM but that's it. And it's a Thunderbolt 5 interconnect -- good but two generations behind current PCIe.
Whereas in the Nvidia/AMD/etc realm, there's options for multiple GPU (e.g. sharing over NVLink), offloading to RAM, Infiniband, etc.
The original post I replied to was highlighting how Apple's unified solution is untouchable and then clustering over Thunderbolt via RDMA takes it further. But clustering via RDMA undermines unified memory, and inherently slower and higher latency than a direct PCIe 6.0 x16 bus. Not to mention other options like NVLink or Infiniband interconnects which just aren't an option due to Apple's design choices.
Then for people where Apple is even an option (e.g. they don't inherently need CUDA and their models fit on the Studio), the ceilings are relatively low and hard.
