New Mac Studio Can Be Clustered Together for Faster AI Performance

[pprior]

Speed is not the only thing that matters. Try running AI image or video models in ComfyUI on Mac and you will quickly find that they either don't run at all, they run on the CPU only, or they run on the GPU but an order of magnitude slower than what you'd expect simply because everything is optimized for CUDA and nothing is optimized for Mac. So you can chain any number of Mac Studios together and you still won't get anywhere near CUDA because the issue is not speed or performance, it's the ecosystem that nVidia has built out. The 4090 may be much slower on paper but it will actually run all the image models and it will do so pretty fast as opposed to the Mac. Every week there are optimizations coming out for CUDA, making generation twice as fast, 10 times as fast, etc... While the top end Macs are still far slower than even the slowest nVidia GPU – and this has nothing to do with RAM, power, or the GPU itself.

@baryon Is this really true? I just bought a studio with one plan being to learn comfyui. My daughter works in fashion and I wanted to explore and learn fashion mood boarding using this tool.

I thought they had native Mac silicone support. Really don’t want to have to build a pc just for cuda.

 

[BeatCrazy]

@baryon Is this really true? I just bought a studio with one plan being to learn comfyui. My daughter works in fashion and I wanted to explore and learn fashion mood boarding using this tool.

I thought they had native Mac silicone support. Really don’t want to have to build a pc just for cuda.

You need Windows/NVIDIA if you want to mess around with ComfyUI

 

[DelayedGratificationGene]

But who can afford more than one???

Enterprise

 

[deconstruct60]

@baryon Is this really true? I just bought a studio with one plan being to learn comfyui. My daughter works in fashion and I wanted to explore and learn fashion mood boarding using this tool.

I thought they had native Mac silicone support. Really don’t want to have to build a pc just for cuda.

There are two aspects of ComfyUI. One is the basic core tool. The other is the aspect where this 'open source project with limited funds' is trying to be everything for everybody by promoting the tool as the core of a ecosytem of 100's of other developers asynchronusly contributing things that just 'plug into' this tool.

So it really matters what 'learn comfyui' actually means. If it is explore every random tool that could possibly plug into the ecosystem, then Nvidia is a safer bet. If there is some specific tool want to explore as a plug-in then check to see if it is locally run or cloud run. If cloud run than the client system doesn't matter as much. If it is some it is some open source contribution were the creator to of the plug-in isn't going to put effort into natively building it for macOS then that is a problem. Technically it isn't comfy it is the herd of 3rd parties.

Comfy themselves doesn't seem to be taking approach themselves to get max performance. From the FAQ on the system requirements page.

"...
Does ComfyUI support MLX (Apple's machine learning framework)?
ComfyUI does not use MLX directly. On Apple Silicon (M1/M2/M3/M4) Macs, ComfyUI uses PyTorch with the MPS (Metal Performance Shaders) backend, which leverages Apple’s Metal graphics framework for GPU acceleration.
..."

System Requirements - ComfyUI

Hardware and software requirements for installing and running ComfyUI.

docs.comfy.org

If this is a 'when have a hammer everything looks like a nail' shader code ported from another GPU architecture mainly at the syntax level, then it really isn't the hardware that is at issue it is the low code optimization. That is fixable if comfy put money and effort into it. But if they are just throwing AI agent bots at it, then probably not (**).

It is appears the major part of the comfy revenue mode is getting folks to buy cloud GPU time. Super duper optimizing the local only stuff only makes that part of the business where they actually make money get smaller.
If the software is really a nudge to get you to buy cloud compute ... that really isn't about the Mac hardware abilities either.

That said there are folks who distribute MLX compatible (and sometimes optimized) local models. It isn't every model but there are some. If one of those in the the subset of models want to try out then it would be fine. Comfy isn't going to do all the homework for you. ( they could have a 'smart installer' that helped with string of questions but they don't. There are lots of 'built it from scratch code' aspects about the ecosystem still. ). It appears the Desktop version tries to be as 'smart' as most classical installers. ( grab a correct python libraries , etc. )


P.S. ** We don't have humans in the loop doing code reviews.

" ... At Comfy, I review a lot of code, and most of it isn’t written by people anymore. ..."

Comfy Internals | How we got four rival AI labs to fight over our code reviews

Four models from four labs, two passes each, one judge - a $200/month GitHub Action that catches the bugs a tired human (and four models from the same lab) wave through.

blog.comfy.org

if comfy has some robot mindlessly porting the code to some Mini/MIni Pro and tuning just for that , then it may not scale to an Ulra. That isn't MPS versus CUDA , it is just mindless robot and testing environment assigned to the job.

 

[Kronsteen]

Can anyone say with certainty whether it is reasonable to assume that an M5 Ultra Mac Studio could be clustered with one or more M7 Ultra models?

(I don’t know enough about how RDMA works over Thunderbolt to know whether if, for example, the M7 Studios support Thunderbolt 6 or have a different version of main memory technology it could cause any difficulties.)

Thanks in advance for any answers.

 

[deconstruct60]

....but an order of magnitude slower than what you'd expect simply because everything is optimized for CUDA and nothing is optimized for Mac.

This is false. It is a grossly sweeping generalization that ha no basis in fact. For example, Hugging Face as a MLX subcommunity.

" ... A community org for MLX model weights that run on Apple Silicon. This organization hosts ready-to-use models compatible with:..."

mlx-community (MLX Community)

Org profile for MLX Community on Hugging Face, the AI community building the future.

huggingface.co

Are there MLX weights for every model in Huggy Face repository? No. Is the number zero. Also No.


Your spin is implying that CUDA is somehow intrinsically intertwined with AI models and that isn't necessarily true.

So you can chain any number of Mac Studios together and you still won't get anywhere near CUDA because the issue is not speed or performance, it's the ecosystem that nVidia has built out.

With relatively unskilled porting robots that moat is wide. With trained , highly skilled robots that moat isn't was wide as Nvidia wants you to believe.

And Qualcomm threw $4B into shrinking the moat also.

Qualcomm Buys Modular to Build an Open AI Infra Standard - EE Times

Exclusive interview: Modular’s Tim Davis explains why Qualcomm’s $4B acquisition hinges on an open, multi-silicon future for AI compute.

www.eetimes.com

The same AI monster that Nvidia is building is the same monster that can devour their software moat also. CUDA is just software. It can be eaten like all the rest of the software they are trying to maximially eat.


It is pretty said though that Apple doesn't have specifically trained tools to that try to help correctly port to their platform APIs. There are some Claude Code plug-ins to XCode , but their efforts here are really rudimentary. There are lots of spots were Apple is letting 3rd parties fill in the necessary pieces and that isn't as well coordinated (and/or funded ) as it could/should be.


P.S. another example vLLM has a MLX port.
"... Then I decided, fine, I will try out vLLM-Metal. This project is a community-maintained plugin that makes vLLM work natively on Apple Silicon using MLX (Apple’s ML framework). ..."

https://medium.com/@alexagriffith/running-vllm-hello-world-on-a-mac-9ea03fba7ec6

 

[deconstruct60]

Can anyone say with certainty whether it is reasonable to assume that an M5 Ultra Mac Studio could be clustered with one or more M7 Ultra models?

(I don’t know enough about how RDMA works over Thunderbolt to know whether if, for example, the M7 Studios support Thunderbolt 6 or have a different version of main memory technology it could cause any difficulties.)

For RDMA to actually work it requires some software in the application you are running to do the correct things.
As long as the 'work grid' manager can deal with nodes working at different speeds then this would not be a problem. If there are some super rigid and large gross asssumptions built into the code then it perhaps not.

RDMA does NOT transparently give app developers global uniform memory spanning systems. Thunderbolt controllers have to go through IOMMU to get permission to access memory. The software manages some data structure to exchange metadata and give everyone a 'workspace' map that crosses sysem boundaries. Some piece of user level software will 'know' that the memory is/isn't local and route as necessary. It isn't totally 100% a hardware thing.

So far the grid/cluster management software seems to somewhat tied to specific hardware set-ups. But by time M7 arrives would be another whole operating system version ( with perhaps better tools for a grid manager to use at the OS level). And the layered grid manager technically isn't in macOS. So it has its own pace of evolution also.

If a Studio Max can be paired with a Studio Ultra now. Increasing the M-series difference in the "Ultra" (or "Max" in that specific example) won't be materially different.

It is pretty likely there won't be another version of Thunderbolt before M7. So it will have TBv5 connectors also which are governed by the same basic IOMMU design (updated for the particulars of M7 generation changes... but mainly just with smaller/faster/more energy conserving transistors. Mainly same thing just 'better'. ).

Thunderbolt is probably comatose for the interium term. It isn't going anywhere fast. Intel has bigger issues. And USB-IF would have a roit on their hands for something that would increase the cost of systems even higher while the RAM/NAND Armageddon is in flight. USB is going no where while that is going on. That makes it extremely hard for Thunderbolt to go somewhere since it is simply layered on top with just less 'optional' cost saving exits allowed (and need Intel's stamp of approval to get the certification. AMD just doesn't care. Neither do Qualcomm or anyone else doing WindowsPC SoCs. Intel's domination of the PC market is way, way ,way down. )

 

[Vincent Hanna]

But what kind of AI would you run on it without CUDA? If you just get any old gaming PC with some nVidia GPU for a fraction of the price, you'll be running AI maybe a hundred times faster.

There was a time when Macs supported nVidia GPUs... There was also a time when external GPUs seemed to be a thing. Wouldn't it be better if you could just plug in an external nVidia GPU with CUDA and just run whatever you want on whatever you want orders of magnitude faster and for a fraction of the price?

The neural engine is great for Siri and genmoji I guess but you can't do any real work with it no matter how fast is, no matter how much RAM it has.

Of all the deeply uninformed posts I’ve seen on these boards, this one might just be the worst. And that’s saying something.

You have no idea about the current state of local AI models, or Apple’s industry-leading value proposition for running them, made even better by the new M5 Studios.

We’ll just put you in charge of Apple on the 1st instead of Ternus and see how it goes.

 

[baryon]

@baryon Is this really true? I just bought a studio with one plan being to learn comfyui. My daughter works in fashion and I wanted to explore and learn fashion mood boarding using this tool.

I thought they had native Mac silicone support. Really don’t want to have to build a pc just for cuda.

Unfortunately ComfyUI isn't fast on Mac. I have an M3 Pro MacBook Pro with 36GB of RAM and can just about run the fastest models: I can generate an image with Z-Image Turbo in about 1-2 minutes. On a simple basic Windows machine with some nVidia GPU with 8GB vRAM, I'd be generating 1 image per second.

Yes it has native Apple Silicon support now which means it runs instead of crashing, which is great. Only a year ago you'd be unable to run almost anything because it lacked FP8 support, but someone fixed that recently. That doesn't mean it will be fast, it just means it will run. The M5 will be significantly faster of course, but not an order of magnitude faster.

The hardware and the vRAM isn't the limiting factor, it's the proprietary software. And Apple unfortunately did everything to stay way out of the software game when it comes to AI, so all development has been happening elsewhere, namely CUDA, which nVidia has ensured is the basis for all AI. RAM prices aren't high because everyone is buying Macs to run AI, they're high because everyone is buying nVidia GPUs.

 

[baryon]

Of all the deeply uninformed posts I’ve seen on these boards, this one might just be the worst. And that’s saying something.

You have no idea about the current state of local AI models, or Apple’s industry-leading value proposition for running them, made even better by the new M5 Studios.

We’ll just put you in charge of Apple on the 1st instead of Ternus and see how it goes.

As someone who has been running local AI models every hour of every day for the past 2-3 years on my M3 Pro MacBook Pro with 36 GB of unified RAM, maybe I do know a bit about local AI models on Macs. And no, it's not fast, at all. Generating a 4 second video at 512x320 pixels of resolution with WAN 2.2 takes 20 minutes. Any higher resolution will crash the whole thing. Generating 1 image with Z-Image Turbo takes 1 minute, and 2 minutes with Krea 2. With other models like Flux 1 or Ideogram, it takes 7-20 minutes for a single image. On any old crappy PC with some 16GB nVidia card, it will take 1 second, maybe 2. We're talking 20-100 fold differences in speed for a fraction of the price.

Sure, the M5 Sudio will probably be significantly faster than my MacBook but will it be 100 times faster? No. Maybe 5-6 times faster. Maybe if you chain 5 of them together it will be 20 times faster. That's still very very slow.

This is because all generative AI development has taken place on CUDA and nothing else. It's not optimized for anything else. You can have a fast Mac but it will only be fast if someone is written for it, which generative AI isn't. It could be, in theory, but in practice, it isn't. nVidia has profited greatly from its position and there's a reason their stock has gone up so much, they're the ones pushing the AI revolution on the hardware and software level. Everyone else is either using their hardware or being left behind.

 

[deconstruct60]

This is false. It is a grossly sweeping generalization that ha no basis in fact. For example, Hugging Face as a MLX subcommunity.

Didn't notice news that Nvidia wants to buy Huggins Face

"... To make things simple, Inference Endpoints allows customers to select the provider, region, hardware type, and instance. What is important here is that the hardware used by Amazon, Google, and Microsoft is not all Nvidia. Hugging Face currently offers, depending on the provider, AWS Inferentia, AMD Instinct, Google TPU, Intel CPUs, and Nvidia accelerators, among other configurations. ..."

https://www.tomshardware.com/tech-industry/artificial-intelligence/nvidia-to-buy-hugging-face-for-usd12-9-billion-report-claims-could-strengthen-nvidias-open-model-strategy-and-shore-up-position-against-rivals

12 Billion just to bury the competition? Probably. And while the USA DoJ anittrust division might have a Bozo-the-clown-car set of bosses above them, the EU ( where Hugging Face has a significant footprint) aren't asleep. If Nvidia tried to kill everything non-CUDA they'd very likely get sued and loose. Hugging face portable datacenter software will enable more non-USA controllerd datacenters in EU zone over time. Making that Nvidia only will decidedly just make that worse than it is now.

Nvidia is paying 80x earnings here. Buying it and then shrinking the business simply just burning money. Nvidia knows also that their moat is going to shrink. This helps them still get some revenue out of the shrinking moat.

 

[deconstruct60]

Yes it has native Apple Silicon support now which means it runs instead of crashing, which is great. Only a year ago you'd be unable to run almost anything because it lacked FP8 support, but someone fixed that recently. That doesn't mean it will be fast, it just means it will run. The M5 will be significantly faster of course, but not an order of magnitude faster.

The hardware and the vRAM isn't the limiting factor, it's the proprietary software. And Apple unfortunately did everything to stay way out of the software game when it comes to AI, so all development has been happening elsewhere, namely CUDA,

Chuckle. You contradict yourself as you switch paragraphs. Non native FP8 in the hardware is a hardware 'problem'. FP8 is being emulated. And CUDA has diddly squat to do with emulating FP8 or not.

Yes, Apple has a bigger fraction of headaches/problems with software than hardware. But it isn't all overly simplstic as you are making it out to be in your 'conclusions' and summaries.

 

[Vincent Hanna]

As someone who has been running local AI models every hour of every day for the past 2-3 years on my M3 Pro MacBook Pro with 36 GB of unified RAM

You can stop right there.

 

[Kronsteen]

-

For RDMA to actually work it requires some software in the application you are running to do the correct things.
As long as the 'work grid' manager can deal with nodes working at different speeds then this would not be a problem. If there are some super rigid and large gross asssumptions built into the code then it perhaps not.

RDMA does NOT transparently give app developers global uniform memory spanning systems. Thunderbolt controllers have to go through IOMMU to get permission to access memory. The software manages some data structure to exchange metadata and give everyone a 'workspace' map that crosses sysem boundaries. Some piece of user level software will 'know' that the memory is/isn't local and route as necessary. It isn't totally 100% a hardware thing.

So far the grid/cluster management software seems to somewhat tied to specific hardware set-ups. But by time M7 arrives would be another whole operating system version ( with perhaps better tools for a grid manager to use at the OS level). And the layered grid manager technically isn't in macOS. So it has its own pace of evolution also.

If a Studio Max can be paired with a Studio Ultra now. Increasing the M-series difference in the "Ultra" (or "Max" in that specific example) won't be materially different.

It is pretty likely there won't be another version of Thunderbolt before M7. So it will have TBv5 connectors also which are governed by the same basic IOMMU design (updated for the particulars of M7 generation changes... but mainly just with smaller/faster/more energy conserving transistors. Mainly same thing just 'better'. ).

Thunderbolt is probably comatose for the interium term. It isn't going anywhere fast. Intel has bigger issues. And USB-IF would have a roit on their hands for something that would increase the cost of systems even higher while the RAM/NAND Armageddon is in flight. USB is going no where while that is going on. That makes it extremely hard for Thunderbolt to go somewhere since it is simply layered on top with just less 'optional' cost saving exits allowed (and need Intel's stamp of approval to get the certification. AMD just doesn't care. Neither do Qualcomm or anyone else doing WindowsPC SoCs. Intel's domination of the PC market is way, way ,way down. )

That’s most helpful. Thank you for taking the time to make such a comprehensive reply.

My main reason for considering an M5 Ultra Studio is for GPU programming (it would allow me to finally retire my 2014 Mac Pro, although I also use an M3 Max MacBook Pro and a couple of Nvidia Jetson boxes), but I would also like to use it for Mathematica (the use I make of it tends to stress my MacBook a bit) and to run local LLMs (hence the desire to have the option of clustering it with an M7 box at some point in the future). But, for me, the GPU work and Mathematica are the principal use cases, so the M5 is enough to get me off the fence that I have been stuck on for several years …. 🙂

 

[dropadrop]

if comfy has some robot mindlessly porting the code to some Mini/MIni Pro and tuning just for that , then it may not scale to an Ulra. That isn't MPS versus CUDA , it is just mindless robot and testing environment assigned to the job.

I don’t know how you go from ”no human in the loop” to ”mindlessly porting the code…”.

Any organization that is starting to have maturity around using AI would likely at least be striving for getting a human out of the loop.

I’ve was working on a fairly critical software platform for well over 10 years. The software was running on millions of corporate computers ranging from office workers to servers, banks, hospitals, headless devices in shops or gasstations etc. It also had fairly complex backends running in the cloud.

There were tens of teams of developers working on different parts of the system, and some of them had automatic daily production releases if there was something new that had passed all the tests and gateways. Already years ago we removed restrictions around releases on Fridays.

The tests and coverage that gave the teams confidence could be thought of as mindless, but it was a question of coverage, sufficient overlapping gateways, professionalism and statistics that gave the confidence. For some other teams working on the same platform, they were not even close to that. Sometimes it was about technology, sometimes a lack of time / interest to create sufficient coverage.

Anyway, the mind is needed to design, plan, create and maintain the automation, not to execute the loop.

 

[MacBH928]

**EDITED** mods please delete

 

[uuee6543]

RTX 5090 hits 1.8 TB/s

Yes but you can't keep this bandwidth with multiple RTX 5090s can you?