MP All Models LLMs on Intel Macs with AMD GPUs in MacOS is here

[engeldlgado]

I see you've been quite busy and I appreciate all the work you're doing. When you get the Vega cards figured out, can you look into enabling multi-GPU for images? I'm limited to 16GB models using only one GPU.

Good timing... I was testing exactly this on the engine this week, and the good news is that the plumbing already works: in my tests the image pipeline can put the text encoder and the VAE on a second GPU while the diffusion model keeps the main one, with identical output.

To set expectations: it won't merge both cards into one big VRAM pool. The diffusion model itself still has to fit on a single GPU (splitting it across cards doesn't really work for image models, and it wouldn't make them faster either). What it does is move everything else off your main card, and with the newer models that's a lot: they carry a full LLM just as text encoder.

For your dual 6900XT specifically, the numbers look like this:

Qwen-Image gets unlocked. Today it needs 24 GB because the 11 GB diffusion model and its 4.5 GB text encoder have to live together. With the encoder on your second card, it fits in 16 GB. And it's the model that renders legible text inside images, so it's a real upgrade, not just a bigger number.

Flux.2 klein 9B gets room to breathe: its Qwen3-8B encoder weighs almost as much as the diffusion model itself (5 GB), so moving it out frees a third of your card for larger frames.

Flux.2 dev stays out of reach, to be honest: its diffusion model alone is 19 GB, and that part can't be split.

It needs UI work, so it goes in the queue right after the Vega/GCN fixes I'm working on now. Your dual 6900XT is actually the ideal hardware for the first test build, so I'll probably knock on your door when it's ready.

 

[engeldlgado]

Alright...

Tried the new build and I could run Qwen image for the first time—nice. It's slow, but seems to produce quality images. Have not used it extensively. But the queue comes in handy for slower models: drop a few prompts and leave the computer to it!

Here are some screens that summarise what I tried to communicate:

First of all... thank you, for the detailed screenshots AND for the donation. Contributions like yours (the mockups, the testing, and now this) are literally what keeps this project moving, so it means a lot. 🤗

Now, going through your screens one by one, because there's good news in all of them:

The taller prompt field: already shipped! Update to the latest version and the queue prompt grows up to several lines as you type. You were one build behind when you took that screenshot.

Per-instance queues: you nailed the real problem in your note... instances can run very different models and settings, and today a queued prompt just lands on whatever instance is free next. That's getting fixed, and it's in the works right now. The implementation will look a bit different from your three-column mockup though: instead of duplicating the whole queue UI per instance, each queued prompt gets a small destination picker ("Any / Instance 1 / 2 / 3") next to the Add button, plus an instance badge on every result in the feed. Same control you asked for, but it keeps working with 4+ instances and narrow windows. Your "split queue per instance" toggle stops being necessary because both behaviors coexist in one queue.

Wider images in the instances view: noted, you're right that there's a lot of unused space there. It's on the list.

And the one I think you'll like most: your "dedicate one GPU to chat, three to images" scenario is NOT actually blocked today. That orange warning is just a warning, not a lock... the real hazard is chat and image generation sharing the same GPU on AMD. If you pin the chat server to one card (GPU selector in Settings) and point your image instances at the other three, you can run everything at once right now. The warning being scary even when your setup is safe is a bug on my side: it's being made GPU-aware, so it will only appear when there's a real conflict.

Thanks again... reports like yours are the best data this project gets.

 

[AndreeOnline]

First of all... thank you...

No, sir. Thank you!

Now, going through the good news...

Great, appreciated!

And the one I think you'll like most: your "dedicate one GPU to chat, three to images" scenario is NOT actually blocked today.

Aha! I'll check that out! Cool! 🙌🏻

 

[AndreeOnline]

Yes! Chat and images at the same time works great—good to know!

In the latest update (.56), I still see the 'single line' textfield for queue prompts (image attached—as you can see, we can only read the last few words of a larger prompt).

In the category 'spit and polish', I wouldn't mind having some control of fonts, mainly in the chat bubbles and image prompts (the text I'm inputting and the chat replies, as well as the same prompts being reflected in the image descriptions under instance/queue).
I understand that some interface text items might need to be locked in to avoid overflow and such.

ChatGPT's chats use 16px with line-height of 26px, which gives everything a little more room to breathe, and at least to me, feels more polished. I know some people like a tighter, more "efficient" text fit, so it should probably be an option.

 

[AndreeOnline]

It never ends, right....

Next 'feature request': for a contained app such as yours, it's great to be able to access as much of the functionality as possible that you would otherwise have access to in a node network, such as ComfyUI. Obviously, one would need to be selective in what to include.

One thing that might be worth it—similar to you already offering the img2img 'well', is another 'well' (per instance) that allows me to point to a xxx_lora.safetensors file. It's one of the things we have to be able to 'massage' the model with specific styles. And since we can train LoRAs locally ourselves, it makes it even more powerful.

The LoRAs are specific to each model, that is why it needs to be per instance, if included.

EDIT/ADDED: under the queue tab, we see already rendered images with their prompts and the time it took and so on. Since ZIT has very weak seed variance, it would be very nice to have the 'copy text' icon from the chat, where I can quickly copy the previously rendered prompt, so that I can reload it at the top and then make adjustments to it before re-adding it to the queue as a variant. No wait, instead of a 'copy text', just have a small 'recycle' icon that takes the prompt and loads it into the live prompt field at the top. That would be sweet!

 

[Dolphins1972]

Good timing... I was testing exactly this on the engine this week, and the good news is that the plumbing already works: in my tests the image pipeline can put the text encoder and the VAE on a second GPU while the diffusion model keeps the main one, with identical output.

To set expectations: it won't merge both cards into one big VRAM pool. The diffusion model itself still has to fit on a single GPU (splitting it across cards doesn't really work for image models, and it wouldn't make them faster either). What it does is move everything else off your main card, and with the newer models that's a lot: they carry a full LLM just as text encoder.

For your dual 6900XT specifically, the numbers look like this:

Qwen-Image gets unlocked. Today it needs 24 GB because the 11 GB diffusion model and its 4.5 GB text encoder have to live together. With the encoder on your second card, it fits in 16 GB. And it's the model that renders legible text inside images, so it's a real upgrade, not just a bigger number.

Flux.2 klein 9B gets room to breathe: its Qwen3-8B encoder weighs almost as much as the diffusion model itself (5 GB), so moving it out frees a third of your card for larger frames.

Flux.2 dev stays out of reach, to be honest: its diffusion model alone is 19 GB, and that part can't be split.

It needs UI work, so it goes in the queue right after the Vega/GCN fixes I'm working on now. Your dual 6900XT is actually the ideal hardware for the first test build, so I'll probably knock on your door when it's ready.

Ok great. I'll be glad to test it when you have time to work on it.

 

[MacClueless]

Thanks! Actually, the latest ToshLLM update already includes an experimental "Split model across all GPUs" option, which is perfect to leverage your 128GB of combined VRAM.

I'm currently waiting on some donated Vega 64 cards to arrive so I can properly profile and optimize the multi-GPU support, but international shipping logistics are heavily delayed right now due to the earthquake situation in my country. Since this feature is still unvalidated on Metal/AMD, your 4-GPU setup is the perfect testbed right now. I'd love for you to enable it in Settings and share your benchmark results!

Splitting an LLM across cards won't work isn't worth it on Mac Pro 6,1 though. On Mac Pro 6,1 each GPU’s memory is isolated and everything crossing cards has to go through PCIe and system RAM; every layer transfer or activation that crosses cards would have that massive latency and bandwidth bottleneck. It would be slower, NOT faster.

 

[engeldlgado]

It never ends, right....

No problem, there's always room for improvement, but first check the latest update. I've already applied most of the changes, but not these yet...

Splitting an LLM across cards won't work isn't worth it on Mac Pro 6,1 though. On Mac Pro 6,1 each GPU’s memory is isolated and everything crossing cards has to go through PCIe and system RAM;

I don't entirely agree with this. Of course, they won't be running at full speed, but they will be very stable. You can check out this benchmark from a tester who used the Qwen3.6 MoE F16, which weighs in at 70GB and required three GPUs. Although the benchmark shows speed degradation in long-term scenarios, in the latest patches, this degradation is minimal. Check it out here:

Multi GPUs run: 4 W6800X on Qwen3.5-122B-A10B-Q5_K_M.gguf · Issue #16 · engeldlgado/toshllm

What would you like ToshLLM to do? Good performance for bigger model: === ToshLLM benchmark · 2026-06-25T11:04:43Z === model: Qwen3.5-122B-A10B-Q5_K_M.gguf GPU: Split · 4 GPUs engine: bundled · K:f...

github.com

I should also mention that in a model like the Qwen3.6 35B MoE with two graphics cards... one RX 6800XT and another via Thunderbolt eGPU, a 6900XT, which would have experienced greater speed degradation... another tester reported a constant and stable t/g of around 56t/g. So it's definitely worth it, since before you couldn't even use these Macs for inference; now you can, and very well, at speeds never before seen on an Intel Mac with macOS.

 

[Anonymous Freak]

Yeah, but does it work on the FirePro D500s in my trash can?

 

[engeldlgado]

Ok great. I'll be glad to test it when you have time to work on

The update is ready; it should allow the use of at least the 16GB models and provide more space for creating images.

Yeah, but does it work on the FirePro D500s in my trash can?

Not yet, maybe in the future. To give you some context, the application was originally designed for RDNA+, not for Vega, but I saw that Vega and GCN boards were also widely used, and I'm currently working with that kernel to improve its performance on those architectures. But a couple of weeks ago nothing was working on them; today they are working, but i'm working on their stability right now. I can't promise I'll be able to do it because it's one of the early versions of the GCN series, and I don't know how Mac handles them now, but I'll look into it.

 

[Anonymous Freak]

Not yet, maybe in the future. To give you some context, the application was originally designed for RDNA+, not for Vega, but I saw that Vega and GCN boards were also widely used, and I'm currently working with that kernel to improve its performance on those architectures. But a couple of weeks ago nothing was working on them; today they are working, but i'm working on their stability right now. I can't promise I'll be able to do it because it's one of the early versions of the GCN series, and I don't know how Mac handles them now, but I'll look into it.

It was mostly a joke. I don't think much useful could be done on the dual D500s with 2 GB each.

 

[AndreeOnline]

Re-ran the benchmark with the latest release and saw some nice improvements. The benchmark results vary wildly based on what model is chosen, which is expected. Not sure if there is a "standard" benchmark (model) that people run that make numbers more comparable?

 

[engeldlgado]

Re-ran the benchmark with the latest release and saw some nice improvements. The benchmark results vary wildly based on what model is chosen, which is expected. Not sure if there is a "standard" benchmark (model) that people run that make numbers more comparable?

I don't have a defined benchmark right now, but let me tell you, those numbers are beautiful... I'd also appreciate it if you could share the benchmarks.txt file from the logs after you've tested any models you have there. That data is useful for me to understand the models performance on your Duos.

Thank you so much for sharing those numbers here... I'm glad to know it's working at a very good capacity now... there's still a long way to go.

 

[AndreeOnline]

Here are my benchmarks so far. I don't see any benchmark benefits from 'split across GPUs'. In fact, it comes with a small cost, most likely from overhead due to the GPUs having to communicate.
I still think it's worth working on, since even if strong open models fit on a single GPU today, they might need to be split as new, bigger models are released?

I still know too little about the inner workings of these LLMs to have any input there.

EDIT: On the chat side of requests: would be cool with 'per chat' instructions, if it's possible. Or if we could have Projects/folders like ChatGPT that collect chats around a common topic. Then 'project instructions' would be sufficient.

 

[engeldlgado]

Here are my benchmarks so far. I don't see any benchmark benefits from 'split across GPUs'. In fact, it comes with a small cost, most likely from overhead due to the GPUs having to communicate.
I still think it's worth working on, since even if strong open models fit on a single GPU today, they might need to be split as new, bigger models are released?

I still know too little about the inner workings of these LLMs to have any input there.

EDIT: On the chat side of requests: would be cool with 'per chat' instructions, if it's possible. Or if we could have Projects/folders like ChatGPT that collect chats around a common topic. Then 'project instructions' would be sufficient.

That's the idea behind split multi-GPU, as well as having more context capacity... yes... it has a cost, but imagine running a 100b+ model where you couldn't run it before on a single GPU... there's a tester trying out multi-GPU on 3 W6800X Duos, using models over 200b, and it's working decently for him so far.

What you're suggesting sounds good, I'll keep it in mind. I might implement both soon, both for a project and for a simple chat... By the way, you didn't mention the adjustment I made to the image generation UI; I'd like to hear your opinion on it now.

 

[AndreeOnline]

By the way, you didn't mention the adjustment I made to the image generation UI; I'd like to hear your opinion on it now.

Yes! Very nice. I'll keep working with the app, and see what comes up, but I think the current layout is pretty flexible!

 

[Dolphins1972]

The update is ready; it should allow the use of at least the 16GB models and provide more space for creating images.

Great work! It nearly doubled the speed on my system. The top 3 benchmarks are from the latest build. The bottom 3 are from an older build, sorry I do not have the logs for the older benchmarks.

Screen recording of ToshLLM in action, both GPUs working.

I'm jealous of the 6800XT duo sharing 32GB of VRAM.

EDIT: QWEN 24GB model

Some of the work is offloaded to CPU now. (see video above)

 

[engeldlgado]

Great work! It nearly doubled the speed on my system. The top 3 benchmarks are from the latest build. The bottom 3 are from an older build, sorry I do not have the logs for the older benchmarks. View attachment 2644686

Screen recording of ToshLLM in action, both GPUs working.
View attachment 2644687
I'm jealous of the 6800XT duo sharing 32GB of VRAM.

EDIT: QWEN 24GB model
View attachment 2644703
Some of the work is offloaded to CPU now. (see video above)

Use the bundle engine on settings, turbo doesnt have the latest improvements, so it will give you better results, i think

 

[Dolphins1972]

Use the bundle engine on settings, turbo doesnt have the latest improvements, so it will give you better results, i think

It gave me a boost with Qwen but not with Gemma or Pixtral.

 

[engeldlgado]

It gave me a boost with Qwen but not with Gemma or Pixtral.View attachment 2644725

Great, the engine seems healthy for now; in fact, I was optimizing some things in Gemma. What I'd like you to try when you can is the gpt-oss-20b model, the Q4_K_M version, or the MXFP4

THANK YOU!!!

 

[Dolphins1972]

Great, the engine seems healthy for now; in fact, I was optimizing some things in Gemma. What I'd like you to try when you can is the gpt-oss-20b model, the Q4_K_M version, or the MXFP4

THANK YOU!!!

Sure will.

 

[engeldlgado]

Sure will.

View attachment 2644753

Incredible, it even works better than an M4 Max, not a pro max of course, but it's in that range now

If you don't believe me... look this: https://github.com/ggml-org/llama.cpp/discussions/15396#discussioncomment-14224424

 

[Dolphins1972]

Incredible, it even works better than an M4 Max, not a pro max of course, but it's in that range now

If you don't believe me... look this: https://github.com/ggml-org/llama.cpp/discussions/15396#discussioncomment-14224424

Yep, The RX 6900 XT crushes all the M4s Metal benchmarks, even the Pro and even trounces the M5 Max Metal benchmarks.

Metal Benchmarks - Geekbench

browser.geekbench.com

 

[engeldlgado]

Yep, The RX 6900 XT crushes all the M4s Metal benchmarks, even the Pro and even trounces the M5 Max Metal benchmarks.

Metal Benchmarks - Geekbench

browser.geekbench.com

I want that card, urgent... haha thanks for testing

 

[AndreeOnline]

Yep, The RX 6900 XT crushes all the M4s Metal benchmarks, even the Pro and even trounces the M5 Max Metal benchmarks.

That's pretty cool!

And even if new Apple Silicon generations will widen the gap in time, the development in this thread by @engeldlgado sure extends the 'new age' usability of the 7.1

It will be interesting to see where this ends and where the Mac Pro maxes out, in terms of AI usability.

Attaching my gpt-oss-20b