Toggle sidebar Toggle sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

MP All Models LLMs on Intel Macs with AMD GPUs in MacOS is here

Flint Ironstag

Flint Ironstag

macrumors 68000
Original poster
There's a large installed base of these machines and inevitably, a clever individual has released an app to serve that base. It's called ToshLLM and looks interesting (no affiliation). I know some folks here have big GPU power but feel forced into Linux - this may be worth checking out, and the dev is looking for testers.

I consider this Mac Pro centric because we are best able to stack massive GPU power for these tasks.
 
  • Like
Reactions: ares1849, keksikuningas, Oval and 1 other person
tsialex said:
I'm impressed with the developer…
Click to expand...

Thanks for the heads up, Alex!

Extremely user-friendly setup, by which I mean: there basically is no setup to talk about. Just download the dmg, download the recommended model and away you go.

I haven't really looked into local LLMs at all before this, but I'll check it out now. No clue yet about expected performance and what a 'good' score might be.

Anyway. Clean install and zero issues for this is nice.
 

Attachments

  • Skärmavbild 2026-06-23 kl. 17.48.16.png
    Skärmavbild 2026-06-23 kl. 17.48.16.png
    772.7 KB · Views: 158
  • Love
  • Like
Reactions: Martin2022 and devon807
Flint Ironstag said:
There's a large installed base of these machines and inevitably, a clever individual has released an app to serve that base. It's called ToshLLM and looks interesting (no affiliation). I know some folks here have big GPU power but feel forced into Linux - this may be worth checking out, and the dev is looking for testers.

I consider this Mac Pro centric because we are best able to stack massive GPU power for these tasks.
Click to expand...

Froze my machine, maybe I was using too big of a LLM
 
capamac said:
LLMs are trashing SSDs according to The Register.
Click to expand...

Note that particular situation is due to an implementation bug/design decision in OpenAI's Codex not inherent to LLM.

However, running LLM larger than the unified RAM on Apple Silicon would likely thrash the SSD, too. That's also typically slow enough not to be an issue anyway.
 
  • Like
Reactions: dblissmn
underthere said:
Think it would work on iMac? I’ll give it a shot!
Click to expand...

With systems with 8GB of VRAM such as yours, start with the smaller models -- for example 7B parameter models at Q5.

Here's a tool to estimate VRAM requirements (there's at least one AMD card on their list with 8GB of VRAM which should be close enough to your hardware for feasibility estimates):
apxml.com

Can You Run This LLM? VRAM Calculator (Nvidia GPU and Apple Silicon)

Calculate the VRAM required to run any large language model.
apxml.com apxml.com
 
Just installed this. I've never run LLMs locally since I have only Intel Macs, so I don't really know how to use it yet. I've only used browser based generators up to now. Anyway, did some benchmarks on my 7,1.
 

Attachments

  • Screenshot 2026-06-25 at 9.21.47 PM.png
    Screenshot 2026-06-25 at 9.21.47 PM.png
    265.9 KB · Views: 83
  • Like
Reactions: bzgnyc2
I'm looking forward to following this development. I think it's safe to assume a couple of things:

At some point, the rubber will have to hit the road when it comes to pricing and popular models such as Claude and ChatGPT. I think the subscriptions will have to be substantially more expensive for meaningful work, even if basic stuff might be free with ads and lesser models.

I also think the open source models will come closer and closer to frontier models as time passes, and not the other way around.

I hope to see the local models keep adding support for multi-GPU setups that effectively split layers/roles between GPUs.

It doesn't really matter that much, but it would be so #chefskiss if the 7.1 Mac Pros with dual Duos all of a sudden become a perfect fit for a niche, but useful, AI use case.
 
  • Like
Reactions: flaubert and bzgnyc2
AndreeOnline said:
I'm looking forward to following this development. I think it's safe to assume a couple of things:

At some point, the rubber will have to hit the road when it comes to pricing and popular models such as Claude and ChatGPT. I think the subscriptions will have to be substantially more expensive for meaningful work, even if basic stuff might be free with ads and lesser models.

I also think the open source models will come closer and closer to frontier models as time passes, and not the other way around.

I hope to see the local models keep adding support for multi-GPU setups that effectively split layers/roles between GPUs.

It doesn't really matter that much, but it would be so #chefskiss if the 7.1 Mac Pros with dual Duos all of a sudden become a perfect fit for a niche, but useful, AI use case.
Click to expand...
Yes, breathing life back into the 7,1 will make many happy. I expect the 7,1 resale value to spike as word gets out. I see the developer (Reddit thread) is planning on adding support for image and video generation.
 
Last edited:
Hi everyone, let me introduce myself... I am the developer of the app, and it is a real pleasure for me that you found it and are testing it out. A friend mentioned that it was posted here, so I wanted to drop by and let you know that I will be keeping a close eye on any bugs so I can fix them as soon as possible. I hope to continue improving compatibility for Intel Macs, which were abandoned by Apple and other local AI apps because they supposedly 'aren't worth it.'

I also want to apologize for the lack of updates over the last few days. An earthquake recently hit my city. Fortunately, my family and friends are all safe, but internet connections were severely affected and part of my home was damaged. I hope to fully get back to the project soon. Best regards.
 
  • Like
Reactions: Basic75, flaubert, edubfromktown and 7 others
engeldlgado said:
Hi everyone, let me introduce myself...
Click to expand...
A serious mistake on your part, to introduce yourself here.... now we won't let you go! 🤗

Thanks for the app! I'm not sure what role it will play yet, alongside the normal commercial clients I have subscriptions on now, but there is a rich tradition of hacking either the Mac Pro hardware itself, or software that isn't supposed to run on it. I love to see developments like this!

I'd be curious to hear from you what you think is possible to achieve with the app, in terms of leveraging Mac Pro hardware. In my own configuration, I (and a few others here too, of course) have 4x 32GB graphics cards, which is a bit 'atypical' for PCs. But as it turns out, this amount of GPUs and memory is great for LLMs.... if they can use it.

I think I need to dive a bit deeper into understanding how LLMs work behind the scenes, but when talking to GPTs, they suggest it might be possible to split 'Layers' or 'MoEs' across more than one GPU. Surely, this is what Nvidia does with their racks—but it's also another architecture.

Hoping to keep a discussion alive!

Thanks again for the super clean implementation and install!

----------------
Wish you all the best for a quick recovery after the quake.
 
AndreeOnline said:
A serious mistake on your part, to introduce yourself here.... now we won't let you go! 🤗

Thanks for the app! I'm not sure what role it will play yet, alongside the normal commercial clients I have subscriptions on now, but there is a rich tradition of hacking either the Mac Pro hardware itself, or software that isn't supposed to run on it. I love to see developments like this!

I'd be curious to hear from you what you think is possible to achieve with the app, in terms of leveraging Mac Pro hardware. In my own configuration, I (and a few others here too, of course) have 4x 32GB graphics cards, which is a bit 'atypical' for PCs. But as it turns out, this amount of GPUs and memory is great for LLMs.... if they can use it.

I think I need to dive a bit deeper into understanding how LLMs work behind the scenes, but when talking to GPTs, they suggest it might be possible to split 'Layers' or 'MoEs' across more than one GPU. Surely, this is what Nvidia does with their racks—but it's also another architecture.

Hoping to keep a discussion alive!

Thanks again for the super clean implementation and install!

----------------
Wish you all the best for a quick recovery after the quake.
Click to expand...

Thanks! Actually, the latest ToshLLM update already includes an experimental "Split model across all GPUs" option, which is perfect to leverage your 128GB of combined VRAM.

I'm currently waiting on some donated Vega 64 cards to arrive so I can properly profile and optimize the multi-GPU support, but international shipping logistics are heavily delayed right now due to the earthquake situation in my country. Since this feature is still unvalidated on Metal/AMD, your 4-GPU setup is the perfect testbed right now. I'd love for you to enable it in Settings and share your benchmark results!
 
  • Like
Reactions: Oval
For Vega/GCN Cards, follow this issue... also test the custom build there for those cards, maybe work better, its currently on alpha testing.

github.com

Models don't load · Issue #1 · engeldlgado/toshllm

macOS version 14.8.3 GPU and VRAM RX580, 48 GB ram Model and quantization any (Q4) What happened? No models placed in ~/models load. I get a 'Engine exited with code 4' error. These models work in ...
github.com github.com
 
engeldlgado said:
I'd love for you to enable it in Settings and share your benchmark results!
Click to expand...
I did actually try enabling 'split across GPUs' but that led to a small negative impact. I guess it was more or less just the overhead of managing more GPUs, without actually being able to distribute meaningful work between them (as it stands now). But I also don't know if the included benchmark is complex enough to benefit for multiple GPUs?

The Mac Pro is not my main computer these days. It's more of a workstation for certain tasks, but I'll stay up to date with this new development.
 
Dolphins1972 said:
I see the developer (Reddit thread) is planning on adding support for image and video generation.
Click to expand...

If I understand it correctly, it's only text generation right now. In looking into it, I've come across Comfy and node based image/video generation, locally for free. That's pretty sweet.

Any change of getting something like Flux into ToshLLM, or are there restrictions there, such as 'only for Nvidia'?
 
AndreeOnline said:
If I understand it correctly, it's only text generation right now. In looking into it, I've come across Comfy and node based image/video generation, locally for free. That's pretty sweet.

Any change of getting something like Flux into ToshLLM, or are there restrictions there, such as 'only for Nvidia'?
Click to expand...
Screenshot 2026-06-28 at 2.19.01 PM.png

Developer says image and video will come after the app stabilizes (Reddit thread). I personally have not found any image/video app that can be run locally on any of my Mac Pros (3 currently but only use 2).

Image/video on Intel Mac? If so, can you elaborated?
 
Dolphins1972 said:
View attachment 2641615
Developer says image and video will come after the app stabilizes (Reddit thread). I personally have not found any image/video app that can be run locally on any of my Mac Pros (3 currently but only use 2).

Image/video on Intel Mac? If so, can you elaborated?
Click to expand...
Hi, yes, I can confirm that this will be implemented in the future, as I tested it in the early stages of the project and it works. Of course, I used a small model, and well, it generated an astronaut with 3 arms and 4 legs haha. But yes, I am planning to do it; it will just require dedicating a bit more time than usual to make it fast and stable. Also need suggestion about what model i cant test...
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back