MP All Models LLMs on Intel Macs with AMD GPUs in MacOS is here

[Dolphins1972]

That's pretty cool!

And even if new Apple Silicon generations will widen the gap in time, the development in this thread by @engeldlgado sure extends the 'new age' usability of the 7.1

It will be interesting to see where this ends and where the Mac Pro maxes out, in terms of AI usability.

Attaching my gpt-oss-20b

View attachment 2644767

What @engeldlgado has done is amazing and yes I'm excited to see the progress. The Mac Pro lives again! Can the 7,1 match the longevity of cMPs? That depends on the developers and innovators like, engeldlgado.

With newer LLMs demanding massive amounts of VRAM, our Mac Pros are handicapped. Apple Silicon's unified memory and Nvidia's Cuda architecture are the choices for developers, but AMD's open sourced ROCm is gaining popularity because it allows developers full access to the codebase.

I think Apple Silicon will eventually rise to the top or at least to the level of Nvidia because of power efficiency. Our power consumption is quickly becoming unsustainable. Nvidia's GPUs are the top GPUs for AI performance but require a lot of power and generate a lot of heat. So they are inefficient in that sense.

 

[engeldlgado]

Hi @AndreeOnline and @Dolphins1972, thank you for your kind words. They really motivate me to keep improving the project... I appreciate it so much...

By the way, Andree, what you requested regarding the projects, etc., is already included in the update, including some UI improvements. If you find anything that isn't working as expected, please let me know, because I have so many things on my mind and I might overlook something...

 

[AndreeOnline]

By the way, Andree, what you requested regarding the projects, etc., is already included in the update, including some UI improvements.

Really cool! I'm away from my Mac Pro now but will look at it tomorrow 👍🏻

After that, I'll be away for 2.5 weeks, so no more daily requests!! 😅

I was thinking about how the Mac Pro will be a bit of a 'stand-alone' machine. Since we can't update to Golden Gate, we won't be able to run the latest version of Xcode, and then we won't be able to... well, it has this ripple effect...

Still, thanks to your app, we now have this newfound power to burn through tokens locally, instead of paying high Frontier-model prices. But how can we make our work on the Mac Pro portable? The images are easy, of course, so the question is more about the chat side of things.

AI apps are popular for coding and generating text files, for good reason.

Is it possible to have a 'sidebar' of sorts, or a shelf, tied to a chat (or project, if we have those now) where a model like Qwen outputs .txt, .md, .py and .swift type of files, instead of text snippets in the chat itself? Maybe this already works to 80% by asking Qwen to generate the file... but even so, it would likely just land just below the prompt inside the chat? I can't check now, as I'm away from the computer, as I said. I'll do some experiments tomorrow!

Part one of the problem: have the model output clean files, as per request
Part two of the problem: have ToshLLM organize the files neatly, and ideally be able to refer to them in context, so that the same file can keep being revised, instead of generating new files all the time.

(I know how this works in normal commercial LLMs, but I haven't really tried it locally, and I'm not sure about what can be done, or not)

 

[engeldlgado]

Really cool! I'm away from my Mac Pro now but will look at it tomorrow 👍🏻

After that, I'll be away for 2.5 weeks, so no more daily requests!! 😅

I was thinking about how the Mac Pro will be a bit of a 'stand-alone' machine. Since we can't update to Golden Gate, we won't be able to run the latest version of Xcode, and then we won't be able to... well, it has this ripple effect...

Still, thanks to your app, we now have this newfound power to burn through tokens locally, instead of paying high Frontier-model prices. But how can we make our work on the Mac Pro portable? The images are easy, of course, so the question is more about the chat side of things.

AI apps are popular for coding and generating text files, for good reason.

Is it possible to have a 'sidebar' of sorts, or a shelf, tied to a chat (or project, if we have those now) where a model like Qwen outputs .txt, .md, .py and .swift type of files, instead of text snippets in the chat itself? Maybe this already works to 80% by asking Qwen to generate the file... but even so, it would likely just land just below the prompt inside the chat? I can't check now, as I'm away from the computer, as I said. I'll do some experiments tomorrow!

Part one of the problem: have the model output clean files, as per request
Part two of the problem: have ToshLLM organize the files neatly, and ideally be able to refer to them in context, so that the same file can keep being revised, instead of generating new files all the time.

(I know how this works in normal commercial LLMs, but I haven't really tried it locally, and I'm not sure about what can be done, or not)

I can tell you for sure that it can be done, but it would be quite a bit of work. So why not connect a code editor directly to Tosh? Tosh has server management capabilities and is compatible with the OpenAI and Anthropic APIs, so most code editors are compatible with it. You just need to point it to the local server, which by default would be localhost:8080. I don't know if that's what you're looking for, or if you'd prefer to have everything integrated into the application.

Furthermore, it even allows you to set up a local server so you can connect another computer, phone, etc. via WiFi/LAN. The only thing missing is the ability to expose that server to the internet, but for that you would need an external service like Cloudflare... which is free...

I could integrate that into the app, for example, the Cloudflare functionality, but that would break the premise of no external dependencies...

 

[AndreeOnline]

I'll check it out!

EDIT: Went through the changelog—looks really promising with lots of little nice things! Looking forward to test it.

 

[AndreeOnline]

Didn't find the time to check things out before my vacation. I'll follow what's going on from a distance and try things out a bit more in 3 weeks. Have a great summer!

 

[engeldlgado]

Didn't find the time to check things out before my vacation. I'll follow what's going on from a distance and try things out a bit more in 3 weeks. Have a great summer!

Don't worry, have a great vacation! Maybe when you get back you'll find the app completely different from what you remember... haha, who knows...

 

[engeldlgado]

Hello everyone,

It has been great being here, reading your messages and feedback. I wanted to post a quick recap of the latest ToshLLM changes in case anyone missed them.

A lot has changed since @Flint Ironstag made the original post:

• Prompt processing is much faster now, especially in long conversations and at deeper context.
• MoE models with experts offloaded to RAM received major work: faster prompt processing, better sustained generation, and no more gradual slowdown or freezes during long outputs.
• AMD GPU Flash Attention now covers more workloads and model families, including Llama 3.x, gpt-oss, Qwen, Gemma, several vision models, attention sinks, and mixed quantized KV caches.
• GCN/Vega support has improved a lot. RX 500-series, Vega, and Radeon VII cards can now keep much more of the pipeline on the GPU, including Flash Attention, MoE math, legacy quantizations, and Qwen3.5/3.6 Gated Delta Net models.
• Benchmarks now include real-generation testing, deep-context measurements, MTP acceptance stats, bottleneck reporting, and local benchmark history. There is also optional benchmark sharing for anyone who wants to contribute results.
• Multi-GPU support has progressed: you can select the GPUs used by a model, split layers across selected cards, monitor VRAM per GPU, and test an experimental shared-event hand-off path between GPUs.
• Router mode lets a single local OpenAI- or Anthropic-compatible server switch models automatically without restarting.
• ToshLLM can now run multiple local servers at once, including a dedicated embeddings server for local RAG clients.
• The chat gained projects, pinned conversations, shared project system prompts, persistent KV caches, and much faster restoration of long conversations after restarting a server.
• PDF chat supports on-device OCR for scanned documents. Vision support has improved too, including direct image paste into chat.
• There is now a local image-generation studio with text-to-image, img2img, custom models, prompt queues, parallel instances, per-instance GPU selection, Flux models, Qwen-Image support, and multi-GPU rendering.
• There is also a dedicated legacy build for pre-AVX2 Macs and Xeon-based Mac Pros.

Also we have a Official website... Yeee:

ToshLLM | Local AI for Intel Macs with AMD GPUs

Run open language models locally on Intel Macs with AMD GPUs. ToshLLM provides native Metal acceleration, private chat, vision and reproducible benchmarks.

toshllm.com

I hope to have the money soon to notarize and register the application in the Apple Store

 

[tomekwsrod]

What a great app! It resurrected my "apple abandoned Intel iMac" for ai. And I like the idea that is has minimal set of dependencies and appears privacy focused local llm as well!

 

[Dolphins1972]

Hello everyone,

It has been great being here, reading your messages and feedback. I wanted to post a quick recap of the latest ToshLLM changes in case anyone missed them.

A lot has changed since @Flint Ironstag made the original post:

• Prompt processing is much faster now, especially in long conversations and at deeper context.
• MoE models with experts offloaded to RAM received major work: faster prompt processing, better sustained generation, and no more gradual slowdown or freezes during long outputs.
• AMD GPU Flash Attention now covers more workloads and model families, including Llama 3.x, gpt-oss, Qwen, Gemma, several vision models, attention sinks, and mixed quantized KV caches.
• GCN/Vega support has improved a lot. RX 500-series, Vega, and Radeon VII cards can now keep much more of the pipeline on the GPU, including Flash Attention, MoE math, legacy quantizations, and Qwen3.5/3.6 Gated Delta Net models.
• Benchmarks now include real-generation testing, deep-context measurements, MTP acceptance stats, bottleneck reporting, and local benchmark history. There is also optional benchmark sharing for anyone who wants to contribute results.
• Multi-GPU support has progressed: you can select the GPUs used by a model, split layers across selected cards, monitor VRAM per GPU, and test an experimental shared-event hand-off path between GPUs.
• Router mode lets a single local OpenAI- or Anthropic-compatible server switch models automatically without restarting.
• ToshLLM can now run multiple local servers at once, including a dedicated embeddings server for local RAG clients.
• The chat gained projects, pinned conversations, shared project system prompts, persistent KV caches, and much faster restoration of long conversations after restarting a server.
• PDF chat supports on-device OCR for scanned documents. Vision support has improved too, including direct image paste into chat.
• There is now a local image-generation studio with text-to-image, img2img, custom models, prompt queues, parallel instances, per-instance GPU selection, Flux models, Qwen-Image support, and multi-GPU rendering.
• There is also a dedicated legacy build for pre-AVX2 Macs and Xeon-based Mac Pros.

Also we have a Official website... Yeee:

ToshLLM | Local AI for Intel Macs with AMD GPUs

Run open language models locally on Intel Macs with AMD GPUs. ToshLLM provides native Metal acceleration, private chat, vision and reproducible benchmarks.

toshllm.com

I hope to have the money soon to notarize and register the application in the Apple Store

Nice website! Wow, ToshLLM has come a long way in a short period of time. Great work!

 

[roobarb!]

Wow, I may have to get Sequoia back on my MacPro5,1 just to give this a try! Attention to Vega cards, special builds for pre-AVX2 systems... this is wild!

 

[Flint Ironstag]

They cannot kill Mac Pro.

 

[moondigger]

I have an iMac Pro with the 16 GB Vega 64X GPU. (It also has 256 GB of RAM, but as I understand it, that wouldn’t help with LLMs on this model.) I don’t know much about local LLMs, but thought I might be able to learn/explore with this system. Would this work on an iMac Pro?

Keeping on topic for this subforum, I also have a cMP and a couple Cylinder/Trashcan Mac Pros, but I don’t think any of them would be useful for LLMs.

 

[Flint Ironstag]

You can work with that machine, AND you can also connect a couple of eGPUs.

 

[engeldlgado]

Thanks a lot for the support and the feedback on the project guys.

Regarding @moondigger question:

Yes, the iMac Pro with the Vega 64X works great with ToshLLM. Vega architecture (GCN5) has dedicated support in the latest updates, so you will be able to leverage the full 16 GB of VRAM for local execution with native Metal acceleration.

As for system RAM (your 256 GB), while GPU VRAM is key for maximum generation speed, ToshLLM does allow offloading layers/context or running MoE (Mixture of Experts) models using system RAM when VRAM isn't enough... though generation will naturally be slower compared to running fully on GPU.

Also, a couple of recent additions worth mentioning:

• MCP & Agents: Model Context Protocol (MCP) integration is already live, along with the first phase of local Agent support to connect external tools.
• Fabric Link on Dual GPUs: I'm currently exploring and testing native Infinity Fabric Link support for dual-GPU AMD cards (like the W6800X Duo or Vega II Duo) to enable faster interconnect communication and smoother multi-GPU scaling under macOS.

And as @Flint Ironstag mentioned, connecting eGPUs via Thunderbolt is also a totally valid path if you ever need to scale up your VRAM.