Toggle sidebar Toggle sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

MP All Models LLMs on Intel Macs with AMD GPUs in MacOS is here

AndreeOnline said:
That's pretty cool!

And even if new Apple Silicon generations will widen the gap in time, the development in this thread by @engeldlgado sure extends the 'new age' usability of the 7.1

It will be interesting to see where this ends and where the Mac Pro maxes out, in terms of AI usability.

Attaching my gpt-oss-20b

View attachment 2644767
Click to expand...
What @engeldlgado has done is amazing and yes I'm excited to see the progress. The Mac Pro lives again! Can the 7,1 match the longevity of cMPs? That depends on the developers and innovators like, engeldlgado.

With newer LLMs demanding massive amounts of VRAM, our Mac Pros are handicapped. Apple Silicon's unified memory and Nvidia's Cuda architecture are the choices for developers, but AMD's open sourced ROCm is gaining popularity because it allows developers full access to the codebase.

I think Apple Silicon will eventually rise to the top or at least to the level of Nvidia because of power efficiency. Our power consumption is quickly becoming unsustainable. Nvidia's GPUs are the top GPUs for AI performance but require a lot of power and generate a lot of heat. So they are inefficient in that sense.
 
Hi @AndreeOnline and @Dolphins1972, thank you for your kind words. They really motivate me to keep improving the project... I appreciate it so much...

By the way, Andree, what you requested regarding the projects, etc., is already included in the update, including some UI improvements. If you find anything that isn't working as expected, please let me know, because I have so many things on my mind and I might overlook something...
 
  • Like
Reactions: Dolphins1972 and AndreeOnline
engeldlgado said:
By the way, Andree, what you requested regarding the projects, etc., is already included in the update, including some UI improvements.
Click to expand...
Really cool! I'm away from my Mac Pro now but will look at it tomorrow 👍🏻

After that, I'll be away for 2.5 weeks, so no more daily requests!! 😅

I was thinking about how the Mac Pro will be a bit of a 'stand-alone' machine. Since we can't update to Golden Gate, we won't be able to run the latest version of Xcode, and then we won't be able to... well, it has this ripple effect...

Still, thanks to your app, we now have this newfound power to burn through tokens locally, instead of paying high Frontier-model prices. But how can we make our work on the Mac Pro portable? The images are easy, of course, so the question is more about the chat side of things.

AI apps are popular for coding and generating text files, for good reason.

Is it possible to have a 'sidebar' of sorts, or a shelf, tied to a chat (or project, if we have those now) where a model like Qwen outputs .txt, .md, .py and .swift type of files, instead of text snippets in the chat itself? Maybe this already works to 80% by asking Qwen to generate the file... but even so, it would likely just land just below the prompt inside the chat? I can't check now, as I'm away from the computer, as I said. I'll do some experiments tomorrow!

Part one of the problem: have the model output clean files, as per request
Part two of the problem: have ToshLLM organize the files neatly, and ideally be able to refer to them in context, so that the same file can keep being revised, instead of generating new files all the time.

(I know how this works in normal commercial LLMs, but I haven't really tried it locally, and I'm not sure about what can be done, or not)
 
AndreeOnline said:
Really cool! I'm away from my Mac Pro now but will look at it tomorrow 👍🏻

After that, I'll be away for 2.5 weeks, so no more daily requests!! 😅

I was thinking about how the Mac Pro will be a bit of a 'stand-alone' machine. Since we can't update to Golden Gate, we won't be able to run the latest version of Xcode, and then we won't be able to... well, it has this ripple effect...

Still, thanks to your app, we now have this newfound power to burn through tokens locally, instead of paying high Frontier-model prices. But how can we make our work on the Mac Pro portable? The images are easy, of course, so the question is more about the chat side of things.

AI apps are popular for coding and generating text files, for good reason.

Is it possible to have a 'sidebar' of sorts, or a shelf, tied to a chat (or project, if we have those now) where a model like Qwen outputs .txt, .md, .py and .swift type of files, instead of text snippets in the chat itself? Maybe this already works to 80% by asking Qwen to generate the file... but even so, it would likely just land just below the prompt inside the chat? I can't check now, as I'm away from the computer, as I said. I'll do some experiments tomorrow!

Part one of the problem: have the model output clean files, as per request
Part two of the problem: have ToshLLM organize the files neatly, and ideally be able to refer to them in context, so that the same file can keep being revised, instead of generating new files all the time.

(I know how this works in normal commercial LLMs, but I haven't really tried it locally, and I'm not sure about what can be done, or not)
Click to expand...
I can tell you for sure that it can be done, but it would be quite a bit of work. So why not connect a code editor directly to Tosh? Tosh has server management capabilities and is compatible with the OpenAI and Anthropic APIs, so most code editors are compatible with it. You just need to point it to the local server, which by default would be localhost:8080. I don't know if that's what you're looking for, or if you'd prefer to have everything integrated into the application.

Furthermore, it even allows you to set up a local server so you can connect another computer, phone, etc. via WiFi/LAN. The only thing missing is the ability to expose that server to the internet, but for that you would need an external service like Cloudflare... which is free...

I could integrate that into the app, for example, the Cloudflare functionality, but that would break the premise of no external dependencies...
 
  • Like
Reactions: tomekwsrod and AndreeOnline
I'll check it out!

EDIT: Went through the changelog—looks really promising with lots of little nice things! Looking forward to test it.
 
Last edited:
  • Like
Reactions: engeldlgado
Didn't find the time to check things out before my vacation. I'll follow what's going on from a distance and try things out a bit more in 3 weeks. Have a great summer!
 
Hello everyone,

It has been great being here, reading your messages and feedback. I wanted to post a quick recap of the latest ToshLLM changes in case anyone missed them.

A lot has changed since @Flint Ironstag made the original post:
  • Prompt processing is much faster now, especially in long conversations and at deeper context.
  • MoE models with experts offloaded to RAM received major work: faster prompt processing, better sustained generation, and no more gradual slowdown or freezes during long outputs.
  • AMD GPU Flash Attention now covers more workloads and model families, including Llama 3.x, gpt-oss, Qwen, Gemma, several vision models, attention sinks, and mixed quantized KV caches.
  • GCN/Vega support has improved a lot. RX 500-series, Vega, and Radeon VII cards can now keep much more of the pipeline on the GPU, including Flash Attention, MoE math, legacy quantizations, and Qwen3.5/3.6 Gated Delta Net models.
  • Benchmarks now include real-generation testing, deep-context measurements, MTP acceptance stats, bottleneck reporting, and local benchmark history. There is also optional benchmark sharing for anyone who wants to contribute results.
  • Multi-GPU support has progressed: you can select the GPUs used by a model, split layers across selected cards, monitor VRAM per GPU, and test an experimental shared-event hand-off path between GPUs.
  • Router mode lets a single local OpenAI- or Anthropic-compatible server switch models automatically without restarting.
  • ToshLLM can now run multiple local servers at once, including a dedicated embeddings server for local RAG clients.
  • The chat gained projects, pinned conversations, shared project system prompts, persistent KV caches, and much faster restoration of long conversations after restarting a server.
  • PDF chat supports on-device OCR for scanned documents. Vision support has improved too, including direct image paste into chat.
  • There is now a local image-generation studio with text-to-image, img2img, custom models, prompt queues, parallel instances, per-instance GPU selection, Flux models, Qwen-Image support, and multi-GPU rendering.
  • There is also a dedicated legacy build for pre-AVX2 Macs and Xeon-based Mac Pros.
Also we have a Official website... Yeee:

toshllm.com

ToshLLM | Local AI for Intel Macs with AMD GPUs

Run open language models locally on Intel Macs with AMD GPUs. ToshLLM provides native Metal acceleration, private chat, vision and reproducible benchmarks.
toshllm.com toshllm.com

I hope to have the money soon to notarize and register the application in the Apple Store
 
  • Like
  • Wow
Reactions: turk101, rb2112, roobarb! and 6 others
What a great app! It resurrected my "apple abandoned Intel iMac" for ai. And I like the idea that is has minimal set of dependencies and appears privacy focused local llm as well!
 
  • Like
Reactions: Flint Ironstag
engeldlgado said:
Hello everyone,

It has been great being here, reading your messages and feedback. I wanted to post a quick recap of the latest ToshLLM changes in case anyone missed them.

A lot has changed since @Flint Ironstag made the original post:
  • Prompt processing is much faster now, especially in long conversations and at deeper context.
  • MoE models with experts offloaded to RAM received major work: faster prompt processing, better sustained generation, and no more gradual slowdown or freezes during long outputs.
  • AMD GPU Flash Attention now covers more workloads and model families, including Llama 3.x, gpt-oss, Qwen, Gemma, several vision models, attention sinks, and mixed quantized KV caches.
  • GCN/Vega support has improved a lot. RX 500-series, Vega, and Radeon VII cards can now keep much more of the pipeline on the GPU, including Flash Attention, MoE math, legacy quantizations, and Qwen3.5/3.6 Gated Delta Net models.
  • Benchmarks now include real-generation testing, deep-context measurements, MTP acceptance stats, bottleneck reporting, and local benchmark history. There is also optional benchmark sharing for anyone who wants to contribute results.
  • Multi-GPU support has progressed: you can select the GPUs used by a model, split layers across selected cards, monitor VRAM per GPU, and test an experimental shared-event hand-off path between GPUs.
  • Router mode lets a single local OpenAI- or Anthropic-compatible server switch models automatically without restarting.
  • ToshLLM can now run multiple local servers at once, including a dedicated embeddings server for local RAG clients.
  • The chat gained projects, pinned conversations, shared project system prompts, persistent KV caches, and much faster restoration of long conversations after restarting a server.
  • PDF chat supports on-device OCR for scanned documents. Vision support has improved too, including direct image paste into chat.
  • There is now a local image-generation studio with text-to-image, img2img, custom models, prompt queues, parallel instances, per-instance GPU selection, Flux models, Qwen-Image support, and multi-GPU rendering.
  • There is also a dedicated legacy build for pre-AVX2 Macs and Xeon-based Mac Pros.
Also we have a Official website... Yeee:

toshllm.com

ToshLLM | Local AI for Intel Macs with AMD GPUs

Run open language models locally on Intel Macs with AMD GPUs. ToshLLM provides native Metal acceleration, private chat, vision and reproducible benchmarks.
toshllm.com toshllm.com

I hope to have the money soon to notarize and register the application in the Apple Store
Click to expand...
Nice website! Wow, ToshLLM has come a long way in a short period of time. Great work!
 
I have an iMac Pro with the 16 GB Vega 64X GPU. (It also has 256 GB of RAM, but as I understand it, that wouldn’t help with LLMs on this model.) I don’t know much about local LLMs, but thought I might be able to learn/explore with this system. Would this work on an iMac Pro?

Keeping on topic for this subforum, I also have a cMP and a couple Cylinder/Trashcan Mac Pros, but I don’t think any of them would be useful for LLMs.
 
Thanks a lot for the support and the feedback on the project guys.

Regarding @moondigger question:

Yes, the iMac Pro with the Vega 64X works great with ToshLLM. Vega architecture (GCN5) has dedicated support in the latest updates, so you will be able to leverage the full 16 GB of VRAM for local execution with native Metal acceleration.

As for system RAM (your 256 GB), while GPU VRAM is key for maximum generation speed, ToshLLM does allow offloading layers/context or running MoE (Mixture of Experts) models using system RAM when VRAM isn't enough... though generation will naturally be slower compared to running fully on GPU.

Also, a couple of recent additions worth mentioning:

  • MCP & Agents: Model Context Protocol (MCP) integration is already live, along with the first phase of local Agent support to connect external tools.
  • Fabric Link on Dual GPUs: I'm currently exploring and testing native Infinity Fabric Link support for dual-GPU AMD cards (like the W6800X Duo or Vega II Duo) to enable faster interconnect communication and smoother multi-GPU scaling under macOS.

And as @Flint Ironstag mentioned, connecting eGPUs via Thunderbolt is also a totally valid path if you ever need to scale up your VRAM.
 
  • Like
  • Love
Reactions: bzgnyc2, AndreeOnline and Flint Ironstag
Back again, and I have enjoyed trying out the latest version—such a nice step up and overall progress! Thanks!

I'm thinking about how to best utilise this local LLM to do grunt work that I can then tweak with frontier, commercial models. Still undecided on how to set that up, but it's really nice to just have Qwen 3.6 sit there and take any number of requests...

It's still a bit unclear to me how to make sure I always run the latest/best models. I guess I can visit Huggingface regularly and check manually... any other way? Would it be possible to have notifications 'in app' when a downloaded model becomes outdated?

UI request: when chatting, it's common for Qwen to generate long, detailed replies. As Qwen keeps generating its reply, the chat window scrolls and we always see the last typed word on screen. That may or may not be the desired default, but it would be cool to have the app recognise 'chat counter scrolling' from the user and then "lock the reply" in place and let the chat continue writing off screen. This would make it easier to start reading the reply immediately. It's difficult to read with the text moving.
I would probably rather have the default be: the chat generates the reply in place and any text just overflows off screen and I would have to manually scroll to read it. That would make for a calmer experience (subjective opinion).
 
  • Like
Reactions: sfalatko
Hi @AndreeOnline!

Thanks for the feedback! I'll definitely take those suggestions into account.

Regarding model updates, the code part of that is actually already implemented, but completing the rest will take a bit of time. I'm currently focused on a round of optimizations... the Vega GPU that was donated to me just arrived, so I'm working on fine-tuning and optimizing for that hardware as well!
 
  • Like
Reactions: bzgnyc2, Regulus67 and AndreeOnline
AndreeOnline said:
Back again, and I have enjoyed trying out the latest version—such a nice step up and overall progress! Thanks!

I'm thinking about how to best utilise this local LLM to do grunt work that I can then tweak with frontier, commercial models. Still undecided on how to set that up, but it's really nice to just have Qwen 3.6 sit there and take any number of requests...

It's still a bit unclear to me how to make sure I always run the latest/best models. I guess I can visit Huggingface regularly and check manually... any other way? Would it be possible to have notifications 'in app' when a downloaded model becomes outdated?

UI request: when chatting, it's common for Qwen to generate long, detailed replies. As Qwen keeps generating its reply, the chat window scrolls and we always see the last typed word on screen. That may or may not be the desired default, but it would be cool to have the app recognise 'chat counter scrolling' from the user and then "lock the reply" in place and let the chat continue writing off screen. This would make it easier to start reading the reply immediately. It's difficult to read with the text moving.
I would probably rather have the default be: the chat generates the reply in place and any text just overflows off screen and I would have to manually scroll to read it. That would make for a calmer experience (subjective opinion).
Click to expand...
I've joined LocalLLM on reddit as a way to learn. I've also taken to asking Grok, ChatGPT, and Claude about best models to fit into my VRAM - at least helps to narrow down the field.

Seems like the next big model people are talking about is Qwen3.8 supposedly being released next week with 27B version out soon there after.
 
sfalatko said:
I've joined LocalLLM on reddit as a way to learn. I've also taken to asking Grok, ChatGPT, and Claude about best models to fit into my VRAM - at least helps to narrow down the field.

Seems like the next big model people are talking about is Qwen3.8 supposedly being released next week with 27B version out soon there after.
Click to expand...
Yes... I'm waiting for the 3.8 35B MoE to finally retire the 3.6 35B
 
A few numbers in contrast to the 3.6 model. The tensor split considerably ups the prompt performance. Running on a single GPU is quite a bit slower than 3.6. I'm running the 3.8 27B Q4_K_M right now.
 

Attachments

  • Skärmavbild 2026-08-16 kl. 12.12.21.png
    Skärmavbild 2026-08-16 kl. 12.12.21.png
    202.9 KB · Views: 25
AndreeOnline said:
A few numbers in contrast to the 3.6 model. The tensor split considerably ups the prompt performance. Running on a single GPU is quite a bit slower than 3.6. I'm running the 3.8 27B Q4_K_M right now.
Click to expand...
Remember that you're comparing a full-density model against a MoE... they're never the same... if you compare it to the 3.6 27b, you might get the same values... MoEs don't execute all instructions at once, that's why they're usually faster. Also, while decoding with tensor is a bit slower, which is something I've been investigating, it does significantly increase the prefill time.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back