MP All Models LLMs on Intel Macs with AMD GPUs in MacOS is here

[Dolphins1972]

Hi, yes, I can confirm that this will be implemented in the future, as I tested it in the early stages of the project and it works. Of course, I used a small model, and well, it generated an astronaut with 3 arms and 4 legs haha. But yes, I am planning to do it; it will just require dedicating a bit more time than usual to make it fast and stable. Also need suggestion about what model i cant test...

The top models are as follows, IMO.

For image to image and text to image:

1. Nano Banana Pro (censored)
2. Seedream 4.5 (NSFW), 5.0 lite is not as good.
3. Kling 01 (NSFW)
4. Grok Imagine (censored), spicy version available.

For image to video:

1. Seedance 2.0 (NSFW)
2. Google Veo 3.1 (censored)
3. Kling 3.0 (NSFW)
4. Grok Imagine 1.5 (censored), but there is a spicy version.
5. WAN 2.7 (censored), spicy version available.
6. HappyHorse (NSFW)
Open AI Sora 2 (censored) is rated high but I don't use it because of the cost.
All of the above video generators generate SFX.

Video editing:

1. Kling 01 Video Edit - edit with elements and references.
2. Kling 3.0 Omni Edit - edit with text prompts.
3. Kling Motion Control - control motion with video references.
4. Grok Imagine Edit - edit videos with text prompts.

I did not mention Flux because IMO they have fallen behind the competition and have not offered any updates in quite some time.

 

[Dolphins1972]

DiffusionBee runs great, but currently uses only a single GPU.

I tried it a while ago and was never able to generate anything other then random noise. Do you have an Intel Mac with AMD GPU? You don't mention Intel in your signature.

 

[Flint Ironstag]

I tried it a while ago and was never able to generate anything other then random noise. Do you have an Intel Mac with AMD GPU? You don't mention Intel in your signature.

Maybe my sig is glitched, but yes MP 2019 with W6800x duos and Radeon Pro VII in external chassis. IIRC it picks a random GPU to use each time, but heavily prefers any Pro VII in the chassis. No way to select which GPU it runs on.

 

[Dolphins1972]

Maybe my sig is glitched, but yes MP 2019 with W6800x duos and Radeon Pro VII in external chassis. IIRC it picks a random GPU to use each time, but heavily prefers any Pro VII in the chassis. No way to select which GPU it runs on.

I tried it again to see if they worked out the bugs. I get nothing but random blotches, it looks like a Jackson Pollack painting.

 

[engeldlgado]

This is a test from the APP, it took about 2min with 8 samples.. before integrating my kernel was 7min+, theres is harware limit for 12gb gpu, i cant generate images more bigger than 1600px on a 12gb card, i made a algorimth to detect how many VRAM you have for that

 

[AndreeOnline]

My initial tests with simple promps seemed to work well, but when I asked for more demanding results, such as more complete files for app development, I got errors.

I also tried the new image generation which also could not produce an output with the default settings. Does your app collect error logs, and if so, are you interested in them, and if so, where should I send them?

I'm keeping this post short, but I'm happy to provide more details if you are interested—if you tell me where and in what format you want them.

 

[Dolphins1972]

My initial tests with simple promps seemed to work well, but when I asked for more demanding results, such as more complete files for app development, I got errors.

I also tried the new image generation which also could not produce an output with the default settings. Does your app collect error logs, and if so, are you interested in them, and if so, where should I send them?

I'm keeping this post short, but I'm happy to provide more details if you are interested—if you tell me where and in what format you want them.

Yes, you can find logs in settings (cog top right). Updates are coming fast, I installed version 0.81.46 today. You can also update within the app now, so you don't have to keep allowing it in Gatekeeper. Oh, you have to install an image model before you can generate images. Click on the drop down menu when "images" is selected.

 

[Dolphins1972]

This is a test from the APP, it took about 2min with 8 samples.. before integrating my kernel was 7min+, theres is harware limit for 12gb gpu, i cant generate images more bigger than 1600px on a 12gb card, i made a algorimth to detect how many VRAM you have for that

View attachment 2642392

I copied your prompt for comparison. I'm capped 1600 px because of 16GB VRAM. I hope to be able to utilize both GPUs so I have 32GB of VRAM, which will allow me to use to better models. This is awesome, I love that I'm able to choose which GPU to do the work. I can offload the work to my 2nd GPU thereby leaving my main GPU (connected to monitor) free. This is very helpful if the main GPU is being taxed by other processes. You're making great progress.

 

[AndreeOnline]

Got it.

Error snippet:

|###### | 9/82 - 0.00MB/s[K
|##################################################| 82/82 - 503.34MB/s[K
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.20s (read: 0.01s, memcpy: 0.00s, convert: 0.01s, copy_to_backend: 0.01s)
[ERROR] ggml_extend.hpp:70 - ggml_metal_op_encode_impl: error: unsupported op 'PAD'
/Users/runner/work/toshllm/toshllm/vendor/stable-diffusion.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp:203: unsupported op
WARNING: Using native backtrace. Set GGML_BACKTRACE_LLDB for more info.
WARNING: GGML_BACKTRACE_LLDB may cause native MacOS Terminal.app to crash.



EDIT: added a log file (not the one with the snippet above) from a failed Qwen image generation. Everything loads fine and the image generation starts and progresses quite a bit before there is 'an unsupported op' in there.

 

[AndreeOnline]

This is what I hit on larger prompts. Any way around that? I'm turning the dials in settings, but it's not obvious what else I can do, except turning everything up to the max....

"the model used all max tokens while reasoning; raise Max (how?) or disable reasoning (I'd rather not...)"

 

[AndreeOnline]

The image generation with Z-image Turbo works well for me.

The model sure seems to love that same fisherman...

 

[Dolphins1972]

Got it.

Error snippet:

|###### | 9/82 - 0.00MB/s[K
|##################################################| 82/82 - 503.34MB/s[K
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 0.20s (read: 0.01s, memcpy: 0.00s, convert: 0.01s, copy_to_backend: 0.01s)
[ERROR] ggml_extend.hpp:70 - ggml_metal_op_encode_impl: error: unsupported op 'PAD'
/Users/runner/work/toshllm/toshllm/vendor/stable-diffusion.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp:203: unsupported op
WARNING: Using native backtrace. Set GGML_BACKTRACE_LLDB for more info.
WARNING: GGML_BACKTRACE_LLDB may cause native MacOS Terminal.app to crash.



EDIT: added a log file (not the one with the snippet above) from a failed Qwen image generation. Everything loads fine and the image generation starts and progresses quite a bit before there is 'an unsupported op' in there.

I can't use Qwen because I'm limited to 1 GPU with 16GB VRAM, so I can't replicate your issue. I have "Split model across all GPUs" enabled in settings but that function is experimental right now. It does not recognize both GPUs for image generation, it appears it's only recognized in chat generations at this time.

After a little research I found that the "error: unsupported op 'IM2COL_3D", means the model (specifically the IM2COL_3D code) is not compatible with Metal GPU. I read that there is a patch issued for Qwen Image model: https://github.com/CLDawes/ggml/tree/patch-qwen-image but I have no clue how you would be able implement the patch into ToshLLM.

 

[Dolphins1972]

This is what I hit on larger prompts. Any way around that? I'm turning the dials in settings, but it's not obvious what else I can do, except turning everything up to the max....

"the model used all max tokens while reasoning; raise Max (how?) or disable reasoning (I'd rather not...)"

You can raise tokens in settings under "context".

 

[AndreeOnline]

You can raise tokens in settings under "context".

Thanks, but I've pretty much maxed everything out in settings.

 

[engeldlgado]

Thanks, but I've pretty much maxed everything out in settings.

There's a setting directly on the chat at the bottom left that let you raise the response size.

I can't use Qwen because I'm limited to 1 GPU with 16GB VRAM, so I can't replicate your issue. I have "Split model across all GPUs" enabled in settings but that function is experimental right now. It does not recognize both GPUs for image generation, it appears it's only recognized in chat generations at this time.

After a little research I found that the "error: unsupported op 'IM2COL_3D", means the model (specifically the IM2COL_3D code) is not compatible with Metal GPU. I read that there is a patch issued for Qwen Image model: https://github.com/CLDawes/ggml/tree/patch-qwen-image but I have no clue how you would be able implement the patch into ToshLLM.

i will check this this week. Also check the multi gpu support for images, but i cant test this myself because i have only one GPU...

 

[Dolphins1972]

There's a setting directly on the chat at the bottom left that let you raise the response size.



i will check this this week. Also check the multi gpu support for images, but i cant test this myself because i have only one GPU...

I'd be glad to help. I can test for you and send error logs to you. Can you implement that into the app so logs can be sent directly to you rather than posting them in forums?

 

[engeldlgado]

I'd be glad to help. I can test for you and send error logs to you. Can you implement that into the app so logs can be sent directly to you rather than posting them in forums?

Perhaps I'll implement it in the future, but I need to think it through carefully in terms of transparency, because some people might find it uncomfortable, as they wouldn't know exactly what they were sharing in terms of privacy, etc. That's why for now I prefer to keep it this way, since people know exactly what they're sharing or what they want to share. That's why from the beginning I made the app without telemetry, without anything that makes an internet connection, only when you download a model from Hugging Face

 

[Dolphins1972]

Perhaps I'll implement it in the future, but I need to think it through carefully in terms of transparency, because some people might find it uncomfortable, as they wouldn't know exactly what they were sharing in terms of privacy, etc. That's why for now I prefer to keep it this way, since people know exactly what they're sharing or what they want to share. That's why from the beginning I made the app without telemetry, without anything that makes an internet connection, only when you download a model from Hugging Face

That's understandable and commendable. I have no problem submitting logs via PM.

 

[AndreeOnline]

Hey, engeldigado! We all appreciate the hard work and many would be happy to "help" even if it might be hard to implement a workflow that is actually useful to you and not just overhead. At the very minimum, consider your work appreciated! 🙏🏻

Looking into the models a bit more, this comparison of Flux models is pretty revealing and INSTANTLY made me want to have access to to Flux 2 dev in the dropdown list:

Flux 1 Schnell vs Flux 2 Dev: Speed vs Quality Comparison - ImageGPT Learn

Compare Flux 1 Schnell and Flux 2 Dev AI image models. Detailed analysis of speed, quality, cost, and features to help you choose between ultra-fast generation and higher quality output.

imagegpt.cloud

Any chance of that? 🤗

Apart from that, ZIT renders images reliably and stable, with good prompt adherence for me.

 

[AndreeOnline]

For comparison, I rendered out the same image prompts in Z-Image Turbo in 1:1 format. It stacks up well and is closer to Flux.2 dev than Flux.1 schnell. More could of course be done by adjusting the prompts, since I think ZIT benefits from slightly richer, more explicit prompts.

 

[engeldlgado]

Hey, engeldigado! We all appreciate the hard work and many would be happy to "help" even if it might be hard to implement a workflow that is actually useful to you and not just overhead. At the very minimum, consider your work appreciated! 🙏🏻

Looking into the models a bit more, this comparison of Flux models is pretty revealing and INSTANTLY made me want to have access to to Flux 2 dev in the dropdown list:

Flux 1 Schnell vs Flux 2 Dev: Speed vs Quality Comparison - ImageGPT Learn

Compare Flux 1 Schnell and Flux 2 Dev AI image models. Detailed analysis of speed, quality, cost, and features to help you choose between ultra-fast generation and higher quality output.

imagegpt.cloud

Any chance of that? 🤗

Apart from that, ZIT renders images reliably and stable, with good prompt adherence for me.

Hi, there, thank you for you words! I hope you're doing well. Regarding the models you suggested, I'll verify their compatibility with the system. If I can't verify it myself, I'll ask you to test them. Regarding the Qwen error, I've been investigating, and yes, it's an instruction incompatibility issue with the GPU. It shouldn't happen if you enable VAE+CPU, but I can't test it because the model doesn't fit the GPU. I would also need to write an instruction kernel to make that model compatible with RDNA+, which could take some time. Additionally, I'd need to find a smaller model that handles the same instructions to implement the most efficient fix possible, without working blindly.

 

[AndreeOnline]

I'm pretty happy with ZIT for now and it feels like a gift to be able to do "meaningful", current things on these Mac Pros and GPUs. It's been kind of sad since Blender gave up on them and the only thing left seems to be DaVinci Resolve, which is still good, for what it is.

Other questions I have directly related to your app: I tried to duplicate the app and rename it, with the hopes of running a second instance. It complained about the port 8080 already being used, but when I changed it to 8081 in my second instance, it turned out to be a global setting common to both running instances. Can I hack this to make it work?

When generating images, I'm only using one of 4 GPUS. It would be cool to be able to send a different random seed to another GPU (or three other GPUs) to generate up to 4 variations of any image prompt!

 

[engeldlgado]

Other questions I have directly related to your app: I tried to duplicate the app and rename it, with the hopes of running a second instance. It complained about the port 8080 already being used, but when I changed it to 8081 in my second instance, it turned out to be a global setting common to both running instances. Can I hack this to make it work?

When generating images, I'm only using one of 4 GPUS. It would be cool to be able to send a different random seed to another GPU (or three other GPUs) to generate up to 4 variations of any image prompt!

It's not possible at the moment, but I understand what you're trying to achieve. This doesn't happen with the regular text server, but it does with generation. In a future version, we'll add the functionality to use different GPUs with different ports or seeds so you can achieve this, something similar to what's available on the regular text server.

P.S I successfully added multi-instance support, which will be available in the next version, possibly later today or tomorrow.

 

[Flint Ironstag]

@engeldlgado just wanted to say that I'm super impressed with the progress you are making with this app. I love the zero telemetry, standard disk image installer, only a single terminal command (and that will go away soon). All running simply, smoothly, plug and play on the most powerful Mac ever. This is how it should be.

And now you have opened the floodgates for Intel Macs with PCIe and / or Thunderbolt to rival or surpass the AI processing power of any current AS. At the very least many Intel Macs can now participate in the AI revolution. The smugness level of studio owners around here is about to be corrected. 🤣

 

[AndreeOnline]

P.S I successfully added multi-instance support, which will be available in the next version, possibly later today or tomorrow.

🤯 Sweet! Amazing work!