MP 7,1 Local LLM on Intel and W6800X Duos

[AndreeOnline]

On a whim, I searched if there was some work being done to leverage the W6800X Duos for AI work. Felt like a long shot, but I was surprised to see that there is a project that shows some promise.

GitHub - Basten7/iRon-Llama: Fork of llama.cpp, a customized LLaMA inference setup optimized for Mac Intel and Radeon like w6800X Duo

Fork of llama.cpp, a customized LLaMA inference setup optimized for Mac Intel and Radeon like w6800X Duo - Basten7/iRon-Llama

github.com

Seems relatively straightforward to setup, but I haven't tried it yet myself, as I just learned about it. Has anyone here checked it out or tried something similar?

I have no illusions of my Mac Pro being a 'monster setup' for LLMs, rather is a case of "this might work, and it might add some value to existing hardware".

I guess I was mostly pleasantly surprised that someone is actively developing it.

 

[tsialex]

Did you already check ToshLLM?

The main work with local LLMs and 2019 Mac Pro is being done there and is not just one model.
I'm really impressed with the developer, for several days he is releasing multiple updates of ToshLLM daily.

https://github.com/engeldlgado/toshllm

Btw, this is the thread for ToshLLM:

https://forums.macrumors.com/threads/llms-on-intel-macs-with-amd-gpus-in-macos-is-here.2484100/

 

[AndreeOnline]

No, but I'll check it out—thanks!

I really didn't expect anything when searching for it, but cool that someone is at it!