Buying used M1 Ultra for Local LLM?

[M3Stang]

I currently have a 14” M4 Max MacBook Pro with the gimped memory bandwidth and cores (14/32). I am running 15-20GB of swap when running models with Xcode. It still runs decently quick but it’s still not great. I do have a server with a 3090 Ti which is significantly faster but uses a ton of power. Was looking at getting a used M1 Ultra to use as my “new” Local server with at least 64GB ram.

Would it be worth to spend $1500-2000 on an M1 Ultra for this purpose being I already have an M4 Max/3090 Ti? I know the 3090 Ti has about 1TB of bandwidth and the ultra has 800 so it’s closer. Way better than the 400 something my M4 Max has.

Wondering what the community thinks. Thanks!

 

[rb2112]

I just got out of that same boat myself. Ran a bunch of models/prompts with my current config, then extrapolated to think of what I would want, and be fine for a few years.
Here is a helpful chart of the performance of Apple Silicon M-series chips on a small LLM.

https://github.com/ggml-org/llama.cpp/discussions/4167

I'm trying to remember what I though about price vs. performance (inference speed) vs. RAM (for model size). Based on my needs, I thought that:
"good" tier: rather than get a M1 Ultra 48-core, and M4 Max 40 core would be better.
Then if I wanted to spend "better", M1 Ultra 64-core = M2 Ultra 60 core = M3U 60c, with M1U being preferred (for price) *unless* you had a particular model that benefitted from 192GB RAM vs 128GB vs 256GB.
Finally, if you just needed max available at the moment, M3U 512GB.

So to answer: depends on what you are doing with LLM...
Your M4M 32c should perform well. If you encounter limitations based on usable model size based on your 36GB RAM (which is what I found on my M4M Studio), :
M1U 64GB does not really get you much more than your M4M, in my opinion, *this really depends on the models you need!*
M1U 128GB 64c would be great at the price you quoted, but that config is (today about 2500 USD from Apple Refurb). If you were willing to spend 2000, I would say stretch a bit for the M1U 128GB 64c. An ebay listing that sold on Oct 14 sold a M1U 128GB 48c for just under $2K, at that point you're faster than your M4M and have plenty of RAM. I think 1.5-2k USD for M1U 128GB 48c should fit you nicely.