MP All Models Radeon Pro VII Is a Hidden Gem for Local LLMs

[BetraySoul]

I recently got a brand-new, unused Radeon Pro VII for $165. This GPU has 16GB of HBM2 VRAM, providing a massive 1TB/s memory bandwidth. I have a spare Mac Pro 5,1 that dual-boots macOS Monterey and Fedora Linux. I threw this GPU in it and ran a quick LLM benchmark on Linux with default settings. The performance with 14B models is great! I think it performs close to an NVIDIA RTX 4080. It's an incredible value for the price.

I also ran a quick Cyberpunk benchmark using GE Proton at 1440p with ultra settings, FSR4 INT8 performance upscaling, and FSR3.1 frame generation, and I got a decent 85 FPS. I'm quite surprised that FSR4 INT8 works so well on this older Vega card. The image quality looks great. I thought the minimum requirement for FSR4 INT8 would be an RDNA2 card.

 

[reader50]

Where did you find a Radeon Pro VII for $165? Did they have any more?

 

[SDAVE]

It's also a heater
Used to have this GPU on my Hackintosh, memory was pretty crazy such a shame HBM is so expensive and no one is using it anymore (unless it's for AI workloads)

 

[keksikuningas]

I recently got a brand-new, unused Radeon Pro VII for $165. This GPU has 16GB of HBM2 VRAM, providing a massive 1TB/s memory bandwidth. I have a spare Mac Pro 5,1 that dual-boots macOS Monterey and Fedora Linux. I threw this GPU in it and ran a quick LLM benchmark on Linux with default settings. The performance with 14B models is great! I think it performs close to an NVIDIA RTX 4080. It's an incredible value for the price.

I also ran a quick Cyberpunk benchmark using GE Proton at 1440p with ultra settings, FSR4 INT8 performance upscaling, and FSR3.1 frame generation, and I got a decent 85 FPS. I'm quite surprised that FSR4 INT8 works so well on this older Vega card. The image quality looks great. I thought the minimum requirement for FSR4 INT8 would be an RDNA2 card.

I googled and it seems that AMD dropped support for Radeon VII and Radeon VII Pro in later ROCM versions, is that correct?

 

[Flint Ironstag]

Bruh I have 4 of these sitting in an expansion chassis and I need Infinity Fabric bridges - does your source have any?

 

[Alex Sanders74]

Bruh I have 4 of these sitting in an expansion chassis and I need Infinity Fabric bridges - does your source have any?

They're out there...Ebay has them listed for $80

 

[Flint Ironstag]

They're out there...Ebay has them listed for $80

I'm only seeing them for the Instinct cards, which are incompatible with the Radeon Pro VIIs. If I'm missing it - please send link!

 

[Alex Sanders74]

I'm only seeing them for the Instinct cards, which are incompatible with the Radeon Pro VIIs. If I'm missing it - please send link!

Apologies, the one I saw was for a Vega Duo. Went down a rabbit hole on these and the model you're after is hard to find. Very specific part and nobody seems to have one....

 

[Mac3Duser]

I have an AMD Radeon Pro Vega II with 32 gb HBM2, do you think it is possible to use it for local LLM and how ?

 

[BetraySoul]

Where did you find a Radeon Pro VII for $165? Did they have any more?

I got it from Mercari. The seller had two new cards available, and I bought the last one.

 

[BetraySoul]

I have an AMD Radeon Pro Vega II with 32 gb HBM2, do you think it is possible to use it for local LLM and how ?

Both Ollama and LM Studio require Apple Silicon for macOS, so you'll need to install Windows or Linux. I've never run a local LLM on Windows before. Personally, I prefer Linux due to better GPU driver support. You'll probably need to do some research if you've never installed Linux on a Mac Pro 2019 before. The T2 chip can be a headache. CachyOS has built-in T2 support based on T2 Linux.
My W6800's 14B performance is slightly worse than my Pro VII's, but its 32GB of VRAM allows it to run larger models with decent performance. I think your Pro Vega II MPX should perform similarly to the Pro VII, and its 32GB of VRAM can handle larger models with probably better performance than the W6800. Ollama's official "Hardware Support" explicitly lists Vega II as supported.

 

[keksikuningas]

I have an AMD Radeon Pro Vega II with 32 gb HBM2, do you think it is possible to use it for local LLM and how ?

https://www.reddit.com/r/macpro/comments/1q9xeov/guide_mac_pro_2019_macpro71_w_linux_local_llmai

this guide should work, don't have a 7,1 myself though

 

[Flint Ironstag]

Both Ollama and LM Studio require Apple Silicon for macOS, so you'll need to install Windows or Linux.

You're always telling people to run Linux. @Mac3Duser , plenty of folks on reddit running LLMs on MacOS and getting results. Check there.

 

[BetraySoul]

You're always telling people to run Linux. @Mac3Duser , plenty of folks on reddit running LLMs on MacOS and getting results. Check there.

Dude, @Mac3Duser explicitly asked whether he could use the AMD Radeon Pro Vega II to run a local LLM. Ollama and LM Studio just do not support macOS on Intel Macs, so he would need to install either Windows or Linux on his Mac Pro. For running a local LLM, Linux obviously has better support.

 

[keksikuningas]

You're always telling people to run Linux. @Mac3Duser , plenty of folks on reddit running LLMs on MacOS and getting results. Check there.

there's no AMD support on Intel Macs. Ollama will only run on CPU and kobold vulkan support doesn't work on macOS either

 

[Mac3Duser]

Yes, I love Linux, but I never tried to install linux on my 7.1 Mac Pro.