I wish you good luck! I've been trying to get BMD's Rocky Linux image running on a 5,1 and feel pretty stonewalled. I know other RHEL distros work too so maybe you're looking at Fedora or something, but if you do eventually get Rocky up and running you should document your process.
To actually answer your question, I'd recommend a Polaris card of some stripe. They're pretty easy to source and work with EnableGOP without too much fuss. I've got a Sapphire Nitro+ 580, but I will say that the Pixlas mod is not for the faint of heart. If you can find something that runs off of the 2x 6-pin connectors on the backplane you're going to save yourself a lot of stress.
Last word of warning so that you can avoid a mistake I made: when you insert and remove any 6-pin cables, be EXTREMELY careful. I accidentally knocked off the voltage regulator and the pads underneath it without too much force so now I'm down one 6-pin connector. I basically have to run the Pixlas mod even if the card I'm using isn't too demanding.
P.S. I just also wanna add that my laptop from three years ago runs Resolve twice as fast as my 5,1 while using a sixth of the power. If you're serious about working in Resolve you're going to wasting a lot of money by using this machine, and that's *before* you start buying parts to meet the minimum specifications. It's a fun project but I really cannot recommend it when refurbished Mac Minis are as cheap as they are.
