To answer my own post above, I've decided to do a "trial and error" on this install. And apparently had some success. (limited but nonetheless)
For reference, the RX460 that I am using is a "generic HP" card, purchased off eBay for around $70. As I said in the first post, it's 2GB. Nothing fancy.
Here is how I managed to get it to work in my MP 3,1:
1) Installed the RX460 along side the HD 2600XT (installed by itself, it did not boot for me).
2) The HD2600Xt had to be in PCIe Slot 1. The RX460 is in PCIe Slot 2.
3) Had to do the PRAM reset before I could move forward (is it required? Don't know, but I did it.)
4) Used the dosdude1 process to reinstall High Sierra from a patched USB to a new partition that I created on a spare hard drive.
5) I had to select the option to include the updated AMD graphic driver, although the dosdude1 installer didn't recommend it. Tried to install without it, but was not successful at first.
6) After a complete reinstall, reboot, I seem to have two display adapters working. I can see two displays and two desktops.
Just posting in case it helps some 3,1 user who does a web search someday. Even me, if I forget how I did it. 😊
Update mode *on*
After some further investigation, apparently High Sierra doesn't enable hardware acceleration for the newer AMD cards on a Mac Pro 3,1. I read a thread on this forum (which I can't find now) about some missing instructions in this generation of Xeon CPUs. I think if you have a Mac Pro 4,1 or higher, this is not an issue? Think that's what I read.
Anyway, the procedure listed above does let you install an RX460 card and boot into High Sierra on a Mac Pro 3,1.
But now, then you need to find a software patch for Mac 3,1, which will give you GPU acceleration. I've read a thread describing that patch. Did I bookmark it? No. If you want to go this route, do some searching and you should find a separate patch to emulate the missing instructions.
Update mode *off*
