I've found a lot of threads on this, so I hope I'm not repeating something that can be found elsewhere. I installed an RX 580 (Sapphire Nitro+ 8GB Dual Bios) in my MP 5,1 and it worked fine under High Sierra. It was flashed for Mac by MacVidCards, who seem to be the only people reliably turning these UEFI PC cards into EFI Mac cards, and it works like a charm. I was able to go into Single User mode and run my fav routine, /sbin/fsck -fy 
.
The problem started when I upgraded to Mojave (not a clean install). I have two monitors, one plugged into the Display Port input, and the other into the DVI-D input. Booted up, and both screens went black. Tried a few things, like resetting the SMS to no avail. I could see the progress bar for a little while on the DP monitor, but then both would go black. Then I found this page on a site called "TonyMacx86.com. It's stuff that's way above my head, i.e. using an app called DPCIManager to look at PCI hardware info, and all kinds of parameters for the RX 580. It seems that, maybe, out of the box, the ports are not configured properly to work in a 5,1. But there was a chart showing that the problems seem to all be related to using DVI-D and HDMI monitors. DP monitors work fine. So, I disconnected my DVI-D monitor, and voila, no more black screen. So, rather than dicking around with the RX 580's firmware, I ordered a 2nd DP cable which I will use (with the VGA port) on my second monitor, and it should all work fine. Will let you all know. But meanwhile, if any of you hackintosh-types out there know about this, and can walk me through using DPCIManager, please weigh in. Looks like I have to set the ports to "5" and remove something called "whatevergreen", but I ain't touchin' it without major help...
.The problem started when I upgraded to Mojave (not a clean install). I have two monitors, one plugged into the Display Port input, and the other into the DVI-D input. Booted up, and both screens went black. Tried a few things, like resetting the SMS to no avail. I could see the progress bar for a little while on the DP monitor, but then both would go black. Then I found this page on a site called "TonyMacx86.com. It's stuff that's way above my head, i.e. using an app called DPCIManager to look at PCI hardware info, and all kinds of parameters for the RX 580. It seems that, maybe, out of the box, the ports are not configured properly to work in a 5,1. But there was a chart showing that the problems seem to all be related to using DVI-D and HDMI monitors. DP monitors work fine. So, I disconnected my DVI-D monitor, and voila, no more black screen. So, rather than dicking around with the RX 580's firmware, I ordered a 2nd DP cable which I will use (with the VGA port) on my second monitor, and it should all work fine. Will let you all know. But meanwhile, if any of you hackintosh-types out there know about this, and can walk me through using DPCIManager, please weigh in. Looks like I have to set the ports to "5" and remove something called "whatevergreen", but I ain't touchin' it without major help...