If you install a standard Sapphire RX580 in my cMP 5,1 will it restart in recover mode (Option-Command-R) for running disk warrior or apple system utilities?
Yes, recovery mode works with any card that has drivers built into macOS, including the RX 580. But the correct key combo to invoke it is Command-R. Command-Option-R is for Internet Recovery which the cMP does not have.
If you install a standard Sapphire RX580 in my cMP 5,1 will it restart in recover mode (Option-Command-R) for running disk warrior or apple system utilities?
Command+R works fine. Few seconds of black screen, then as recovery mode loads it's drivers, you will have normal video.
I THINK Command-Option-R is for internet recovery mode (not sure though). This doesn't work with a cMP. They NEVER had internet recovery option. Command-R works fine however.