Just one last question, should I remove the 580 and put the 680 back and than install Mojave and install the the 580 back?
Mojave does not install from createinstallmedia USB installer if you have a NVIDIA 6xx/7xx GPU, you have to install it from High Sierra to another drive.
Since right now your RX 580 does not have pre-boot configuration support - this can be solved with BootROM reconstruction service later - and will make your life miserable trying to do a clean install, do it with two steps. Install High Sierra with your AppleOEM GPU or the GTX 680 if is a flashed one and then install Mojave to a second drive.
You can even install the RX 580 to High Sierra, it will work more or less poorly, but will work. Apple only really cooked the RX 580 drivers with Mojave release.