Truthfully, boot over HDMI is flaky sometimes. It is supported and does work, but I’ve run into reliability issues. Plugging, unplugging, changing adapters, etc can help and is sometimes needed. Make sure all adapters are ACTIVE and that can also help. On monitor side, turn off all image processing features, disable HDR, etc. End of the day, mDP or DP is more reliable in my experience.
Do not have recommendations about your power situation. Personally look for 6-pin, dual 6-pin, or single 8-pin powered GPUs when making recommendations. Outside of that, external PSU may be required for the card to function properly. Your current setup might work, but may be worth looking at for troubleshooting.
Is the original GPU a GT120 or GT210? Does that GPU get you to boot correctly?
The GT 710 works in some hackintosh builds but unsure if there are confirmed reports of it working in official Mac Pros with 10.13.X in conjunction with another card. May be worth searching this forum to see.
Only keep one GPU installed for troubleshooting. Keep in slot 1.
APFS has issues with non-Apple SSDs. People have gotten it to work and are using it, but easy fix is to try using HFS+. Clone system drive to an HFS+ formatted drive with Carbon Copy Cloner and that issue can be fixed quickly and easily.