This has been the case for a LONG time. Nvidia's drivers for MacOS have always been third-rate. It sucks because the GPUs themselves are decent; it's just that either Apple or Nvidia (or both) just can't write good MacOS drivers for them.
I've had three desktop Macs with GPUs and the only ones that have had graphical glitches and performance issues have been the Nvidia-based ones. AMD or Intel GPUs are the way to go with MacOS, period. These issues date back to 2006, too, which is why it's so maddening that Nvidia drivers are STILL not sorted out on MacOS. Twelve years.
- 2006 Mac Pro (came with an Nvidia 7300GT, absolutely garbage and glitchy. Radeon X1900XT and later 3750 GPU swaps fixed it)
- 2012 iMac with Nvidia GT 650M. Mostly stable but performance was lacking. Would crash in some games that exercise the GPU. Had all kinds of problems with High Sierra.
- 2015 iMac with AMD Radeon R9 M395. Absolutely perfect GPU performance. No issues at all.
I'm going to make every effort I can to never buy a Mac with an Nvidia GPU again. Lesson learned!