New MBP 16, 10.5.3 Have done some more testing/research and for me I've found the issue only occurs when the discrete GPU is active and the screen saver is something other than Drift. The discrete GPU becomes active if you are running an app that requires it like (Photos, iMovie, etc) or you have disabled GPU switching in the Energy Saver Prefs in which case it's always on or if you have an External Monitor connected as this I believe requires the discrete GPU to be active. So for me as long as I use the Drift screensaver the problem does not occur.
So think it's a problem either with legacy screensavers not behaving properly (Drift is new to Catalina) as I think screensavers have become sandboxed in Catalina OR a graphics card driver issue
You can determine which GPU is running by checking About this Mac and the Display tab, or getting an app called gswitch (search: gswitch macOS) Also Activity monitor , Energy Tab, has a field: Graphics card which when made available to display will show you which apps are using the discrete GPU.