My iCloud Drive seems to work fine now.
Make sure to immediately tick "Optimize storage on this Mac" after turning on iCloud Drive so it won't hickup, because iCloud Drive is indeed still broken, however, it works as long as you don't give it too much work load at once. Downloading a file or a folder at a time does the trick.
At least, it doesn't move my files randomly around anymore.
The problem that persists however, is that the "app library" of Readdle Scanner Pro (if you happen to use that) is displayed as "Documents" in iCloud Drive. Whereas the actual macOS "Documents" folder is named "Documents 2", but works just fine. macOS doesn't mess with the wrong Documents folder anymore, despite the weird and wrong naming scheme.
When looking at my iCloud Drive in Mojave, it does show Scanner Pro's folder as "Documents", however, Mojave keeps the name of the macOS "Documents" folder as "Documents" too.
I think it's not wise of Readdle to come up with such a confusing name for their app's files. And on Apple's behalf: They need to fix and polish Catalina's iCloud Drive much more.
I am glad though it's sort of usable now though.
Folder after folder, I downloaded all my iCloud Drive contents back to my 3TB Fusion Drive iMac.