This pretty much confirms my suspicions about the UI.
The graphics/UI test seen here shows Yosemite getting absolutely annihilated by Mavericks. Yosemite is a clean install btw.
I also did a CPU and Memory test which was close, but Mavericks won that too.
This difference has real world effect. The UI feels very sluggish. Mavericks' UI is about 4.5 times faster!
Why is Yosemite SO slow?
Videos demonstrating this issue:
Resizing a window in Mavericks: http://youtu.be/174ugKsj6z4
Resizing a window in Yosemite: http://youtu.be/1TMh5Pjj4uk