I found more info on this. It took me some time to go though a lot of threads but these guys explain the behaviour and causes pretty well:
https://forums.macrumors.com/showthread.php?p=16215038#post16215038
Hope it helps!
I'm a lot more comfortable with the issue right now. From what I'm gathering, there will be an update to OSX that will solve the "issue".
----------
This is exactly what I was thinking... best I can remember (haven't played with an external recently) it's not even like you could use both with itunes in full screen mode when using an external display.
I'm never going to run iTunes that way, but noticed it once when the full screen went on automatically due playing a video in my library. CTRL-1 to switch to my first desktop where I was working on some stuff in excel. That's when I noticed the performance issue. As I've never seen this before on my old 17" MBP I questioned the graphics card. So I started doing some QA on it
Pushed it to the limits to see what's going on. As I can tell so far it's a known issue with the retina and others have explained the causes pretty well.
https://forums.macrumors.com/showthread.php?p=16215038#post16215038