I have difficulty buying this. The 2013 rMBP has a faster GPU and double (or quadruple) the VRAM so how VRAM can be an issue? I am not familiar with mudbox, but the only reason I can imagine for it performing worse on the rMBP is that they have messed up the programming and are actually rendering to a HiDPI framebuffer. Which would be quite a silly thing to do, given the GPU speed...
Yeah, I wondered that too, but that's what i have experienced so far.
I personally do not own a 2013 rmbp but I had to use one for a couple of months for a project, so i had enough time to test and work with it in parallel with my units: a late 2012 i7 iMac and a 2012 classic mbp.
Based on specifications, the 750m is about 10%-15% faster compared to the 650m and have one extra GB of VRAM. That of course, if both cards are running stuff at same resolution, OS and software.
But here's what happened to me.
When running a 2013 rMBP Mac OS 10.9 tend to keep busy around 800mb-1,5gb out of 2gb of vram, this greatly depends on wich resolution is set up and how many monitors you're using.
When working with a Cintiq connected to the rMBP it goes out of memory quite fast and get a lot of out of memory errors when using multiple retina enabled applications such as After effects and photoshop. Multitasking Maya, Mudbox, Mari and Photoshop was simply impossibile in a multi monitor environment, i was forced to either work in clamshell mode or give up the cintiq.
just for comparison, when using the imac with the 680MX (2 GB) and 10.8 you usually have 1,7 gb available, and on a cmbp with the hires screen, mac os 10.8 and the 650M (1 GB) you have about 900 MB free. Connecting a Cintiq have a minimal impact on performance.
It is logical to me that gpu memory problems are also tied to the mac OS 10.9 improvements(!) and additions. If I'm not mistaken apple did increase the memory reserved on integrated gpus with mavericks and upcoming yosemite, probably to keep up the bigger OS vram needs.
The results for my workflow is that I cannot do production quality 3d and compositing work without compromises on the retina line, and I probably won't upgrade until we see adequate hardware for it. I rely a lot also on CUDA wich is vram dependant.