I don't think Apple ever used GDDR5 in their cards at any point in the past. 4850 used in the iMac right now has a variant with GDDR5 but Apple chose to use the one with GDDR3. Its the same with the amount of VRAM as well. While other companies are moving to 1.5 and 2 GB VRAM Apple is still stuck at 512 Mb tops. That was great like 5 years ago. So dont expect Apple to offer a card with 1 GB DDR5 anytime soon. Hence the 57xx series wout GDDR5 is really not better than the 5650. The main reason Apple would choose the 5650 is the extremely low TDP considering the performance. And you are right 330M really sucks...I mean it is rebranded almost 4 year old technology...
Just because Apple hasn't used it in the past, doesn't mean that they never will. Apple didn't use quad core CPUs in iMac before, now they do. GDDR5 is the future, I'm sure Apple will use it in high-end, otherwise they shouldn't even update it.
VRAM is nothing more but a marketing trick. Mobility 5650 cannot take advantage of all 1GB, heck it can barely utilize half of it, especially with resolution like 1600x900. iMac has such great resolution thus it needs more VRAM, it's currently a bottleneck. If Apple wants to get rid of the bottleneck, they must use more VRAM.
57xx is better than 5650, no matter what GDDR it uses. They are all based on "Redwood" chip but obviously, 57xx has higher clocks. The TDP is irrelevant in iMac as it has no battery. All 57xx have lower TDP than 4670 thus, heck, 5850 has 39W, that's 4W more than 4670. Plus, 4670 was the fastest non-48xx chip so we don't know how much the 21.5" can handle, 5850 should fit there fine. There is no need for Apple to put over two times cooler GPU in iMac, it handles the heat fine.
330M ain't 4 years old. It's just renamed GT 240M with little higher clocks and 240M is based on GT216 core which is used in desktop GT 220. The 240M came about a year ago so it's not old chip, not at all when knowing NVidia's standards