You know that the mobile 5850 is based on the desktop 5770 which operate on 850 MHz, so technically you should reach 850 MHz no problem.
The question is the cooling.
Desktop 5770 is 108W chip so we are looking at pretty high power consumption and thus more heat if you OC it to 850MHz as 4850M is ~55W and 5850M is 39W. It's achievable if you speed up the fans enough but it isn't very comfortable then. However, the Juniper chip doesn't seem to go much over 900/900/1300 (core/shader/mem, in MHz) before running into issues so 850MHz is max for iMac IMO. In some cases, increasing the clock speeds by only 30MHz or so from the stock speeds caused a lot issues though this may have been fixed with driver or revision update. 750-800MHz seems to be the sweet spot, it's still not too loud but performance increase is noticeable