My personal opinion is that overclocking in an all-in-one enclosure is a risky idea and you should avoid it. The heat tolerances are much tighter in an iMac than in a standard desktop case. The general rules of thumbs that apply to overclocking a PC add-in card in a relatively open and ventilated desktop case cannot be carried over to overclocking a GPU in the tight spaces of an iMac. There's a reason why Nvidia underclocked the 680MX in the iMac in comparison to the 680 used in desktop add-in cards - heat! ...
I'm not saying that no one has ever overclocked their iMac successfully but
First, some background.
I've owned and successfully overclocked the 2010 iMac making many games play a solid a 60 fps that otherwise wouldn't. I then sold that after a year and bought a top-of-the-line 2011 iMac and did the same thing. (I have a thread about it on these forums). Neither time did I
ever experience any problems and I would play sometimes for 8 hrs straight when I had free time when the wife was out of town. As the poster before mentioned, you just have to be careful. It may indeed void your warranty, though knowing apple, I'm sure as long as you're not messing with hardware itself, they probably would fix it anyways. But I would buy the AppleCare protection just in case. Many many PC overclockers scoff at the notion of being able to overclock an iMac, but they're wrong. While you may not be able to get dramatic gains like in a thoroughly cooled PC, an iMac can absolutely be safely overclocked assuming the following:
1) You're just overclocking the GPU. Don't try a CPU and a GPU overclock. Although it may be possible, I've never attempted it and so can't speak to its safety. As this guy said, ambient temperature could certainly affect other components.
2) You don't attempt to mess with the voltage at all. Most desktop overclocking software lets you apply more voltage to the chip -- this would be HIGHLY RISKY in an iMac, where voltage requirements are considered very carefully by the engineers and usually the power supply unit (which determines the amount of juice the entire machine will draw from your wall outlet) usually takes just enough power to supply all the components under full load.
3) You mind your temps!
4) You mind your expectations! Don't expect crazy overclock numbers. We're talking a couple of hundred MHz on your memory clocks, and probably only a hundred MHz on your core clocks if the past 2 generations of ATI Radeon iMacs are any indication. That's probably it.
5) Go slow! As the prior poster said, only go up 20-50 MHz at a time. And start with memory clocks as these tend to have less of an effect on temperature than core clocks.
Just crank up the fans
before you do the testing, and err on the side of cooler temps. There is GPU overclock headroom because there is cooling headroom! Usually, mac fans seem to come on much later after the temps have really started spiking, which leads me to believe that either A) the engineers are stupid or B) the mac is designed to allow higher operating temperatures than in a typical desktop environment (in the 90ºC degree range). So you're not going to be seeing temps in the 30-40ºC range like in a custom built gaming rig. I personally don't like seeing my temps hit 90ºC under any circumstances - in fact, usually my temps on the previous iMacs were in the 60s and 70s. So I would err on safety first, especially if you plan on extended gaming sessions -- I find, e.g., "just one more game" on Black Ops 2 is never that. A fan is a lot cheaper to replace than a mobile GPU!!!
Get SMC fan control for bootcamp/windows, it's a simple app let's you crank up the fan speeds ... it just requires some basic ms-dos knowledge. I usually crank up the fans to near max.
E.g., on the mid-2011 iMac, max fan speeds are 2600, 5000, and 2600 rpm, respectively. I would set the fans to 2500, 4500, and 2500 rpm.
I downloaded SMC fan control (the .exe file is called macfanx64.exe). I would unzip the download and rename the folder 'macfan' and place it in the desktop. Then, I would enter the dos command window through the start menu. Type in cd\{insert name of where you put folder}. (e.g., cd\users\YOUR WINDOWS NAME\desktop\macfan. Then type macfanx64 2500 4500 2500. Then I would proceed to get my overclock on with Sapphire Trixx.
Haven't yet gotten my 2012, but I'll be sure to start a new OC'ing thread once I do! If you'd rather wait a month or so for that thread, then be my guest, otherwise, feel free to play, and please share you experience if you do!
If you have any additional questions about overclocking, just google 'GPU overclocking guide'. You'll find ample resources. Read a few and learn as you go. And have fun tinkering!