For me the noise / fan is a big problem. I use the iMac for graphic design, and spend many hours in front of it. The last two generations of iMacs that I have owned, were what I would call "dead silent". The new 2012 is not.
My configuration is the 27 inch, 3,2 i5, 3TB Fusion drive model with 2GB graphic card. I'm not sure whether it's the i5 processor or the fusion drive causing the noise / fan. But it's on even in idle, so I'm leaning towards the processor.
The first machine I received in the middle of december. The replacement came early January - still with loud fan. The third I got in the end of January - and it also has the problem.
However - I've heard two base models of the 27 inch, which were silent, and the air coming out from behind wasn't as powerful as the model I have. So I support the more demanding models might have more air ventilation/fans coming out causing them to sound louder in the thin design.
I'm currently considering whether I should keep my 3rd replacement, or try getting the base model. I have spoken with Apple support many times -they have all confirmed they should be silent. However - since the machines are still new, I doubt they have much knowledge regarding whether it's normal behavior or not. I can only compare to the previous generations.