I'm comparing the sound of my Retina iMac "at rest" to the sound of my MBA when the fan first kicks in. I don't think this is "whisper" quiet; maybe there's something wrong with the fan or maybe it's just meant to run loud, but if another iMac Retina owner can say that they have to strain to hear the fan than there is something wrong with mine.
You shouldn't be hearing anything above a barely audible "whisper", and it would be about the loudness of my rMBP when the fan first kicks in at a low speed. I say you're making too much of a big deal over a minuscule amount of idle fan noise.