Not the ones mentioned here. The ATI 5000 series and the Nvidia 400 series are all full- or over-sized GPUs -- in the case of the really high end (ATI 5800s, Nvidia GTX 400s) they are full-width, take up two slots of vertical space and are nearly a foot long. Those sort of GPUs would never fit in an iMac.
Even the somewhat lower-end single-slot, standard-length cards (lower numbered ATI 5000s) would be hard pressed to fit. Maybe if Apple had a heavily customised build of them to fit into an iMac in a non-standard way, though.
There is a Nvidia GTX480 mobility card for laptops. There is also a mobile ATI 5870. Both would work in an imac.