Games will play fine. I was able to play C&C Red Alert 3 (in native resolution but all features set to their lowest settings) just fine on my Macbook (Aluminum 2.4Ghz with Nvidia 9400m, 2GB of RAM at the time) while in Boot Camp (dual booting) in Windows Vista. No lag, no slow downs, no crashes.
If you get the 24" iMac with the ATI 512MB card, you will be JUST FINE for all modern games as far as gameplay. You may not be able to max out resolution and features, but they will work and be playable just as they would be on a PC.
Even the 120 and 130 cards from nVidia should run most games just fine. The 9400m which is the integrated card on the mac mini, macbooks, and the entry level iMacs is not going to do so well, though. It will run most games including modern games, but you will have to really tone down the resolution.
If you are a casual gamer as you said you are (a couple hours here and there) then you will be just fine with a mac, don't worry!
And guys, let's not keep repeating this "macs aren't good at games" crap, they really are just fine. Yeah, high end PCs (and DIY ones) can run circles around macs in gaming when resolutions are cranked up to the max but it doesn't mean games will totally suck or be unplayable on macs, especially the current lineup.