I have the 17" Intellimac, bumped to 2GB, and the duo, bumped to 1GB.
The overwhelming majority of the time, response is very similar. The iMac does 3D graphics much better, and the mini has that 80MB hit due to integrated graphics which can slow things a lot on a 512MB setup.
For 2D stuff, I see a slight edge to the iMac in normal usage (I have them side-by-side right now). 3D stuff (all I really have is WarCraft III) shows a clear and large iMac advantage.
Specs and benchmarks aside, for most usage, I'd guess they'd "feel" the same to you if you didn't have them - as I do - right next to one another.