I have an early 2009 24" 3.06ghz C2Duo iMac with 4gb of ram, and a 2011 2.3ghz i5 Mac Mini running 8gb of ram. Both are running 10.8.3 Mountain Lion. The Mini is much more responsive than the iMac. When I had 2gb of ram in the Mini, it was a slug.
8 or more gigs is the way to go