I have 8gb, but i got it when i bought my mac, and don't have any way to compare performance.
however, i have noticed that during heavy processing i chew into the extra ram, get lots of page outs (about 10% or so), and an occasional beachball. this is fine with me, because i am coiming from a netbook that wouldn't even let me type emails when it was under a heavy load. 4gb is probably enough for me most of the time, but it is nice to have the extra ram when it is needed.
anyhow, i think it probably depends a lot on what kind of tasks you are performing. the good news is that ram is a super inexpensive upgrade, so you get a lot of bang for your buck.