If you only use one application at a time and exit it before moving onto the next it will not make a huge difference.
If you are like me and have Web Browser, Email, iTunes, and what ever I am working on (say Excel) open all at once - it will speed things up noticably. Not a huge change, but noticably IMHO.
BTW memory swap is a piece of cake. Getting the top of the mini off the first time is a pain. Second time it is actually easier than most of my other computer gettting into the case. Memory DIMM is right there when the lid if off.