Mac Mail: Slow Incoming (will more ram help)

I've noticed a problem with my mac mail. The number 1 over the mail icon comes up to show i have a new mail, but upon clicking into mail to see it, who it is from and the subject line is there, but it takes 20-30 seconds for the mail to download.

This is happening across all email accounts, coming from a mixture of imap and pop accounts.

I've also noticed in my activity monitor that my amount of free system memory is very low (dropping to under 50 meg when i have things open). If i install more system memory, do you think this will solve the email problem, or does it sound like another problem/issue?

Thanks