Why does it take forever to send attachments through Mail?

I've had this problem for awhile now. It takes forever to send attachments with emails when using Mail in OSX. I am trying to send a 4MB MP3 file and it's uploading at a rate of 1KB/s - 15KB/s. It sometimes even drops to zero for a few seconds before jumping back up. A lot of the time the message fails and I have to resend it. Is there something I'm doing wrong? My internet is pretty fast when doing anything else, so I don't see how it's that. This happens ALL THE TIME when I have to send a file. I have a MobileMe address by the way.

Thanks for any help!