Gmail IMAP worked terribly for me with Apple Mail in Leopard, I switched back to POP. It decided to download all of my mail, including all my conversations that had been archived and decided to put them all in my inbox as unread. Very annoying. Also it made changed to my Gmail inbox as opposed to the versions that it downloads (with POP). I'm annoyed I will have to use IMAP for the iPhone.
When using IMAP, Mail will try to download all your messages (because it's designed to show exactly what's on the server), but you can set the preferences to download only headers until you view a message.
I have more than 3,000 messages (down from 10,000 last week, before I decided to clean it out) in my GMail account and it didn't take very long at all to get it all viewable on both my Mac and my iPhone.