If you uncheck All Mail to sync in Gmail Settings > Labels, this might speed up the download.
GMAIL'S RECOMMENDED IMAP CLIENT SETTINGS
Sending:
Do NOT save sent messages on the server. If your client is sending mail
through Gmail's SMTP2 server, your sent messages will be automatically
copied to the [Gmail]/Sent Mail folder.
DO save draft messages on the server. If you want your drafts in your mail
client to sync correctly with Gmail's web interface, set your client to save
drafts to the [Gmail]/Drafts folder.
Deleting:
Do NOT save deleted messages on the server. Messages that are deleted from
an IMAP folder (except for those in [Gmail]/Spam or [Gmail]/Trash) only have
that label removed and still exist in All Mail. Hence, your client doesn't
need to store an extra copy of a deleted message.
Do NOT save deleted messages to your [Gmail]/Trash folder because this will
delete a message in all folders.
Do NOT save deleted messages to your [Gmail]/All Mail folder as some clients
will try to empty this folder and ultimately fail. This can lead to delayed
mail access or excessive battery consumption on a mobile device.
Junk mail and spam:
Do NOT enable your client's junk mail filters. Gmail's spam filters also
work in your IMAP client, and we recommend turning off any additional
anti-spam or junk mail filters within your client. Your client's filter will
attempt to download and classify all of your existing messages, which may
slow down your client until the process is complete.
