I was having issues when the IMAP was just launched. Some of the issues are due to incorrect config of the folders in Mail.app, meaning that it will be doing of unnecessary things if it's incorrectly configured. back then Gmail hadn't put the recommened settings for Mail.app on their website... After checking that config, everything seems to work smoother.
Give that a try; it worked for me 😀