Delete and start over
I have had HEAPS of trouble when setting up Google's 2-factor authentication on Yosemite. Mail was sporadic, and I couldn't get the system preferences thing to work. I found a method which actually works, though I'm not sure if all the steps are necessary. Hopefully this might be useful if someone else has troubles.
1) Close all apps that are connected to google (Mail, etc).
2) Create an app-specific password from google (I only needed one for my MacBook Pro, so you can revoke any other you made for this computer). Keep the page open for a while, or write it down (destroy it later).
3) In system preferences, I deleted my google account. I had to navigate a bunch of "enter password" prompts (that always fail), but eventually it was removed.
4) Create a new google account, and enter the app-specific password. It should, after a few seconds, connect to all the services you selected.
5) Open Mail. It should have copied the app-specific password and your Gmail inbox should work. Make sure to send an email via the gmail account and re-enter the SAME password (with email, the inbox and outbox use completely different protocols and need to be authenticated separately).
6) Finally, open Calendar (if you sync to it) and it will prompt for a password. Use the same app-specific password.
It think it is necessary to reuse the same "app-specific" password, because system preferences wants to use the one password for the various google preferences. Other apps like notes don't seem to ask me for the password again (I've checked note syncing with iPhone via Google; works well, but I haven't tried Messages).