in the release notes there's mention that starting up Mail for the first time will take a much longer time and it will also take a much longer time to update the mailbox. I didn't notice this and simply deleted mail ~/Library/Mail folder and had it re-download everything since I have IMAP mail and its all on the server.
All my old emails (pop) disappeared. Only what's on the server was downloaded - all marked unread of course.
Tried to restore mail from Time Machine, the same thing happened. I abandoned 10.11 and reverted to 10.10.4. I'll wait for the next update.