For me it's mostly the frustration of mail problems, battery life, and it's glitchy.
Yesterday, I looked down at the phone because the lock screen lit up, and for some bizarre reason it was showing the welcome screen you get when you first restore. The screen went black again, but when I pressed to home button, my normal screen was there.
On another occasion, the phone decided to dial out by itself. Got no idea why. It just did.
Here's the thing... it works for the most part. But it's the unpredictability that has me regretting this upgrade. I don't use maps, so that's not an issue for me.
I use the phone mostly for work, hence email has to be rock solid. One annoyance is the Swoosh before SMTP bug. You have no way to know if the email was actually sent because the sound plays when you press send, not when it has finished negotiating with the server.
I also noticed that with mail, you get the ding sound to say there's new mail, but it doesn't show up on the lock screen like it should. Takes forever for it to show. iOS5 worked perfectly in all these regards.
I've emailed and fed back to apple, but a deafening silence returns.
I'm telling you, I'd happily pay to roll back to iOS5. But cannot. The only way to get back to iOS5 on a 4S would be to buy another phone that was not upgraded.