I upgraded to the 4S from the 4, and I have definitely felt buyer's remorse. For me personally, Siri and a better camera do not make up for the truly horrendous battery. I'll likely go back to my 4, but this is the first time ever I have actually flirted with the idea of Android.
Josh Topolsky's review of the Galaxy Nexus got me interested enough to at least consider Android. Ice Cream Sandwich embodies all of the things I wish Apple included in iOS. At the very least I have to be jailbroken, and this stock 4S is killing me. I really hope post-Jobs Apple does not become stagnant and fail to innovate.
other 3GS and 4 users upgrading to iOS5 are also reporting battery issues.
My friend and I both go the iPhone4S on the same day together. Both did a clean setup with no backup sync. I did not have battery issues. I get 5-7 hours usage time while he gets 2-3hrs after a day at work with me.
Out phone setup is pretty similar. Only difference I notice is he works in a room where he gets 1bar 3G while I got 5bars. My guess is the iPhone was boosting power to get better reception or going in/out trying to re-establish connection too much draining the battery.