If you don't use the phone a lot, and you are used to the awesome standby battery of the iPhone 4, it'll be noticeable.
If you game on your phone a lot, neither the 4 or the 4S would last through half a day.
So I think it depends heavily on your usage. But in my opinions, the 4S is worth it because it's a lot faster and more responsive than the 4... in general. Plus I get a feeling the 4S lasts just about the same as the 3GS anyway. The iPhone 4 might be a special case when it comes to battery life.