What is the point of an ultra-mobile laptop (stress of "mobile", meaning not plugged to AC) if you run out of juice in only 3 hours of real-life usage?
One could also look at it this way: what's the point of a long battery life if it's too big to have on you all the time? (Kind of like "the best camera is the one you have with you.")
I have an 11". Hubby has a 13". Mine is definitely, noticeably more portable. It goes places (even if it might eventually run out of batteries in the course of a long trip) that a 13" simply would not come with me. I do not have to think about whether this thing comes with me because in my case, it fits in my (regular-sized, every day) purse and is not a pain to simply have on me.
It really does depend on how you're going to use the machine. I would argue that for the Airs, that's potentially even more important than what you're going to use the machine for.
Are you going to spend long stretches in places where you have access to an outlet if you need it? (home, coffee shop with plentiful outlets, office where you can plug in, airport lounge, business class flight) Are you going to be using wifi all the time, or do you have things to do on the machine that don't require it? Are you going to be places or using applications where turning the brightness down isn't a big deal? Do you use the machine constantly for hours at a time, or do you use it a bit, put it away or a bit, then take it out again? If you get 3 hours battery life, will you use that up all in one sitting before you have access to power again? Or, will you be away from power for 6 hours but only on the machine half the time?
I find that it's answers to the above questions that can help determine whether you can live with the limitations of an 11" machine's battery. If you think you can get away with it otherwise (smaller screen, smaller SSD, less power to run certain programs), then you probably can. But I find the Air is really about how you use it day to day.