The 11" has plenty of downsides.
Lower resolution, lower battery life, etc. Most annoying for me is the lack of built-in SD card reader. I borrowed a friend's MBA for a trip (great friend) but was annoyed to no end in having to get an external USB SD card reader which didn't fit in his custom MBA carrying bag. Speaking of carrying bags and cases, more generic choices are available for the 13" MBA.
Yeah, the lack of card reader is a downer. I mean, I have an external card reader, but I don't like the fact of HAVING to use that. I take a LOT of photos (3 kids), and one of my main reasons for a MBA is to view (and possibly edit) photos when I am not home.
I was also considering getting rid of one of my iPads for the Air, but they are totally different machines with different uses, so I will end up having both.