I thought the same until I purchased one. While I love the Smart Covers, I actually prefer to use the device naked. The Smart Cover is really only for propping up and covering the screen when I'm walking around with it. As soon as I sit down, the cover comes off with ease.
Of course I always purchase insurance on my iPads so I don't care of I drop it, spill water on it, it gets stolen, etc.