Don't use HDR all the time. It's not meant for a lot of conditions and only really shines in a few.
You wouldn't just leave the flash on regardless of conditions. Or keep any other setting on that has to do with exposure for EVERY photograph. HDR is no different.
Well, HDR is different because you get to keep both shots. But I get your point. If you understand photography well enough to know when it may be an advantage, then use feel free to toggle it on and off. Otherwise just leave it on and delete the one you don't want (unless you don't like the 5 sec processing delay)