You should be a little concerned about any technology that allows your card to be read without you taking some positive action.
Unless there have been significant developments in the last year or two, tags supporting cryptography cannot be powered inductively off the antenna, and as a result must be larger than you'd find in a credit card form factor. Therefore there is little meaningful way for the manufacturer to secure the data on the card that isn't susceptible to cloning. Even some crypto tags have been found to incorporate uselessly weak algorithms.
Implementors of technologies like this have a somewhat bad track record anyway. Chances are really good that reading the tag will present, in the clear, exactly the same information you'd get by swiping the magstripe.
But with a tag read someone who can get sufficiently close to you, such as in an elevator, on a bus, or walking behind you into or out of the store, might be able to surreptitiously read the chip while it's still in your wallet. It's just like a pickpocket, except he's just palming a reader, so you won't know the guy who just "accidentally" bumped into you lifted your debit card until the money starts disappearing.
With a little hacking one can come up with an antenna that, though it wouldn't be reliable enough for production use, could get lucky and nab a few successful reads a day from a few feet away. This sort of attack has been demonstrated to increase the read range for some short-range tags from a few centimeters to a few feet.
There's a lot of paranoia about RFID tags, some of it exaggerated and panicky (no satellite tracking), some of it quite legitimate cause for concern. The permanent solution is to whack the chip part with a hammer, which is easy enough, but really burns your bridges. I wouldn't do that myself. Ironically enough, keeping your card in a sleeve made of aluminum foil will probably provide enough RF shielding to block reads. Some people will sell you fancier sleeves, and even wallets with built-in shielding, but whether that's something you'd want to shell out money on is up to you.