What's easier to steal? A Wallet or phone? Someone can walk by as you're holding your phone and snatch it at any time, not just in "bad" parts of the city. Now not only is your phone gone, so is your wallet. Getting your phone replace is easier than betting your phone plus your CC and driver's license replaced.
That just seems like a weird reason to not use one. Does that happen… sure but it can’t possibly be that big of a risk in most areas. Look at all the women that carry a purse of some sort. Same concept. If the purse gets snatched phone, wallet, cards, ID and all of that are gone.
If in a public area and it is a concern it’s very easy to pop the wallet off while in the pocket or a purse/bag and just bring out the phone.