If you don't really care about cosmetic damage, it means that the iPhone's appearance is not a priority for you, so why not keep that case on anyway, in order to play it more safe?
Usually people who want to go naked, do it because they want to enjoy the beauty of their device and feel its materials, so they take the risk to leave it without a case.
If that specific case makes the device feel too bulky and heavy, you can always buy another type, that will at least offer some protection.
I agree with the statement that the back glass is more fragile and sensitive compared to the rest of the phone and I have held iPhones with severely shattered back glasses. Trust me, they do not feel comfortable in hand, so for me, it's more than just inner electronic compoments damage that you need to take into consideration...