That rubbing almost every day drives me nuts but at least my hands are soft as well)) and I never had a silicone case yet.
Saddle brown I think is the best agingleather case from what I've seen.
I don't have any issues with the silicone case. But it could be because I'm a female and don't put my phone in my pocket as much anymore. I have been using ocean blue since launch and it's worn nicely. Also have a product RED as well.