I went for the X then XS Max is plain stainless steel for this very reason, but then braved it with 11 Pro Max and getting it in Space Grey.
I've had it in mainly two cases, Ringke Fusion-X and Spigen Gauntlet and when I take it off every few weeks to clean it's always been in pristine condition.
I dare say if you work in construction or similar then you may have to maybe be a bit more wary, but otherwise I don't you're have the problem you're worried about. 🙂