95% of the time, I don't use a case.
I tried silver, and I loved the back: it was so much prettier than my silver X, and I love how the camera cutout looks. However, I didn't like the shiny stainless band: it kept glaring out at me, and it's only shiny when you take it out of the box or for a brief instant after wiping it off. The rest of the time, it's just a smudge-fest. The darker colors are smudgy, too, but they don't show very much. Also, I'm not looking at the back very much even though I carry without a case, and the appeal wore off for me during the 14 days. If you use a case, you have more color options with silver.
I settled on Midnight Green: I really like the color, and in some lighting, it looks gray, so I feel like I have two colors. I had a gold XS Max, so this is a nice change.
Both are beautiful.