Ultimately, the color doesn't matter too much to me, as I will be using a case.
But from several iPhone Xs that I've seen, scratches show much more easily on silver iPhone X's shiny stainless sides. Even those that were protected in a case seems to develop scratches over time.