Alright, for anyone interested to find out how this turned out: I tested the XR and the XS side-by-side for three days. Almost from the first minute, I knew XS was the way to go for me. As I said in the beginning, really my only concern about the XR was the size. The XS is not that smaller or lighter if you just compare the specs, but in person, as it were, it is a huge difference. The XS is a large smart phone, the XR is a phablet. It really felt like a brick in comparison, especially if you add a case. Most importantly, I can use the XS almost entirely one-handed.
While testing, I have also come to like a couple of other things about the XS which the XR lacks. First, the display is indeed sharper and the blacks are blacker, which is nice. But really, I very much liked the LCD on the XR so this would never have been a dealbreaker. Also, the whites on the XS are indeed somewhat yellowish, which is manageable, but certainly not a plus.
Second, I am quite happy to have the dual lens camera system. Portrait mode with the XS is a whole other story than with the XR. It is not only that you can take "portraits" of everything, not only persons (which is especially nice to have if you want to take a picture of a person who happens to look in the other direction). It is also the simple fact that you use a portrait lens for a portrait and not a wide-angle-lens, which is something a photographer would never dream of. Portrait shots thus turn out really nice and undistorted. It is true the XR lens works better in low light, but this advantage pales in comparison to the disadvantages.
Third, and most unexpected, I found the XS to sound much better than the XR. The speakers on both devices are great and (almost equally) loud, but the XS produces a much more balanced and nuanced sound. The XR is somewhat heavy on the trebles, while the XS sounds natural and you can actually hear much more detail in the music. I am bit of an audiophile, so not everyone might hear this or share this impression, but I am actually quite surprised that this has not been much of a point of discussion before. Of course, there are much better ways to listen to music than through your iPhone, but sometimes it's your only option and it is nice to have a great sounding device for such occasions.
In sum, I am very happy XS user now, enjoying even more about it than the size alone. However, none of the additional benefits would have made me get the XS over the XR. The XR, to me, remains a wonderful option for anyone who is happy with the size, which is a very individual thing indeed.