I was in the same situation, and kind of went for it on an impulse, and god the Xs is a wonder, the screen, the photo you really have the "wow effect", but i agree i didn't really want this, but i can afford it without eating potatoes so i'm enjoying as much as i can right now (may not be the same in a few years..)
Yeah, I keep getting tempted. I've had my SE longer than any other phone I've had (from memory), I did take another look at the XS in a shop last week but I noticed there was image retention on the OLED screen, something I'm sure people said was a "thing of the past" but the rounded-square home page icons were clearly visible on all light backgrounds! I know that wont likely be a problem on my own phone not displaying the same screen 24/7 but still, losing the headphone jack, the home button, the size and weight and then that on top are a lot of things going in the "cons" pile. I can certainly afford it, but at £700-800 I can forgive some things, at £1000 for a phone I expect a lot.