Well, I did it. Yesterday I made an impulse buy and bought an AT&T Prepaid iPhone SE from Best Buy for $150. For that price, I figured I wouldn't sweat it and could try out living with a small phone again. The SE is sort of a sideways move from my iPhone 6s. Mostly the same guts (aside from the older-gen Touch ID sensor - which seems to work just fine). Other than that, the only real/noticeable difference is obviously the smaller size and smaller/slightly-lower-res screen. Same PPI, though.
I had to shuffle some apps around this morning to get my most-used stuff to fit on my home screen, since I've lost one row of 4 icons.
My other concern was whether I'd be able to squeeze down from 64GB to 32GB. I think I was using a little over 40GB on my 6s, but much of that was for photos. I pay for iCloud Photo storage, so I'll just lose some cached local photos. I don't store movies, music, or podcasts locally, I stream podcasts and music, so that's not a concern.
The phone feels great. Maybe the one usability downgrade is the smaller keys on the keyboard.
So, I'll play around with this for a week or two and see how I feel. If I decide the smaller screen is too big of a negative for me, I may decide to splurge on an iPhone X. But if I'm happier, I'll send a feature request to Apple asking them to make an edge-to-edge SE-sized X variant ASAP.