I have had a 6S (non-plus) for the last year, and 6 before that. I'm planning to either switch to 6S+ or 7S+, so my vote was for 6S+.
The SE is appealing to me, because I loved the smaller form factor and one-handed usability, and begrudgingly came to the 4.7" 6 because I really needed an upgrade (from the 4S; in retrospect I should have bought a used 5S at the time). However, the SE is missing a few things. The most important thing is that it has the slower Touch ID, not the same as the one in the 6S. I found the Touch ID in the 6 to be painfully slow, and the 6S was such a relief, I would not go back. The other things I could live without, but I still don't want to: I like and use 3D Touch (especially that app switching shortcut!). And 3D Touch is becoming more crucial in iOS 10. And I want the largest storage I can get, and 64GB is too small when you're used to 128GB (and I'd get 256GB if I got a 7S+). An updated SE to make it on par with at least the 6S features would be appealing.
Like you, I'm not happy in the middle, and I want to go to one of the extremes. The 4.7" model seems like the worst of both worlds. Too big for easy one-handed use, and too small for two-handed use (e.g. dual-thumb typing). If I'm going to have to do contortions to use it one-handed (which I prefer over Reachability), I might as well have the larger phone. Maybe I'm wrong here, but every time I try someone's 6+/6S+ it doesn't feel that much more difficult to use. Additionally, after I got the 6S I started using my phone much more, because it's an absolute beast power-wise (the 6 was underpowered and laggy in comparison). If I'm using my phone so much, I might as well benefit from a larger screen. The 6S actually feels small to me now. Weird.