Going from a 6 to an SE shortly before the September press conference, I can safely say that I don't want a phone much bigger than the SE ever again. For what I use my phone for (music, light texting, a few photos & Uber), I don't do anything that a bigger screen enhances. At this point, I'm sending more and more texts via my Watch than my SE at this point (the vast majority of it is still done on my Mac though). Even as someone that can comfortably palm a basketball, I found the 6 to be just a little too large and awkward to hold one handed.
Judging for the sales and customer satisfaction with the SE, I think it's safe to say that a 4 inch phone will be made again at some undefined point in the future. The only thing I'd want in an SE 2 is an iPhone 7 quality screen. The Duel Domain pixels, polarizer & P3 color reproduction really does make a difference. If they can engineer that in a 4 inch phone that has the same battery life as the SE in my pocket, I'd be running to the Apple Store right now for it.