I think a 5.4" iPhone would fit the bill as an SE replacement so long as its full screen, or as near as full screen as Apple can make it.
My only concern would be the size of that necessary notch for Face ID. The smaller the display the larger that notch becomes proportionately to the screen and the smaller the space they have left and right to fit things as they are now. It is this that makes me feel that there has been no smaller iPhone to date. The big question is when. People are getting tired of waiting and I stopped waiting for an SE2 last year which is the only reason why I'm using a Samsung Galaxy S10 today.
S10 is even bigger than XS, never mind SE....so your logic leaves me dumb founded ....clearly u moved to samsung for other reasons, so your post is just a load spin!