How far into the future do you want to peer? I think the near-term answer would be no. The fact that the new SE has a larger display than the previous SE is a good hint.
Overall, users seem to prefer larger displays, even though there are certainly those who would line up for a smaller model if it was available. Not only can larger displays be easier to read, but touch actions don't have to be as precise. Part of the point of edge-to-edge displays is to maximize display size while minimizing physical dimensions.
Overall, if one is willing/able to deal with the larger physical size one gets greater ease of use. Basically, a trend back towards smaller size means a significant number of users has tired of the larger physical size and are willing to trade that for usability, and/or there are some improvements in smaller-device usability (UI changes, etc.). Folding displays has been one attempt at reducing physical dimensions while maintaining or expanding display size... so far that approach has had problems, but who knows what the future brings?
So in 5 years, who knows?