The problem I have with that argument is that it just flies in the face of the physical geometry of the hand and the contexts and environments many of us actually use our devices in.
Just making something bigger doesn’t make it better for everything and everybody and when the solution is to just “move on“ it’s ultimately a pretty crappy solution.
Apple built a very flexible app design architecture specifically to address multiple sizes and I really wish they would use those and at least keep something at the bottom with maybe a 4.5” screen which, with reduced bezels, would be an acceptably small physical size for SE fans also.
You do have a point about hand geometry. My iPhone 4s was great, I could use it single-handed. My S7 is about at the limits of single-handed operation, but it's better with two hands, and when I get the Note 9, that will be a two-handed affair.
The problem is that the whole ecosystem has moved to larger screens. Websites, apps, etc, are all designed for a 5-6" screen now, as that's what most phones and most people have. The Android battery issues originally pushed people to larger screens, but now it's the whole ecosystem and the idea of a pocket computer as opposed to merely a smartphone. Further, in many developing markets, and even low-income markets in the US, people's primary or only access to the internet is through a relatively low-cost smartphone that is really a pocket computer, so a larger screen is better, even if it's not high resolution.
I think Apple has too much fragmentation in their lineup. They do need a somewhat smaller device, but I think the SE form factor has come to the end of the line, and the future of relatively small phones is in the iPhone 8 form factor. I think Apple could have gotten many more years out of that form factor, and then done an XS and XS Plus, but instead they decided to do a full-screen notch phone as their base model, and make it comically large at that. I find the larger iPhones to be sort of pointless. You do get the battery battery and camera specs, but you don't get any unique features like the Galaxy Note 9, which packs in the S-Pen in addition to industry-leading specs in every single other category. I think the iPhone X design is the future of the iPhone, but I think Apple is pushing into it too fast, and they should have kept the entry-level device at the $650 price point, and should have kept it in the more "traditional" form factor, at least for a couple more years. That being said, I think the iPhone 4/4s/5/5s/SE basic industrial design and form factor, particularly the original size of the 4/4s are the best design that Apple has ever made in an iPhone.