As a $2T company, Apple went in for 2 years into the small phone market. They had the entire market to themselves. Yet, Apple left the market without anybody copying or following. That should tell you a lot.
If you were a product manager walking into Tim Cook's office, would you ever want to re-enact failure of the 5.4-inch iPhone? And propose it be used for iPhone SE? It never reached economies of scale like 6.1-inch models either.
"People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying no to 1,000 things." - Steve Jobs
Apple wants iPhone to be your camera, wallet, streaming video player, handheld gaming console, safety device, etc. Having a small iPhone is inconsistent with all those features. It doesn't match Apple's focus or their goal.