Lots of people want small phones. Do you know why there is so much demand for large phones? Because no one makes a small phone that people want, they are either gimped on specs or are from some no-name manufacturer that no one knows about.
I strongly disagree that they make them low end because of the size. Look how thin they are able to make phones now, the ability to make small phones isn't that difficult. Most of what takes up large phones is simply the display and a bigger battery. The actual circuitry of a phone is very small and could easily fit in a 4" body. Also, seeing as how Sony has a 4K display in a 5.x" phone proves that you can get much higher PPI than what Apple currently offers. Besides, the display is the biggest battery culprit. Smaller display = ability to use a smaller battery. Besides, Android phones have twice the capacity of Apple phones in roughly the same size, so it's possible to get larger batteries in that form factor.