Have they? The Galaxy S4's sell in rather spectacular amounts, and even the relatively giant ass Note 2's and 3's have a healthy market. That's hardly most of the world rejecting something.
See, correlation doesn't equal causation. That's basic statistics. You need to ask yourself this: do people buy iPhones in drove because of the smaller screens, or because of the strength of the brand? Does each sell of an iPhone SPECIFICALLY point towards wanting smaller screens, or that they like iPhones in general? You're assuming X because of Y, without having anything to back up your argument.
Likewise, assuming the S4 is selling well as a result of the large screen is equally fallible. I know you are not making this argument, just wanted to make sure the other side of the coin was represented.
They sell because they are arguably the best Android phone on the market - although an argument could be made that screen size is a differentiator among flagship Android handsets. Screen size is much less a differentiator between the iPhone and the various flagship Android phones. The platform is the differentiator. Note phones do have a niche market relative to the S4 and iPhone.
The thing is that iOS being the platform that it is, introducing a new screen size, as troublesome as it may be on the developer side (both internal and 3rd party), would most likely only have a positive effect on sales of iOS devices. People that want to jump (or return, or continue with from a different iDevice) to the Apple ecosystem but liked a bigger screen will now have very little excuse not to do so.
Fans of Android won't move regardless, and the same is true of fans of iOS. The average consumer, though, may not be platform loyal. However, 6 years down this crazy road we've been on, and the average consumer's mobile savvy has grown to the point where a larger and larger section of the market know that because of their software investments (i.e. apps and ecosystems) they are more and more locked in to whatever platform they have been with, and thus more and more platform loyalty is embedding itself in the market. This process will only continue and both platforms will be fighting over fewer and fewer first time adopting consumers, and even fewer flipping consumers.
This actually works out better for Apple, because iOS is Apple, whilst the various Android OEM's will always fight among themselves for Android's share of the pie. This is why you will continue to see an Apple product near the top of, if not at the top of, the best selling lists - because everyone who chooses iOS is by default also choosing iPhone.
Obviously not the case with Android, where the consumer can choose from a multitude of hand sets, even if you limit it to the "flagship" models from the various OEM's. That's why the Galaxy S flagship models have been impressive from a sales perspective, because it must be a pretty damn good phone to have been taking 2nd to the iPhone in sales for the past few years, it having more competition to deal with - other Android phones as well as the iOS juggernaut.
And it has less to do with the screen size than the total package, I would venture to say.