If appealing to popularity doesnt prove jack crap, why should Apple cater to the need of a minority group of users? Why should they labour themselves manufacturing something that will not be as popular as their traditional product? You need to come up with a better persuasion.
First of all, it depends on why the iPhone 5 sold more than the Note 2. Was it because of the screen size, or the strength of the brand name?
Secondly, it's all about sales and profits. If Apple sells 15 million 4" iPhone 5s, does that mean they should ignore the potential 5 million extra they could pick up with a larger screen size? Sure, it's not as large as their main demographic, but there's still profits to be made there. Hell, they'd likely sell more large screen iPhones then they would retina Macbook Pros. And it wouldn't them nearly as much in R&D, so there'd be a better return on investments.
In the end, which would be preferable to you? Grabbing 15 million people, and letting the competition have the extra 5 million because they're "niche", or having 20 million people pay you for two products?