There must be something terribly wrong with me. I really dig size of the iPhone 5 and I'd prefer not having a larger phone. 4.3" might work out OK, although I'm not entirely sure on that one. Anything larger and I'd probably have to look elsewhere for my next phone, though.
Anyway, from a hardware development perspective, I think it's a tough sell. One of the factors that is routinely omitted in discussions on this topic is that having a different-sized phone very likely means having to develop and manufacture a new display. With notebook computers (or desktop monitors), this isn't the case. The vast majority of them use existing display models in mass manufacture, and the ones that don't typically just use minor revisions of the main units.
Larger iPhones won't have it so lucky - Apple isn't going to be using existing panels. There are several reasons for this, but chiefly it will be due to Apple's specific resolution requirements, performance requirements (power, gamut, brightness, viewing angles), physical dimensions, and integration of whatever touch solution they ultimately favor. In short, increasing the size of the iPhone yet again means tooling up another custom panel at Apple-sized production volumes. That is a very expensive proposition for both Apple and its suppliers. Having different sizes is arguably even worse. Either suppliers will have to have parallel infrastructure for two panels developed for a single buyer (exceptionally rare in the industry), or Apple will have to go to different suppliers for each. Option 1 is a tough sell for suppliers as it levies a lot of risk on them, Option 2 is tough for Apple given past difficulties with meeting demand.
I think one of the traps we often fall into is that we start with a personal preference and desire a world that meets that preference. We then marshal evidence in favor of arguing for that future, and in the process our impression can shift from a simple ought to inevitability after wading knee-deep through confirmatory evidence. However, for the most part, my impression is that Apple still appears to primarily sell to consumers for whom Android and Windows Phone are not strong competitors (or perhaps not competitors at all). This may be due to things like vendor lock-in (via the app store), size preference (some people actually prefer the smaller form factor), brand loyalty, influence by salespeople, or something about the Apple ethos (e.g. "ease of use"). As a result, while there appears to be demand for larger phones, it's unclear whether meeting that demand will translate into counterfactually greater sales that would adequately compensate for the costs of such a strategy. Thus, overall, Apple has a strong disincentive from a development standpoint to retool for yet another display size (or especially multiple sizes), and the incentive to do so is at least ambiguous.