That's just it. It doesn't have to be an either/or situation. When it comes to screen sizes, it's all subjective opinion, and no one's wrong. If you like smaller devices, then...well...you like smaller devices. You're not wrong. Neither am I for wanting a larger device.
That's not fragmentation. A larger iPhone would only be a concern to developers, and wouldn't prevent end users from getting the same updates, or having trouble running the same apps as anyone else with other sized iDevices.
See this is your problem right here, you don't see a problem with a larger screens being added to the bunch just because you want a bigger screen.
While this does in fact cause fragmentation, because fragmentation also happens to cover apps that do not work properly or at all across all devices.
Which is going to happen, I can 100% guarantee you that.
People who bought the 4S back when it was released crawled their own eyes out or instantly had to upgrade, when the 5 came out.
Why? because developers who are alone like me JUMP the boat. I can't be bothered to keep developing my app at a lower resolution, while everyone is slowly upgrading. If you think this is all a simple task in iOS please try for yourself. The point that you so fondly neglect, is that what effects the developer also effects the user in that they might lose support for certain apps. Just like the iPhone 5 got exclusive apps, and iPad is getting exclusive apps.