I personally think the iPhone 5 hit my sweet spot. The 16:9 is great, and Typing is amazing on it, while still having more space for apps to put more controls, and wider viewing angles. Its amazing for games to. I used to hate playing FPS's and games like GTA on the 4/4s as my thumb would cover most of the screen since the controls were at the sides. With the iPhone 5 its like having the whole screen on an iphone 4/4s and having the controls off the the sides. Its much more enjoyable.
I've always used my iphone with just my right hand and thumb, and could reach all corners of the phone with just my right thumb. Holding the iphone 5 the same way, I can't even reach the right side of the status bar now without doing some awkward repositioning of the phone in my hand to reach the top, and then again to get back to my normal "typing" position. This is quite uncomfortable. Yes, I know I could simply use both hands, but then again, Apple Went out of their way to BRAG that the IP 5 is the perfect size for one hand use. Remember the IP 5 commercial where they have some guy with HUGE hands demonstrating that your "thumb goes from here to here". (It's funny because if you watch closely, you can tell that the guy is having to overcompensate the stretching of his thumb to create the illusion it can reach the opposite top corner) The commercial actually offended me somewhat. Anyway, to answer the question, for me 3.5 inches was the perfect size in terms of comfort of use, but I do like the bigger screen for games and video.
Back when the iPhone 4 came out everyone else was already going 4"+. I was shocked that Apple kept 3.5".
I compared the iPhone 4 with the HTC 4.3" at the time and MUCH preferred the HTC look, size and feel. I opted to skip the iPhone 4 and just held onto the 3GS (Apple ecosystem and all that). Several years later and I'm still on an iPhone (4S), but would love to see a 4.3-4.5" screen. The 4" 16:9 iPhone 5 screen was largely pointless IMO.
Also, going larger, Apple would have to round out the back of the phone. Going larger and keeping it square would be terrible to hold and pocket and make it even more susceptible to bending.
I'm slowly getting over the fact that we're just expecting way too much out of such small devices. The iPhone is terrible to browse the internet on, terrible to type on and too small for all but basic puzzle games and too small to seriously watch videos. Making the iPhone 5 one icon row taller improves none of these things.
But hey, I can reach the whole screen with my thumb.
My 4S screen size is fine. My wife's 5 screen size is fine, but the phone does look alittle narrow. But, if Apple realized an iPhone the size of the Note 2, I'd get that. Either way, I'll be happy with an iPhone.