I'm in the same boat, 4.5 max. Although the Moto X feels pretty damn good in the hand at 4.7".
I think the screen size will end up being whatever will allow the OS to scale accordingly.
The MotoX is perfect. That is the best feeling phone and best screen size I've ever seen on a smartphone. I wouldn't buy it because I hate Android. But they've totally nailed the screen size and form factor. You'll notice on the MotoX that they've effectively eliminated the black bezel around the screen. This subtle shaving off of this clunky bezel is what makes it still small but with a larger screen.
Apple seems to be in love with ugly 90s bezels. The iPhone has just too small of a screen. I can't deal with it anymore. Everything is crammed. It's not wide enough. And those ugly top and bottom black bezels...
My prediction is after the first few months here, the sales of the iPhone 5S/5C are going to taper off.
The Galaxy Note 3 and more innovative Android phones are coming. Apple could have made the screen bigger here and they made a mistake. Now all these people rushing to buy these new phones are going to be on contract. They had a chance to do a bigger screen and start from the beginning again... particularly since they released a new iPhone model. Now all these iPhone 5C buyers are going to be stuck on 2 year contracts.
So next year? Well, go ahead Apple, but you're going to fragment yourself.