I'm basing my thoughts on Apple not growing from 3.5 because Steve said Apple put alot of thought into it and research. Obviously he is no longer with us, so that may change.
I have friends that have snapped their Android phones in their pockets when they sat down...or end up carrying it around in their hand alot.
But I also have larger friends with bigger pockets...or women with purses whereas a larger phone isn't that much of a bother.
Steve and crew made those decisions as long as 5 years ago. Things change. The market changes. Customer needs and expectations change.
I have friends that have broken screen after screen on the iPhone from dropping them, sitting on them, slipping out of their hands, cases, etc.
Accidents happen no matter what size device is being used.
Like I said - I think Apple stands more to lose in the long term if they "never" change their screen size.