If Apple sold a 4.5" phone, they'd likely still sell huge numbers (maybe more). Everyone here underestimates just how much people buy for the apple on the back.
I agree that there's some truth in this argument, but the size still matters. The important question is, how much does each of it matter? If Apple suddenly stopped making 4" handstes and switched to 4.5", most certainly many people will keep buying it, but some might be put off with the larger size and start looking for smaller handsets.