Actually, that makes sense to me too. If I was in charge, I'd roll out all 3 sizes to take a bite out of all 3 segments: 4" and below, 4" to about 5" and 5" to about 6". 4" has been very profitable and seems favorable to continue to be profitable.
Until the rumors piled up about the 4.7" iPhone, this crowd backed Apple's stance that all sizes above 4" were too big per "one-handed use" and so on. Words like "abomination" and similar were slung. Now we're in the "maybe" period where rumors have piled up and there's some pretty tangible evidence that Apple is going to go bigger than 4". So the crowd has shifted itself toward the likely 4.7" while "forgetting" about "abomination", "stupid" and "99.9% of the market don't want phablets".
So, if Apple believed what it said as recently as with the launch of the 5s, it should indeed roll out a 4" iPhone 6 too. Just as there are Apple laptops with various sizes of screens to appeal to various tastes, there should be iPhones at various screen sizes to appeal to each group.
Do I think that's going to happen? No. I think Apple can see the revenue math says that the <= 4" or bust crowd can be wooed up to 4.7". They may think the >=5+ inch crowd can be wooed down to 4.7" too. But if I was Apple, I'd be rolling out at least both 4.7" and 5.5" to bite into Android smart phone's most tangible hardware differentiator
and I'd roll out a 4" too.
But I'm not Apple, so I expect 4.7" and hope for 5.5"
with nearly zero expectations of the formerly "perfect" screen size of 4". Personally, I'm the opposite of the example you offered (I'm tall with big hands), thus in the 5.5" or bust camp.