Nope, you're not the only one, because there are plenty of Apple customers out there that prioritize easy pocketability and one-handed operation. Depending on how Apple makes the phone thinner and reduces the bezels, the rumored size might still be okay. Doesn't matter to me yet because I have a 5s and don't intend to upgrade it until at least when the "6s" cycle hits.
If anything, I'd love to see Apple continue to make a flagship-class smartphone in the 4" size (something that none of the Android OEMs do). We would then have a true apples to apples screen size option -- pick a 4" or 4.7" screen, without compromising the performance or features. Apple already offers a true comparable option with the iPad Air and Mini. But, all indications seem to point at the 4.7" screen size replacing the current 4" form factor.