Having gone from the 4 size to 5 to the 6 now, yeah, I'm now in the 4 camp. I miss the small form factor and portability. I'm not complaining as I preordered a 6 at launch. I hope they offer a smaller size version in the future. I use my phone as a phone moreso than any other task.
Many of us do not use it as a phone. I talk about 40 mins/month on my phone, mostly to my wife and to my parents for a few mins on the weekends. The rest of the time it is internet, email, text, apps, etc...
for me, a larger screen was a must. I was tired of reading the internet, emails, etc... on a small screen. I had to keep switching to my rMini which then a lot of times I brought with me. Since getting the 6+, my rMini stays at home and is only really used now for video content or iPad specific apps.