4" is the perfect size for ME. I had the original iPhone, the iPhone 4, then a Samsung Galaxy S2 Skyrocket, then a Huawei Fusion 2, and now a Nokia Lumia 520.
3.5" is too small these days. This became apparent when I started buying Humble Bundles for my Huawei. Many of the more elaborate games that are out now benefit from the half extra inch, especially when it comes to text.
But the 4.5" screen on the Skyrocket was huge. Barely fit in my pocket, and was slippery. I'm a one handed typer, and it'd start slipping out of my hand after I started to type. And you can barely swipe down the notifications on it with one thumb when you're just using one hand.
I'm happy with the 4" screen on the Lumia. It's enough space to play games on without it being too awkward to hold, it's enough space to do small sketches with my Jot Pro, and it makes a great reader.
Unless it would run iPad apps, I wouldn't go for the 6" iPhone. That, and it would have to have 128GB storage if I'm to run both iPhone and iPad apps. 🙄