After the iPhone 4, I tried out the Galaxy Note 1. I had no issue with size and portability. I had issue with its specs not matching its needs, but that's another story.
With the Note and any large phone, I find the number 1 concern is how long your calls are. 5 minute calls, great, 10+ minute calls, your hand will start to hurt from gripping it. When I take a long call, it's usually at home where I have headphones or other device to talk through. Plus, with any smartphone, I find using them for long calls a bit gross due to the glass touching a face for so long.
One last concern is privacy, a 5.5 inch screen, and now viewable from wider angles, the whole world will see what you are doing. With sites like Gizmodo, Kotaku, you never know what might be the next thing you scroll to or find posted in comments.
Despite that, the 6+ is what I'm currently considering, coming from a 5s. I hope it's a nice alternative for a having a smaller tablet like device at home and on the go.
I'm a bit skeptical of it's battery life considering the larger display and higher resolution games have to run at.