See, that's the thing for me. I have different devices for different needs. I have an iPhone, an iPad and a MacBook Pro. Yes, my every day device is my iPhone, my casual media device is my iPad and my workhorse is my Mac. I bring with me the device that will best suit my needs at the time. Famous quote (and one of Apple's/Jony Ive's recent ones as well) "Don't try to do everything because you wont be truly great at anything.". Personally, I I'm happy with each of my devices and the niche they fill.
Well you initially mentioned your iPad to negate the need for a larger screen.
What my point was, was that when you leave the house, the biggest screen you have, is the screen you have with you.
So in this case the 5.5 would be a more vibrant experience than the 4 and the iPad is probably at home unless you're doing something special with it.
So therefore it would still be useful to have the 5.5 screen over a 4/4.7 even if you own the iPad.