I don't know that anyone is really making an argument that screen size is stealing users. I think they are just saying that it would be nice to at least have the option, is all. I do a lot on my phones. I do prefer a screen around 4", which I didn't know I did until I started using one. Do I do less on my iPhone as a result? Not at all, but that doesn't mean a little more real-estate wouldn't be convenient.
The bottom line is that the fact that Apple's current phones sell well is not evidence that a larger-screened iPhone wouldn't sell equally well, or perhaps even better. The question is whether or not Apple views that as a risk worth taking or not. You can bet that screen size is a topic of discussion amongst the engineering and design teams, much like widgets and many other features "first seen in Android" were and continue to be.