I was always against the larger sizes of phones when I had mi iPhone 5, but then it got stolen and I managed to get a Galaxy S3 to replace it, I initially was leery about the extra size of it. However, since then I've really grown to love the phone, as I don't really do calls on it that much.
However, since I don't have a purse nor really baggy/cargo pants (I don't wear skinny jeans either, I wear stuff that looks good - and that's not anything really baggy and sloppy), I couldn't get over the size of the 6 Plus as I had worries about how it'd fit in my pocket - particularly when sitting down.
Truthfully I'd love a size in between what Apple offers - something such as a 5.1 or 5.2" screen with smaller top and bottom bezels.