Ah I'm part of this crowd, love my iPhone SE
I used to be a huge fan of big phones too, I was on the Galaxy Tab 7 (7" phone), than the ASUS PadFone (11" Laptop/Tablet Phone), than the Xperia Z Ultra (6.3" phone), than iPhone 6+... and finally went back to a super small phone (iPhone SE).
The main reason I went to smaller phones was simply because when I'm home, I'm on my iPad/MacBookMacMini, when I'm at work I'm on a MacBook... so it didn't make sense to carry a bulky phone around.
Although I wish the Apple Watch had a SIM so I didn't need to carry my phone around, than I finally have nothing in my pockets (my money is on Apple Pay, my phone is my watch).
/end of my long boring story