I had the 6 and sold it when the SE came out. I liked the 6 at first due to the screen. Then after a while I noticed how awkward it was to live my life with it in my pocket due to the reasons I gave above. When I got my SE the moment I walked out of the shop with it I remembered how much I loved it. I never loved my 6. It had zero apple magic.
I also bought my SE sim free.
I have a 6 right now & I completely agree with you. Never managed to love it. Maybe I'm getting old and losing the whole "love for my phone" mentality. But the main annoyance for me was to not be able to prop the phone up without any support, which was possible with the 5/5S/SE design. I hardly use any of the really cool camera features, like time lapse, long exposure shots, that I used to use with those phone.