I don't regret getting the SE for the screen size. I don't regret getting the SE for the selfie camera. I don't regret getting the SE for its lack of 3D Touch.
But what I regret getting it for is the poor build quality. The vibration is weaker than in my old iPhone 5 to the point, where I don't feel it in my pocket anymore. There is also a loud and very annoying rattling noise when it vibrates in horizontal position. Last but not least the Home Button feels broken and makes a very loud plastic noise when releasing it.
I really don't want to be too nitpicky about this, but it just feels wrong and doesn't do my overall impression of Apple products any justice. Of all the Apple devices I owned, this feels like the cheapest by a mile.
I will probably contact support and try to get an exchange SE, because I really love this phone apart from the issues above. It is blazingly fast, its battery lasts forever, it takes great pictures and it is the perfect size for me. I really hope I just got a bad SE.
If anyone is wondering how it turned out: I went to the Apple Store, showed them the vibration rattle and got a new SE right away. No rattling issues during vibration with that one. The home button feels the same though, it just seems to be the way it is. All the other SEs on display had the same strange home button. Well, I'll just live with it, I read somewhere that 5S users had the same problem and that it got better over time!
With my new SE I now have no regrets anymore and I'm going to keep it until the next big iPhone (considering the rumors probably the 8).