I think if anything, people that went from a 6 to a "SE" feels a little silly for paying for a new device when the 5S has been around for years.
Thats not fair; not everyone owned an iPhone 5 or 5s who is purchasing an SE from a 6 (not everyone experienced that "older" design)
The SE is an upgrade in pretty much every respect over the 6, other than the slightly better screen tech the 6 is using or if you have a major barometer fetish.
I mean, it is more powerful, its faster, has a better camera, has 2x the memory (and faster ram too), its lighter, battery lasts longer, has an always on co-processor, among many other features all the while having a lower MSRP
I sold my 6 for 300$ more than what I ended up paying for my SE, both 64gb models.
I don't feel silly for getting a better phone and making 300$ doing it