I had the same doubts as you, for some years.
last year I almost bought the series 3, but at the last second decided that I don’t need it.
Then I started going to the gym, and took my phone for music, and noticed I was distracted by my phone too much.
Few months later Apple announced the SE, and the rest is history.
I live on a budget for 1,5 years now, (buying a home in the Netherlands is HARD) so I can not spend money like I could before on gadgets and cool things.
I use the watch for gym, music in combination with Apple Music and podcasts. Apple Pay is wonderful, and read notifications quick and reply is cool and useful.
I use the health app, and I absolutely love the alarm when you sleep with the watch.
after my Phillips wake up light needed to go, after complaints from my girlfriend, I woke up every morning with my phone alarm. It works, but for me it’s hell to wake up.
The watch taps my arm gently and softly plays an alarm sound, and I absolutely wake up a lot calmer with this method.
the SE is a great deal! But personally I regret not buying a series 6 🙄.