I did and have no regrets. Screen size, speed and improved internals has been worth it, almost for those items alone. Like you, notifications are a primary use but I also use it for fitness and sleep tracking. I have the aluminum GPS only models and they fit my needs perfectly. The re-designed case and haptic feedback from the crown are great. My sport bands with Series 2 are on the next to last hole. With Series 4, I can get the same fit on the 3rd hole from the end, so wearing it in the next to last hole is much more comfortable as a result. I also like that the case height has been slimmed down.
Health wise, it will get ECG when "later this year" as Apple has said currently has fall detection. Heart rate alerts can be set through Watch OS 5, so you don't need Series 4 for that. You also get the ability to read an email much more legibly rather than at times getting a link.
My Series 2 has been wonderful and it's perfectly serviceable.Watch OS4 seemed a little bit clunky by comparison to OS 3 and had a more noticeable impact on speed. OS 5 seems to have been an improvement, though there are some apps it seems to slow it down. Workout and Timer are two that I know of.
Is there anything about your current watch you feel isn't meeting your needs? Only you have the answer to is it really worth it.
I posted these photos in another thread recently but this might be helpful to you so you can get an idea of the difference in screen and case size.