I agree, the biggest change from your 4 would be the always-on display that came from Series 5. If that's something you're interested in (to me, it's been fantastic, as I upgraded from a Series 2 > Series 5), you could look into secondary market for Series 5's now that the 6 is out.
Apple Watches tend to lose resale value really quickly, especially once new models come out. Lot of brand-new-in-box Series 5's you could probably get at a pretty healthy discount to the Series 6 prices, if you're comfortable with buying from not Apple.