I had been running dev preview 4 with WatchOS 3 for a while. The only thing that didn’t work was the Music face complication. It wouldn’t show what was currently playing as it did in iOS 10, as Apple has switched to the “Now Playing” complication.
I’ve upgraded my watch recently to WatchOS 4 beta 4 (I think it is, yeah?) and everything seems fairly smooth. The one gripe I have about the beta is the watch face sometimes takes a second to turn on when raising your wrist. Other than that it’s smooth sailing.
All of that being said, you should be fine staying on WatchOS 3 should you continue to do so.