This happened to me twice... phantom toggling

After I turned things off the second time, the problem seems to have corrected itself.
I've currently only have background app refresh active for weather, messages, activity, and workout.
I had the settings like this for several days and the battery life on my watch was still terrible.
So in addition to limited background app refresh, I now only have the workout app in my dock (got rid of weather, activity, messages yesterday). I have the feeling my dock was what draining the battery. Time will tell.