I get the same amount of usage as you when I am actively using the AW a lot, but it matches more to what I know how I used it compared to what you are saying. If you have the watch on your arm and don't use it other than to look at the time and to see alerts, you shouldn't have 4 hours of usage. Something is wrong... Battery draining faster than you expect and then you see 3-4 hours of usage, something is going on in the background. I wouldn't keep re-pairing anymore since the same thing happens. I can't even blame a bad battery because of the usage being reported.
I finished the day with over 60% of battery after putting it on at night at 100% and then a workout in the morning. That's if I don't charge while showering. I would just double check background refresh off on the phone and watch.
Hoping you figure it out. That much usage when you say you are not using and repairing makes me blame communication between the watch and phone... app, bluetooth problem or something.