I've wondered about wearing it for heavy labor like that. Fortunately that isn't usually a problem for me
I do bike to work though, and I often go for longer bike rides before work or on my lunch break. Fortunately where I live it rarely gets above 75 degrees in the Summer, but it can get up into the 90's and it is extremely hilly. On milder days I can manage to climb the hills without getting too sweaty, but other days I will be dripping with sweat. I have two third party leather bands: A brown leather band from Clockwork Synergy, and a black leather(ish?) loop knockoff. I switch to the sport band on hot days when I know I will be sweating a lot, but some days I forget and just wear one of the leather ones. I'm not worried about the Clockwork Synergy strap, but I don't know if I want to test my fake leather loop. I'm more likely to take the watch off for getting dirty with that one on. Other times I keep the watch on when I'm biking, doing dishes, and for light yard work. If I need to do some heavy yard work getting into the dirt and dust, I will take it off so I don't have to worry about particles getting stuck in crevices.
I have knocked my silver aluminum Sport against walls and desks on accident and it is no worse for the wear (knock-on-wood). I don't feel like I need to baby the watch, but I certainly don't need to wear it every waking minute either.
Of course the OP is talking about taking it off for a different reason: To disconnect or tune out from the hectic technology-driven work/lifestyle that our gadgets help us maintain. In the case of the Apple Watch, I feel like it is the accessory. The iPhone is what keeps you connected to your communications apps, your task and project lists, your email, reminders, etc. I get the desire to remove the tech from the body though. That will make a difference for some. Personally I feel like it takes more than removing an Apple Watch or enabling DND on my iPhone to leave all the stuff it helps me do and track behind. Many of us like these devices because they help us live and work within a technology driven environment with more convenience and less stress and anxiety. Disconnecting your devices will not necessarily disconnect your brain from what you need to do later. If I'm enjoying my weekend and I suddenly remember something I have to do at work, being able to say "Hey Siri: Remind me to submit my TPS report on Monday at 9:00 am" relaxes me because I no longer have to remember to do that. If I want to totally disconnect I need to postpone all my tasks and reminders for a period of time, which helps me feel like I can afford to go on DND and not worry about it. If I just hit DND without first doing a quick review (and I'm talking seconds to just make sure I don't miss something I'm supposed to do that evening), I get anxious that I'm going to miss an important reminder. Sad, but that's modern life. Too many things to remember to do at specific times or places.
Sean