I've worn my Series 0 44mm SS from launch doing yard work. I just wore a little clear bumper on it, and later on I bought an arm band. I've worn it installing a rock landscaping retaining wall, putting in new bushes, mowing, trimming, seeding, spraying, tree trimming, and while clearing brush and using my chainsaw. I also wore it while finishing my studio space in my basement, which involved a lot of sawing, nailing, screwing, insulation cutting, etc. It was always fine. The only time I ever damaged it was when it didn't have the bumper on and I accidentally banged it against the stainless steel handle on my refridgerator and put a hairline crack down the right side. So it's very possible to damage you just have to be careful. The gloves that I wear have a bigger cuff on them that party covers the watch and I think that might've helped. The full coverage bumper case is probably a safer bet, especially if you're kinda clumsy.
Now that I have a Series 4 44mm SS I haven't decided what I want to do yet. I did yard work right before my Watch arrived, so I haven't had to do it yet. The problem is I don't have a case yet. I'd like to get the kind of case for doing manual labor that has the bumper with a top part that wraps over the whole thing, but the few I've seen on Amazon don't have reviews yet. One that has some reviews and seems like it's from a reputable maker is out of stock and had some mixed reviews like they didn't get the fit quite right so I bet they're fixing it now and will be selling the updated version soon. So I'm waiting for that one to come back in stock. But I'm also thinking about wearing my Series 0 for yard work. But I'd like to retire that to a display case with my original iPhone and iPad. Also, I didn't end up being much of a fan of the arm band and it barely fit and would slip down on my bigger arms. It was also more of a pain to use and I'm not sure if the exercise data or heart rate was as accurate when placed that high up.