The only time when I take off my Apple watch off my wrist is if I feel a bit of discomfort on my wrist (generally I leave it off for 30 minutes).
Occasionally this happens to me as well - kinda sudden and I just have to take my watch off. Not often but it happens.
That's due to the wrist swelling, which is normal. Rather than take it off, you can simply loosen it to the next hole, or if you have the link bracelet, add another link to the little finger side and get used to wearing it that way (it'd waver between being a bit loose and fitting just right depending on the time of the day).