Do you always sleep alone?
Sleeping with watch on your wrist is going to end up scratching your loved ones' faces.
Also fitness trackers do that, not all smart watches.
I would guess that most normal people don't wear their watch to sleep, smart or not.
My Garmin Fenix 3 is on my left wrist, my partner's Vivofit is on her right wrist. So far we've managed without any accusations of marital abuse. It is quite possible that because most of my friends are sport minded my view is skewed by experience, but yes most of my friends wear their relevant smart watches / fitness trackers to bed. Except those of course who own an Apple watch.