Is it tough? I do the same. But my Apple Watch Series 6 already charges from 10% to 100% in what feels like about 1 hour. I haven't really measured it, but it feels really fast.
Apple made a point about you only needing to charge it for 8 minutes to get 8 hours of sleep tracking. That is nice I suppose if I end up going to bed and my watch is already dead or close to being dead. But it's only 33% fast, that would normally mean you could achieve the same with a Series 6 by charging about 14 minutes. I have to admit, 8 minutes versus 14 minutes doesn't really make a difference to me in any meaningful way.
I always use my watch for sleep tracking. I have a MagSafe Duo charger at my desk and drop my watch on the charger during my workday. If the charging takes me 45 minutes or 1h and 15 minutes doesn't make much of a difference to me.
But each to their own I guess.