It takes mine about two hours to charge from anywhere close to 0% all the way to 100%. This is the case both with the Sport-specific plastic charger and the stainless steel charger.
You just have to resist the temptation to use the Watch while it's chargingI've found that it simply stops charging entirely whenever the display is on.