Are you trying to do this overnight while the watch is on it's charger? That's the best time for this to happen as it is done over Bluetooth and takes a very long time.
I'm having this same issue - not sure what to do. I am afraid whatever process that spinning logo represents could be draining battery life of the phone. I've tried turning things off and back on, but that has not helped.