I'm not aware that you can change that behavior.
There is a setting to enable swipe for changing watch faces but I doubt it would disable the default behavior that you describe above.
It happens to me occasionally, bit not frequent enough to be bothersome. My U2 might behave better than the regular AW due to its flat screen, but not sure