I'm not aware that you can change that behavior.
There is a setting to enable swipe for changing watch faces but I doubt it would disable the default behavior that you describe above.
It happens to me occasionally, bit not frequent enough to be bothersome. My U2 might behave better than the regular AW due to its flat screen, but not sure
One annoying workaround is to use Water Lock, which locks out all the controls until you specifically disable it again. I do this when I go out in winter so there's not random bull**** on my screen because I put my hand in my coat pocket and pressed the crown against the fabric. (The Apple Watch should have the option to ignore ALL input when it's not face-up, because 99.9% of the time it's accidental.)