The screen brightness does depend on surrounding light levels. For example, if I use my shirt's cuff to cover most of the screen and leaving just a tight bit of screen visible, and when the screen does turn on after I flick my wrist, the screen will be very dim because it deems the surrounding to be "very dark" (because my cuff almost covered it). After I pull up my cuff to reveal the entire watch screen, the screen brightness will gradually turn up to a much more readable level.
So the screen brightness setting on the watch does not really control absolute brightness, but rather a perceived brightness that automatically adjusts based on the surrounding environment. Therefore I always set it to the middle brightness and even when under mid-day glaring sun, I don't have to manually turn it up because the watch does it for me.
Kind of intelligent if you ask me.