It could be a combination of several factors. When I first started using the Sport band about 2 years ago (originally got the Milanese loop, but chose to wear something less formal on a day to day basis) I didn't fasten it as tightly as I do now. I started out using the second notch, now I use the third. However, I don't recall having a choice in the beginning - iirc, the third notch was too tight at the start, second notch was a hair too loose. I'm thinking that it may have stretched about a half-notch. Yeah, maybe my wrist is a bit more lean, or I'm wearing it in a slightly different location. Hard to say.
Another possible factor is stretch/distortion of the notch/hole that you use regularly. Looking at the third notch, it does seem a slight bit out-of-round, but far less than I've seen on many a leather band/leather belt notch.
I think if we were to compare the performance of the fluoroelastomer band it probably stretches permanently less than any other band material, with perhaps the exception of metal. Unlike leather or cotton webbing, it won't stretch due to perspiration/exposure to water, and among synthetics (woven nylon or polypropylene), it retains far more elasticity/rebound after being stretched. Even metal bands may expand a bit due to metal-on-metal wear.