I found that the size down (4) that I first ordered did stretch a good bit. I ended up going with the size Apple recommended (5)instead because I was nervous that the size down would be too tight in hot months.
The 5 has not stretched nearly as much. I think it is partially because when you size down, you’re having to stretch the band a good bit more to put it on and take it off. Additionally, sizing down means it’ll have to stretch more to accommodate your wrist.
As the weather is warming up, I’m very glad I didn’t keep my size down braided solo loop. The 5 fits perfectly.