There are plenty of ways to be active where phone size isn't really an issue. Runners tend to wear less clothing and smaller clothing, and they don't carry much with them. The Apple Watch with LTE capability is popular among runners because it allows them to leave their iPhone behind. That said I've seen a lot of runners sporting belts that carry water bottles and have a place for things like a smartphone as well. I'm sure there are also arm bands that can securely hold an XS Max.
I am quite active, but it's mostly bicycling, hiking, and walking. Carrying a smartphone for those activities, even a large one, isn't an issue. Cycling jerseys even have back pockets large enough to hold the iPhone XS Max, but some cyclists mount their smartphone the handlebars and use it as a cyclometer display. For me it depends on the bike. When I'm on my road bike or mountain bike I usually carry it in my jersey pocket. When I'm on my touring/commuter bicycle I have many options: Jacket pocket, handlebar bag, panniers...
Don't get me wrong: I absolutely think Apple should update the iPhone SE and continue to offer a smaller iPhone option than the XS. The XS is the largest iPhone I could see myself ever wanting to carry, and I have also had many of the models going back to the 3G. Yes the 3G/4/5 were more pocketable, but I still didn't like to carry them in my pants pocket. Pocketability depends on the pockets and the person, but the last truly pocketable phone I owned (in my opinion) was a Motorola Razr. I do sometimes carry my iPhone X in my pants pocket, but only for short periods of time. Most of the time I have a shoulder bag or a jacket I can put it in.