Same experience. Ended up selling the MBP and getting a MBA M2, which is much softer on my eyes.
Can't say why. Could be the PWM or something else about the mini LED backlight, could be the extremely saturated colors or the temporal dithering process, could be the extra contrast, or something else.
I love the fact that I can use the MBA display for many hours without issues, even if I had to trade in some nice-to-haves like pure blacks and 120Hz.
Edit: my primary suspect is the PWM, because when using older "harsh" screens, I was usually able to get some relief by lowering the screen brightness, whereas on the M1p MBP the lower the brightness, the worse my eyes felt (especially in dark rooms, using either light mode or dark mode).