Which monitor models? Hooked up which way? I'm at 6x (technically 7x, but one was long ago so I'm not 100% about that one) I've tested where, with HDR on, every other non-HDR program looks like below, right side. White in non-HDR program with HDR off is ~375 nits, white in non-HDR programs with HDR on is ~175 nits with HDR, making everything appear gray and washed-out whenever HDR is activated:
So I had more 'ammo' with the senior tech's at Apple I collated the similar posts on discussions.apple.com and the "me toos" listed there... and there's 2000+ people with this same (or very similar) issue... Check out the crazy level of my testing (complete with professional coloriometer measuring nits and color values) in my linked posts above if you would like – it's definitely a real thing.
...Actually getting a replacement 2023 Mac mini tomorrow (other issues), so I'll be sure to test that one as well + getting a new 75" TV delivered on Saturday so that'll be one more screen I can test (if 7x wasn't enough lol).