Not a matter of should or shouldn't, but I always turn off transparency (under Accessibility) because I don't like the patchy discoloration it sometimes causes in window headings. Unfortunately, doing that on the new M1 machines makes it impossible to hide the notch.
I turn off Tap to click because I get too many accidental clicks otherwise. Also turn off Look up & data detectors, Secondary click, and Force Click and haptic feedback because I sometimes get mistaken clicks and odd results with those too. So all of those.
Turn off True Tone, which always looks untrue to me.
Turn off auto-brightness because it doesn't adjust the way I prefer. Same with the keyboard backlight (which I keep on at the lowest level all the time unless I'm watching a movie).
Not things I turn off, but:
I change Show scroll bars to always, even though it doesn't look as good. Saves a step. Also select Jump to the spot that's clicked, great for long pages.
Select Automatically hide and show the Dock.
Put Keyboard Brightness out on the top menu bar, since it's no longer controllable on the keyboard.
Show Percentage for battery and Sound on the menu bar so I can see at a glance how they are.
In Safari I switch to Compact tabs, because I hate it when tabs disappear.