Yeah, you hit the nail on the head. And everything about Big Sur is just slightly slower - whether it be your eyes quickly skimming for an app icon, the fact that lesser information density = more scrolling and/or eye movement, the way that the smaller global text size, especially of drop down menus, makes it harder to discern clickable elements quickly, etc.
And it's especially annoying, since all these changes are actually rather subtle, and not quite severe anough to cause enough commotion to incentivise them to do anything about it. In an effort to make everything pretty, they've purposely treaded a fine line between a design that most people will tolerate in terms of functionality (especially 'slower' users), and what is clearly aesthetics prioritised over functionality. A practice I absolutely detest.
(Also, their 'space everything out approach for future touch devices' logic is dumb. Now we have an OS that is 'Jack of all inputs, master of none').