Another vote for this being a Big Sur issue, or it also being a Big Sur issue. I have a 2019 Mac Pro with Magic Keyboard, Magic Trackpad 2, and Magic Mouse 2. The keyboard / trackpad / mouse were all pretty solid under Catalina (not perfect, but pretty solid). Under Big Sur I get intermittent pauses, lasting seconds, where no input device works. If I was typing the keystrokes will be queued and all blurt out after a pause. The trackpad is nearly unusable. Nothing has changed but the upgrade from Catalina to Big Sur.
I reconnected the keyboard and trackpad via USB to Lightning. They work fine over USB.