I can confirm that I was having the same issue on my 2020 MacBook Pro 16". I have 400Mbps service and it would drop to 120-200Mbps as the day progressed unless I performed a restart. I caught a thread where someone mentioned disabling Unlock with Apple Watch and I have had any issue since. It's been over 3 weeks and when I had the Unlock with Apple Watch enabled, it happened every day of use.