Even more anecdotal evidence that someone else is prob biting into the bandwidth pie when you are also making high demands of it. When they finish, a bigger slice of the pie becomes available to you again. No reboot necessary because AppleTV is simply accessing whatever bandwidth is available.
That’s still my best guess.
It could also be old router equipment, throttling software, appliance or other stuff interference, etc… or faulty AppleTV. You could completely rule out the AppleTV with the Ethernet connection experiment or taking it to somebody else’s home for a movie marathon, hooking into their wifi and seeing if the dramatic slowdown happens on a completely different system.
Once you can rule out AppleTV, you will know it’s the wifi being shared with you… and then can make decisions about what to do about it: get your own, convince landlord to run an Ethernet cable up to you and/or perhaps update and aging router, find a different place to live with better wifi, etc.