Logic would dictate that the sole device with the problem would be the problem. However networking operates outside the boundaries of logic. Power cycling the router should also fix it since it seems to need to renew the IP the ATV is using. I would try power cycling the router AND modem just to see the duration before it happens again.
Also when its happening log into the google app and look at data usage on your network. Make sure the AppleTV or something isn't saturating bandwidth. 100 mbps isn't difficult this day in age, Apple, Sony, Microsoft, etc can do that. Google gives you the ability to see that in the app though.
You are correct, network works in funny ways.
There is no saturation in the bandwidth, my internet is 600mbps, and I'm not even using internet to stream these. As I mentioned I have these files saved on my network hard drive (WD MyCloud), and stream them via the wi-fi, this doesn't consume (or wouldn't be affected by) some other app/device using the bandwidth, regardless the usage is negligible, I checked.
It really came down to what you said, restarting the router prevented the issue from occurring, and it is pretty stable now, so I guess just cycle the router once in a while to keep things fresh. I didn't even do a power cycle, just a reboot via the google wi-fi app.
I hope this helps others with a similar issue.