Probably not server side. Everything is already loaded. However, when I scroll sometimes it just slows down for a brief instant. A fresh reboot seems to clear it up so it makes me wonder if it has to do with memory.
However video streams are fine. No stutters at all even with 4K Dolby Vision.
AppleTV strikes me as the kind of device that people would keep for multiple years rather than - for example - an iPhone or iPad that also slows down with future iOS updates but is also constrained by battery life.
For this reason I'd say that's why these devices need to be slightly overspecified on release as people will be keeping the streaming box for years. I have a couple of mark 3 AppleTVs with the excellent Apple Remote and skipped the Mark 4 because it didn't support 4k (years before I got a 4k TV). I was agnostic about the Siri Remote.
I also skipped the Mark 5 because to wait for an updated model because I saw the A10X CPU (with 3Gb of RAM as it turns out) as underwhelming. It turns out that it's similarly specced to the iPad Pro 2017 SoC.
So in theory a mark 6 AppleTV with A12X CPU could use the iPad Pro 2018 SoC - with 4Gb RAM - but the same chipset also supports 6Gb of RAM as per the 1Tb 2018 iPad Pro SKUs.
It's already been noted that this AppleTV has been sitting around potentially waiting for release for months - Jon Prosser mentioned it in the Gadget Cast podcast too. Judging by that previous article Apple must have decided to postpone the project to fit the A12X CPU rather than the A12 CPU - I would imagine testing pre-production A12 hardware on 4k screens might have shown up some lag.
If Apple reconfigured their plans around A12/A12X then it would be logical to give the AppleTV more RAM to cope with increasing numbers of consumers switching to 4k TVs. In effect the 2020 SoC shows how Apple have already moved on from 4Gb - leaving the mark 6 AppleTV on 4Gb would be a retrograde step.
I'd like to think that Apple would be forward thinking and do the latter of course as it would make the Mark 6 a capable platform for a good 4 years. They would only need to give it a mid life storage bump to 128/256Gb to keep sales up.