Well, I have a little more info.
In iPhoto/Aperture library there is a "Apple TV Photo Cache" folder. Within that is the "Apple TV Photo Database". After selecting my 3000+ image album to share with the ATV, the database is created, but it's only 149.9mb. No matter how compressed the images are, there's no way to store very many in only 150mb.
Seems like a necessary limit for the ATV2/3, as they had only RAM, no onboard storage, but with the ATV4, there's no reason to set it to only use 150mb of photos for the screen saver.
I'm going to see if there's a hidden setting somewhere.
There's definitely some sort of archaic limit in there for the screensaver. I've just sat and watched through one, it got to just over 300 (ish, keeping count wasn't so easy) unique photos before it started to recycle them again.
Looking at the Photo Cache folder on my Mac it's total size is 4.8MB so opening them up and investigating, they are just database files storing the locations of the images. There's nothing but links to the image locations in my files.
Now, as I type this, the screensaver has started up again and it's using a completely different set of images to the last run through.
So my suspicion is that it does have access to the entire library (which is what I told it to set as the screensaver source) but limits each run-through of the screensaver to a batch of about 300 from the complete collection.