Based on the models it is limited to I would think it is based on RAM.
It seems the newest iPod Touch isn't able to run it either. iPad 2 and iPhone 4 both have 512MB vs 256MB for iPad 1 and iPod Touch 4g.
I imagine it tries to cache as many photos as it can so it can smoothly operate as you use it.