I think I found the issue.. I have about 500-600 photos in Photostream. I've had all these photos (backed up of course x2) of my daughter since she was born (she's 6 months now). I tried to keep photostream clean with only pics of my daughter, so when my family is over I can have the screensaver with pics of her on my Apple TV.
I think moving to my iPhone 5 on Saturday, it loaded photos only from 30 days ago because that's the way it's supposed to work. I never erased or messed with it on my 4S. I still have my ALL my photos through Photostream on my Mac and my Apple TV.
I think I may just erase it and upload my album of my daughter to Photostream and start over...