For the AppleTv to work correctly it expects the first track to be AAC and the second track to be AC3. You can just have the AAC track, but you cannot just have the AC3 track.
iTunes *will* take it if it just has the AC3 track but no apple iOS devices or the AppleTV will play it.
Thanks, that is what I thought. It would have sucked for me to do all of this searching if I was wrong
For the future I will use the Apple TV preset in Handbrake but now I need to find a way to add that AAC in to my existing videos without having to re-encode the video.