I don't think it will play them. My ATV will only play up to 720p content via syncing or streaming. I don't even think the file will show on your ATV when your iTunes library is open on your computer.
Bingo! It'll be in your iTunes library, and play fine on the Mac, but it will not appear in the menus on the ATV.
Recently, I ripped a Blu Ray and encoded it to 720p for my ATV. But it wouldn't sync. I tried futzing with the bit rate and re-encoding it, thinking that it got a little too heavy for the ATV, but still no joy. Eventually I checked the media detail, and found that it was 29.xxx fps. Had to re-encode at 24 fps AND lower the bit rate to get it onto my ATV. It looks like crap.
Bottom line: the ATV is very picky and will simply refuse to list unplayable media in its menus. It also won't tell you why it's refusing the file, even though it knows...