Ok - take a deep breath and lets stop throwing random things at the wall so to speak and see what sticks.
First of all - download the latest version of Handbrake from here:
https://build.handbrake.fr/view/Nightlies/
Obviously make sure to download which version you need - 32 bit, 64 bit, etc. Make sure to have the appropriate version of VLC installed as well. Google it if you don't already have it.
From there - make sure to update presets by click Preset - update presets or something along those lines.
Now - you can either try a DVD or one of your iso's - i suggest doing a dvd first but that's just me. I'm sure you have at least 1 laying around.
From there make sure you select the AppleTV preset but instead of doing the whole movie - you should see a chapter selection near the top that says 1 through however many chapters you have. I suggest doing something like 1 through 1 or 2 though 2, something that gives you a small enough chapter but represents the actual movie you want to encode.
DON'T TOUCH ANYTHING ELSE!!!!! Can't stress this enough - just use the default preset.
Now go ahead and press start and you should have a much smaller file but generated much faster. Now load this file up on the AppleTV and see how it looks.
If you are still having problems - one suggestions is maybe delete everything handbrake related and re-install.
I have ripped literally 2400+ TV episodes and 100's of movies and Handbrake has been fantastic.
If you are still having problems - detail what you are trying to encode - movie, pal/ntsc, homemade movie, documentary, etc. Maybe take it to the handbrake forum and post a log.