Also, try to set your media type to Movie (instead of Home Video) in Subler. There was a discussion here in forum, that someone experienced the video stutter exactly with home videos on aTV.Thanks for the heads up. Will check it out tonight. I have used Handbrake for so many years but never for mp4s until recently.
If your source already is in AVC (H.264/x264) video, you really don't need to re-encode, as orpheus1120 said.
You might still want to do this for very large/high bitrate files, just to ease up the life of your aTV and network.
For example straight blu-ray rips that weigh anywhere from 30-50GB in size (bitrate 30+Mbit/s) will give less problems, if you Handbrake them down to 5-10Mbit/s rate (4-10GB file size).
AFAIK, iTunes movies are still encoded with 5 Mbit/s bitrate.
Actually, Handbrake does amazing job in picture quality vs bitrate. That might explain the time it takes to encode.