Ok heres my guess. It's a frame rate issue. What type of dvd are you ripping? Movies, tv shows, home video? If they are movies, they run at 24 FPS (23.976 to be exact). If you set the encoder to rip at 30 fps, or some may be defaulted to that setting, you will get choppy video. The encoder has to double every fourth frame. as in 1234456788.... ect to fit 24 frames into 30. When played back, it leaves a slight annoying stutter in the motion.
If your source is video as in home movies and TV shows( not all tv shows), you have interlacing issues. Video in America/north america is 30 FPS with 2 fields in each. Each field contains one whole fame, so every real fame carries 2 interlaced frames. So american television is actually 60 Fields per second, and when played back on TV, 60 FPS is what you get. Well most encoders try to deinterlace fields. So what they do is chop out 1 field out of every frame leaving only half the motion, and back to the choppiness. Mpeg4 and H.264 dont to well with interlaced frames. You can deinterlace to 60FPS, but the methods are not easy to do and im not quite sure if the apple TV will play them. I know the video Ipod can.
3rd reason. is that the dvd may be a film with 3:2 pulldown (telecine) hard encoded into the video. Most films are stored progressively at 24 FPS (soft telecine), but some are not. They are stored interlaced at 30FPS like a vhs tape. When you just use a plain deinterlacer back to 24 FPS, you run the risk of the encoder dropping frames and duplicating others, leading to major choppiness. You have a 2/5 chance of this happing. But some movies have the 3:2 pattern change throughout the movie, So a deinterlacer is useless against those. You need a encoder that can perform inverse telecine (IVTC) to be sure that it comes out right.
Now the tuff part is to identify each scenario in the dvd. a good explanation of some of what im talking about (interlaced frames) is at
http://www.100fps.com. It mainly talks about European video, but if you can translate 25FPS to 30FPS and 50 fields to 60 fields, you can get american framerates.
Good luck and hope ive shed some light.
PS I think your problem is the first issue, 24fps to 30. so check that issue first.