For DVD's I use the Highprofile setting, change the RF to 19% and get a very good quality rip. 5gb per film.
I spend a great deal of time on this recently to encode my DVD collection for use on Plex. I'm using the ATV2 with the Plex App and tried many Handbrake formats and Presets. I also used MakeMKV which takes the DVD VOB files and puts them together in a container so you can watch at full quality with no compression. I spent hours reading posts and doing rips and encoding tests because I have over 300 DVD's and don't wan't to do this twice.
The MakeMKV program produced files that were essentially exact copies of the source DVD's. If you have the Disk Space for storing and backing up then this will be the best quality but a typical DVD is 6-7 gigs so I ruled it our for that reason. All my other encoding tests were done in Handbrake (latest version) with the original ripped VIDEO_TS folders.
After doing all the research I decided to use the MKV file format in HandBrake. I did try a few MP4's but the file size seemed to be a little larger for the same RF (constant quality) number. Anyway I decided MKV H.264 was the way to go for me. Not sure if the ATV2 will play that natively and like I said I'm using the Plex App. (on jail-broken ATV2).
After doing a lot of comparisons of quality and video settings I decided to go with the High Profile Preset because it produces a smaller file size with very good quality (but longer encoding time). That also uses the Loose Amamorphic setting, and then I manually change the Format from MP4 to MKV. Finally I use a RF of 19.25 not sure why as it's hard to tell the difference between 18 and the recommended 20. But it works for me.
The encoded file sizes are averaging between 1.3 and 2.5 gigs and when I watch the movie it's hard to tell the difference from the original DVD. I haven't found a reliable way to do a batch convert on the mac, I tried DropFolders but it didn't work for me. So I load up the HandBrake Queue with about 30 movies at night and they are done by noon the next day. On a iMac27 i7 I'm getting encoding frame rates averaging 90fps or so depending on the DVD.
You can spend way too much time with this and get way over critical. Just pick a Handbrake preset and do a few tests on the device(s) you want to view the video on and then go from there. One thing to watch out for is that HandBrake doesn't always select the correct chapter to encode. Some DVD's have Fullscreen & Widescreen versions on the same disc and other times it will just select the wrong chapter all together. Because I'm doing so many movies now to enter my collection, I just do the encoding and then fix it later if I see any problems.