OK for that go with the "AppleTV 3" preset in Handbrake, then tweak the "Constant Quality" RF slider. General consensus seems to be setting the RF to around 16-18 will result in the best possible picture. Setting it any lower doesn't visually improve anything but just makes the files bigger. Default RF is 20 for that preset.
I'd suggest you take a DVD and choose just one chapter to encode, and try it a couple different ways and view the results. I took a random 5-minute chapter from a DVD and encoded it with straight "AppleTV 3" setting (RF 20), then at RF 16, then at RF 0. File sizes were:
RF 20: 81.2MB
RF 16: 145.4MB
RF 0: 977.4MB
Full resolution screen shots from each are attached. I can discern the difference between RF20 and the other two, but I'd be hard pressed to pick between RF16 and RF0 visually.