Insert a DVD, and point it to the disc location -- it should scan and find the Vob files. Sometimes you may have to navigate to the VIDEO_TS folder for it to find the files. If you're doing a DVD already ripped to the HDD, point it to the folder.
On the right hand side, at the top, there is a "toggle presets" button. This will show a list of pre-defined settings that should work just fine for what you need it for. I usually point towards "Normal" which outputs an MP4 file compatible with Quicktime, iTunes, possibly Apple TV (but not necessarily the iPod)
The bottom portion of the Handbrake window will have tabs, the first tab, "Video", has the settings you need to look at.
Frame rate: Same as source
Encoder: x264 (h.264 Main) -- this is quicktime, iTunes etc..
x264 (h.264 iPod) -- this should be selected if you use an iPod
Quality:
Target Size: if you select this, enter a size (like 700 MB) and it will make it a file roughly that size, sacrificing quality where it has to.
Average bitrate: IF this is selected, you may want a large storage drive. this is usually pre-determined based on the selected source. It would normally output a high-quality file over 1 GB in size.
Constant quality 50%: i never used that one.
With these settings, or just select "Normal" in the presets and limit it to 700 or 800 MB in size -- it should output a good quality file that plays in Quicktime, frontrow, iTunes, and more. If you selected the ipod (h.264 ipod) encoder, it may not play in Quicktime or Frontrow, only iTunes and the iPod