What settings did you use?
I used the CLI (linux server farm), here are the standard CLI options I used for all movies, you can set most if not all these in the GUI as well.
--longest --format mp4 --encoder x264 -P 16 --aencoder aac+ac3 -B 160 -R 48 -2 --markers -x264opts bframes=5:ref=1:subme=5:me=umh:no-fast-pskip=1:trellis=1:cabac=0 -T
Then, depending on the movie and how important quality was for that movie, I used one of the following in addition to the above standard options:
-b 2500 (Really low quality movie to begin with. Like an old 60's movie in 4:3, etc)
-b 3000 (Comedies, etc that don't have much panoramic or action)
-b 3500 (more action oriented movie, but still not something I'd be picky about quality)
--size 4000 (Anything I wanted maxed out quality on, this is by far the one I used most often)
--size xxxx (I found this an easier way to control quality than bitrate. I'd try a small size, like 2000 or so and if not good enough, increase it until I was happy with the results)
As I already said, these options (using --size 4000 especially) aren't for the faint of heart. It's going to take a minimum of 3 hours to encode an hour and 20 minute flick. (Your results may vary depending on hardware)
Frankly, the guys at handbrake can probably tear apart my options and make even better recommendations, but barring spending months playing around with possibilities, I'm very happy with the results on both my 50" from my mac mini and my 42" from my ATV in the bedroom.