At low to moderate bitrates Handbrake will definitely produce better quality than Compressor and Handbrake is faster too (go figure). However, you may find that Compressor's H.264 output is a little bit more compatible with Apple's hardware products than Handbrake (but many use Handbrake without problems, so YMMV).
However, if you are doing relatively high-bit-rate encodes (let's say above 2Mbps for standard definition output) then I'd probably stick with Compressor since you're likely to see little difference in the relative quality of the output. The only exception to this "rule" is if you need to do any deinterlacing, in which case I'd stick with Handbrake's decomb filter which is much, MUCH, MUCH faster than anything of equal quality from Compressor (in fact, in terms of basic quality, Handbrake's decomb/deinterlace will typically exceed anything you can do in Compressor). That's not to say that you can't get good quality deinterlacing and rate conversion from Compressor, it's just that you may find that Handbrake will work with a much wider variety of cadences and sources (for deinterlacing).