BluRay is 22mbps - 40mbps depending on the studio and length of film, and how many extras.
AppleTV3 can only handle 25mbps, so my guess is Handbrake encodes down to the 20-25mbps level.
iTunes 1080P is 4-5mbps, to allow streaming to the widest audience.
Those are some huge differences in encoding level... A better encoder gets you a little differences, but not a 400% improvement.
Some people don't like the clarity and color of the iTunes highly compressed video. For most, myself included, its sufficient, good enough. Actually I have mine set to pull down the 720P variants. Better bitrate/pixel, and smaller file sizes.