Let me try again...that Dexter rip likely had quite a few dark scenes. Live action movies (as opposed to a cartoon/CGI Disney Pixar film) is going to have a good amount of grain, and image noise in dark scenes. Image noise will look grainy on a high bitrate source. When you convert it with Handbrake, you/Handbrake have a choice: It can maintain a high bitrate (at least in these problematic scenes), which will drive up your filesize, or it can essentially de-noise the scene and compress it more (smaller filesize), but you'll end up with blotchy blocky patches.
If you're working with less grainy sources (and especially squeaky-clean CGI), Handbrake can do a remarkable job of compressing things down to a very small filesize while still maintaining a high level of PQ.
As for the iTunes file sizes...you also need to recognize that these files may be produced from even higher quality sources and there may be human involvement in evaluating the movies scene-by-scene during the compression process and adjusting different values as needed. It's also quite possible that you'll see similar artifacting in the same problematic scenes. Compare apples to apples: Find out specifically which episodes are on that disc and add up the total filesizes of the individual 1080p episodes sold on iTunes. Then, buy the problematic episode and compare it directly to what you/Handbrake can produce.
Along the lines of my previous post...one approach you could take is to take the ATV3 preset and dial back the RF value (higher value = lower PQ but smaller filesize). Start by adding 2 to the value, compress just 2 minutes worth (Handbrake gives you this flexibility), and evaluate the end result. Keep doing this until the artifacting gets reintroduced. Then, set it back to the previous value, convert the entire disc, and see if you're happy with the resulting filesize.
Lastly, a really dumb/obvious question, but just in case...this Dexter disc has multiple episodes on it. What is the total runtime? It could be 3 hours or more. Obviously, the average 3-hour disc should have a 50% larger filesize than a 2-hour disc.