It's been a long month plus of testing, and I've got a few results to share....
Handbrake .96 is awesome if you don't care about filesize (or don't want to do the bitrate math to figure out what the final size will be.
Because of this I've reverted back to Handbrake .95 since that gives me the option to set what the final file size will be (+/- a few hundred MB).
These are the settings I'm using to encode my ripped 1080p BluRays (Ripped using MakeMKV).
First thing I do is select the High Profile preset on the presets tab, then I change the following:
Picture Settings: Anamorphic: Strict (didn't touch any other settings)
Tabs:
Video
Target Size (MB), I've been using 3500 for dramas, and 4000 for action or longer movies. Also check 2 pass encoding. I don't use Turbo first pass...
Audio
I just use AAC (CoreAudio), Dolby PL II, Sample set to Auto, and set my bitrate to 320. I delete all other audio tracks besides the main, and only include the one listed above since these are only for my iPad. (My use, yours may be different)
Subtitles
I don't do anything here.
Advanced This is where things get interesting......
Ok, I've tried nearly everything under the sun, some work great on a computer, but won't sync to iPad, some won't play period. My encodes are ONLY for the iPad, which makes 2-4 day encodes annoying when they don't work. When using Handbrake .96 I was getting amazing quality picture, but that mostly was because each encode wound up being 7-12GB. Great quality, but waaay too big for my iPad. At that size and since HD's are pretty cheap I might as well just keep the 25-50GB MKV rips and play them natively on my machine..
So back to .95 I went using set file size as listed above.
I may revise as I tweak a little more, remember this is just the best I can squeeze out of a mkv in the 3.5-4GB Range.
OK, going right to left here:
Partition Types: All
Trellis: Allways
Deblocking: This depends on the type of movie. If it's a grainy HD film I set it to -1 or -2. If it's a HD video, I leave it at zero. I want to preserve that "Grainy film look" that most films have so I do the -1 or -2 so Handbrake doesn't remove the grain. Actual video (TV shows, etc) is crystal clear and mostly grainless already so leaving it at 0 is ok.
Adaptive B--Frames: Optimal
Adaptive Direct Mode: Automatic
Motion Estimation Method: Transformed Exhausted
Subpixel ME & Mode Decision: 10
Motion Estimation Range: 64 (48 is ok too, but I'm going for max quality)
I don't mess with Adaptive Quantization, Psychovisual Rate Distortion, or Psychovisual trellis.
Reference Frames: 4
Maximum B-Frames: 4
CABAC Entropy Coding: Yes
8x8 Transform: Yes
Weighted P-Frames: Yes
Pyramidal B-Frames: (Default)
No DCT Decimation: No
There is one final thing: The Advanced option string at this point should look like this (with deblocking set to -1, -1)
b-adapt=2:rc-lookahead=50:analyse=all:trellis=2:direct=auto:me=tesa:subq=10:merange=64:ref=4:bframes=4:deblock=-1,-1
See where it says lookahead=50? If you have 4GB or ram or more change this from 50 to somewhere between 80-100. If you have 4GB or less leave it where it is. I have between 16-32GB of ram depending on what machine I use, and I set it to 80 personally. YMMV.
Finally sometimes when I have all this set and click "Start", it won't start. I quit HB, (it complains, but I quit) and then relaunch. It asks me to reload the queue, I do, then click start and it will go.
Seems like the Reference Frames, and Maximum B-Frames are the stickler for compatibility, also if you check "No DCT Decimation" the encodes won't work.
Be aware these encodes take awhile, a long while. On my main workhorse (2008 MacPro, 8 core 3.2Ghz, 16GB ram) they take 2-4 days depending on movie length. This obviously isn't for everyone, but my goal with all this was max quality when time and horsepower wasn't an issue. If you don't care about filesize and want a quick encode I just say use .96 and click the High Profile preset, and let 'er rip!
I'm still tinkering with the Ref Frame and Maximum B frame settings, so I'll report back if I can successfully crank those up any higher and still get the movies to sync to the iPad.