How can I achieve 720p Handbrake encodes that don't stutter?

Number6 · May 14, 2010

Is there anybody here who has no problems with HD files that conform to Apple's specs for the Apple TV? I just finished a long encoding and tagging session only to be totally frustrated. Two out of three files stutter (or drop frames) during playback on the Apple TV

At first I tried steaming them over my n-network wirelessly but that was totally unwatchable. After syncing them to the Apple TV things improved a little bit but it still seemed like the movie freezes and picks up speed again every once in a while. That is just not good enough.

These files play just fine in Quicktime and iTunes and I encoded them with Dynaflash's advanced options which I added to Handbrake's (Version 0.9.4 x86_64 (2009112300) Apple TV setting. The advanced options read like this:

ref=3:mixed-refs=1:bframes=3:me=hex:subq=7:b-adapt=2:8x8dct=1:weightb=0:trellis=0:weightp=0

Constant quality is at 60.78% or RF 20 and I bumped the resolution to 1280x whatever the aspect ratio requires. Is there anything wrong with these settings?
I went with the standard Apple TV settings in Handbrake at first but if I bump up the resolution the problems are the same and the 960x540 resolution doesn't cut it. The framerate on all these files is 23,98 so a real 720p resolution should not be problem.

The crazy thing is that one file works and plays without stuttering (even over the network!) so what can I do to make all files come out that way?

I've had these problems ever since I started encoding HD video and I find myself re-encoding the same movies over and over and I'm sick of it. The iTunes HD movies look great and have similar specs and I want my collection in that quality, but it seems unachievable. My WDTV HD Media Player plays nearly everything I throw at it but the interface sucks so hard I would rather live with 720p video if I could get it to work.

Any ideas are greatly appreciated...

NightStorm · May 14, 2010

I'm sure dynaflash will comment here soon enough, but you should remember that he syncs all of his content which gives him a little more flexibility on what the AppleTV can handle (eliminating the streaming overhead). My advice would be to download one of the nightly builds of Handbrake (lots of fixes for things you'll likely come across in HD sources) and try the stock AppleTV preset, modified to use 1280 width. The updated preset uses vbv limits to help prevent the bitrate spikes that cause stuttering on the AppleTV.

The only dropped frames that I see anymore are in the first few seconds as the AppleTV starts to buffer the content. If the problem persists, you could try increasing the RF value to 21-23 which is more in range with the values recommended over on the Handbrake forums for AppleTV 720p encodes (although I use RF20 without issue).

Number6 · May 14, 2010

That was quick! Thank you, I will try it tonight and post results.

NightStorm · May 14, 2010

Number6 said:
That was quick! Thank you, I will try it tonight and post results.

Sorry for all the edits to my original post; I should know better than to read Macrumors while sitting in a meeting.

There is a lot of info regarding Bluray/HDDVD sources and Handbrake in the thread linked below, although I admit the info I provided in the first post is a little dated. One of these days I'll get around to updating it.

http://forum.handbrake.fr/viewtopic.php?f=14&t=9258

gnasher729 · May 14, 2010

Increasing the resolution is a sure way to make it stutter. Use the original image size that Handbrake suggests, Strict Anamorphic in Picture Settings. Anything higher is a waste of CPU time. If your originals have bad quality, add a noise filter. You can try turning CABAC off, that is quite hard on playback, and you can try changing deblocking to (-1, -1) or (-2, -2); both options lose a bit of image quality but make it a lot easier on the decoder. And you might try what happens with "average bitrate" instead of "constant quality". Average bitrate would make every bit of the movie equally hard or easy to decode; "constant quality" would use more bits for critical parts of the movie to keep the quality up, but that might be what produces stuttering.

NightStorm · May 14, 2010

gnasher729 said:
Increasing the resolution is a sure way to make it stutter. Use the original image size that Handbrake suggests, Strict Anamorphic in Picture Settings. Anything higher is a waste of CPU time. If your originals have bad quality, add a noise filter. You can try turning CABAC off, that is quite hard on playback, and you can try changing deblocking to (-1, -1) or (-2, -2); both options lose a bit of image quality but make it a lot easier on the decoder. And you might try what happens with "average bitrate" instead of "constant quality". Average bitrate would make every bit of the movie equally hard or easy to decode; "constant quality" would use more bits for critical parts of the movie to keep the quality up, but that might be what produces stuttering.

The only thing that I really agree with you is disabling CABAC as it is likely the "straw that broke the camel's back" in this instance. The updated AppleTV preset disables CABAC and enforces vbv limits to prevent bitrate spikes larger than what the AppleTV can handle (as shown through testing). The AppleTV preset defaults to 960x540 to ensure maximum compatibility, but you can create 1280x720 encodes as long as you're aware of the playback limitations of the AppleTV.

That said, we're all entitled to our own opinions on what settings produce the best output. All that matters is that the results look good to you.

tommylotto · May 14, 2010

I know others here disagree, but my stuttering problem on certain files (e.g. Gladiator) would not go away no matter how I reduced the resolution of the encode. I was reducing a 1080p MKV to DVD resolution and it was still stuttering on the AppleTV. Then I changed the frame rate from "Same As Source" to the hard setting "23.98". BAM! Everything was perfect -- no stutter. I could even boost the resolution using the custom anamorphic setting to something like 1723x720 and still no stutter. It was beautiful. So, now, I first check in VLC to find out the actual fps of the MKV file and hard select that frame rate in handbrake as oppose to allowing handbrake to determine and select the frame rate for me. I think I have read that there is something wrong with some MKV's made by MakeMKV.

rayward · May 14, 2010

I have encoded a ton of Blu Rays through Handbrake. Every once in a while, one will stutter in the manner you describe. I simply re-encode the same rip using the same specs and so far not one has failed to come out perfectly on the second (or very occasionally third) try. It's a ballache to have to run the encode more than once, but it's either that or not have the movie in the library.

I have no idea why some come out that way. The only correlation I can infer is that the stuttering happens when I try to do anything else with my Mac while it's chewing on an HD encode. This makes no sense, but since I have made it SOP to avoid touching my Mac while it's at work (which includes not watching anything on my Apple TV that may be streamed) I have had 100% stutter-free results. If nothing else, this allows me to justify to myself the planned upgrade to an i7 iMac...

FWIW, I use the Apple TV preset, but adjust the picture settings so that anamorphic is set to "None" and the resolution is fixed at 1280 x xxx. That's just a personal preference, and I doubt it has anything to do with stuttering encodes.

roidy · May 14, 2010

These are the advanced options I use:-

All of my HD video is encoded at 1280x720 and has a bitrate of around 6-6.5Mbps, I stream all of my content over a wired connection and my ATV never has a problem. I curently use HB 0.9.4svn3154 but have never had problems even using older versions.

bucksaddle · May 14, 2010

As others have mentioned, and also shown in roidy's screenshot, disabling CABAC is the key to this one. Although not officially supported as per Apple's ATV specs page I think it is safe(ish) to use on SD material but I would say it was a no-go on an HD source.
Your other advanced settings look pretty close to what I use and I have no problems with HD playback, with CABAC off.

Number6 · May 14, 2010

First of all - thanks to all the people who replied. Just finished my re-encode and I am streaming (not syncing!) a 720p HD file with no stuttering to my Apple TV right now

The first thing I tried was what NightStorm suggested and got the latest nightly build of Handbrake. Using the new Apple TV preset I raised the 960 resolution to 1280 and that's it. The resulting file looks very sharp and finally there is no S-T-U-T-T-E-R!

The only thing that's not quite iTunes store HD quality are complex shades of color (mostly in backgrounds, sometimes in faces) which seem to become "busy" or having some kind of solarization effect. Hard to describe, might be because I am dealing with material that has been compressed before?

In the last column of roidy's settings there are some different values but I really don't know what they do. Maybe these could help with the "moving walls"?

Anyway, I will try the other failed encodes now, experiment a little bit and report back.

NightStorm · May 15, 2010

Number6 said:
First of all - thanks to all the people who replied. Just finished my re-encode and I am streaming (not syncing!) a 720p HD file with no stuttering to my Apple TV right now

The first thing I tried was what NightStorm suggested and got the latest nightly build of Handbrake. Using the new Apple TV preset I raised the 960 resolution to 1280 and that's it. The resulting file looks very sharp and finally there is no S-T-U-T-T-E-R!

Glad to hear that worked for you. I've been using those settings for 50+ Bluray/HDDVD encodes with no issues so far.

The only thing that's not quite iTunes store HD quality are complex shades of color (mostly in backgrounds, sometimes in faces) which seem to become "busy" or having some kind of solarization effect. Hard to describe, might be because I am dealing with material that has been compressed before?

You're on the right track here... Apple/"the studios" have access to much higher quality sources when they encode for sale in the iTunes store.

roidy · May 15, 2010

Number6 said:
The only thing that's not quite iTunes store HD quality are complex shades of color (mostly in backgrounds, sometimes in faces) which seem to become "busy" or having some kind of solarization effect. Hard to describe, might be because I am dealing with material that has been compressed before?

In the last column of roidy's settings there are some different values but I really don't know what they do. Maybe these could help with the "moving walls"?

Yep if you look at my screen shot you'll see I use both the 'No Fast-P Skip' and 'No DCT-Decimate' options and a deblocking value of -1,-1.

First the 'No Fast-P Skip' option is the one you want to stop the problem your having, namely the color banding(solarization effect). Don't let anyone tell you that color banding is because iTunes uses higher quality source because it isn't, I've seen the same problem on quite a few iTunes purchased encodes. If there is no color banding on your source material then there is no reason it should be on your encoded version.

Second the 'No DCT-Decimate' option will prevent the encoder from destroying subtle detail in the source material like filmgrain.

Third, the -1,-1 deblocking values are because I think Handbrake makes your encodes too soft and blurry with the default values.

The only downside to these options is a slightly longer encode time. Give them a try and see.

roidy · May 15, 2010

NightStorm said:
You're on the right track here... Apple/"the studios" have access to much higher quality sources when they encode for sale in the iTunes store.

And yet alot of the content sold on the iTunes store is still poor quality. Things have been getting better lately through

This thread is a good example:-
https://forums.macrumors.com/threads/877155/

stevol · May 15, 2010

Wirelessly posted (Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16)

Can someone tell us where to get the latest build of handbrake? I went to the supposed "snapshot" page, but no download was available. Thanks.

NightStorm · May 15, 2010

stevol said:
Wirelessly posted (Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16)

Can someone tell us where to get the latest build of handbrake? I went to the supposed "snapshot" page, but no download was available. Thanks.

Handbrake switched to nightly builds... There is a sticky in the handbrake forums with all the info you need.

stevol · May 15, 2010

Wirelessly posted (Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16)

Does the more recent nightly build(s) have any significant changes or improvements to the ATV preset? Would love to have a built-in preset for mkv from blu-ray to ATV (like an ATV 720p preset). Then we can relax on these forums knowing that the issues we are having are resolved (for now) and that the frustrating process has now become a piece of cake!

brentsg · May 16, 2010

The nightly builds aren't designed to address the presets, but to enhance code through bug and performance fixes.

You will have to experiment with the detailed settings on your own to identify a set that looks good to your eye, is optimal for your source material, and is optimal for whatever hardware you have on hand to encode with.

NightStorm · May 16, 2010

brentsg said:
The nightly builds aren't designed to address the presets, but to enhance code through bug and performance fixes.

You will have to experiment with the detailed settings on your own to identify a set that looks good to your eye, is optimal for your source material, and is optimal for whatever hardware you have on hand to encode with.

There are some changes to presets from .9.4 to the nightlies. For example, the AppleTV preset now uses vbv limits to prevent bitrate spikes that cause issues with smooth playback on the device, especially for HD sources.

brentsg · May 16, 2010

NightStorm said:
There are some changes to presets from .9.4 to the nightlies. For example, the AppleTV preset now uses vbv limits to prevent bitrate spikes that cause issues with smooth playback on the device, especially for HD sources.

Good to know, thanks.

gnasher729 · May 16, 2010

roidy said:
Third, the -1,-1 deblocking values are because I think Handbrake makes your encodes too soft and blurry with the default values.

People will argue whether stronger or weaker deblocking looks better, but deblocking needs to be done during playback, so less deblocking (-1, -1 or even -2, -2 instead of 0, 0) will use less CPU time during playback and therefore reduce stuttering.

roidy · May 16, 2010

I agree things like deblocking values are subjective, what I think looks good may not look good to everyone. However when I was testing setting to get the perfect encode I tried a wide range of deblocking values from -4,-4 right upto 4,4 and the ATV never had a problem playing any of them so while deblocking might have some effect on cpu usage during playback I doubt it would be to the extent as to cause any playback problems.

Number6 · May 16, 2010

Finished a couple of encodes with roidy's deblocking and anti-color banding settings and here are my (mixed) results. The movie that was finally stutter-free but still had "busy" walls sometimes (see post #11) was redone and showed very slight improvement in the walls, but the difference was so small that it could've been subjective. An A-B comparison was difficult since there was always a little time gap but in the end the results were very similar. File size and encoding time were nearly identical as well.

I checked the original mkv (a rather large 1080p encode) in VLC and the artifacts were definitely not there. So my encoding should be the culprit. I wonder if I increase the file size would the encode profit from it? Going from 13,23 GB to 2,63 GB, there must be some visible loss. The iTunes store file of "Quantum of Solace" is 3,8 GB, looks great and is roughly 20 minutes shorter in running time. I have a Drobo and space is really not much of an issue. I would even go for 6 GB if it yielded better results.

Than I checked another file that was very interesting. A VERY grainy 70's movie that originated from a 6,59 GB mkv and came out at 2,57 GB with only 88 minutes runtime. I started streaming the movie and after a couple of minutes I encountered the dreaded freeze and stutter again. Not as severe as before but still there. So I decided to sync the file and continued watching. There was no more stutter but the most amazing part was that the grain was intact! And I'm talking a lot of grain and fog here. No compression artifacts anywhere to be seen and the movie looked like it should. I never encoded this before but surely roidy's settings must have been a factor here.

This result is sure worth syncing and maybe if I had a wired network that wouldn't even be necessary. Anyway I am at loss on how to not just create bigger files but better ones. I know that I can set a desired file size in Handbrake but that does not seem very scientific. Which size is "best" for which movie? Most Handbrake users seem to advise on using constant quality.

If anyone has any further thoughts please share.

roidy · May 17, 2010

Umm... The fact that your original 1080p source file is only 13.23Gb means it's already been heaverly compressed to start with so getting a good re-encode from it may be difficult, plus 2.63Gb for a encoded HD movie seems very small to me. Use 'Get info' in iTunes to see what the actual bitrate for the file is. iTunes purchased HD content usually has a bitrate of about 4500kbps, all of my HD encodes have a bitrate of about 6000-6500kbps. Looking at your file size I think your bitrate maybe too low. Post all the settings your using.

NightStorm · May 17, 2010

roidy said:
Umm... The fact that your original 1080p source file is only 13.23Gb means it's already been heaverly compressed to start with so getting a good re-encode from it may be difficult, plus 2.63Gb for a encoded HD movie seems very small to me. Use 'Get info' in iTunes to see what the actual bitrate for the file is. iTunes purchased HD content usually has a bitrate of about 4500kbps, all of my HD encodes have a bitrate of about 6000-6500kbps. Looking at your file size I think your bitrate maybe too low. Post all the settings your using.

I have quite a few HDDVDs that, once extracted wtih eac3to and remuxed using tsmuxer, are approximately 13-15GB when all extras have been removed.

While 2.63GB may seem small, you've really got to trust that the x264 encoder knows what its doing when it decides how many bits to use for a particular section of video. For the longest time, I had trouble with how small my HD encodes were coming out of Handbrake (for example, my HP6 HD encode finished smaller than the digital copy included from the studio). But after doing a lot of A/B comparisons, I finally accepted that x264 knows what how to efficiently allocate/distribute bitrate without compromising quality or bloating encode size. If I had to give a range, I'd say my encodes typically finish somewhere between 2.5-4.5GB in size and look great on the AppleTV.

As stated previously, I use the stock AppleTV preset (from the nightly), with the width increased to 1280. Many devs/mods recommend increasing RF with HD sources, but I decided to keep it at 20 to try and keep my presets as similar as possible.

How can I achieve 720p Handbrake encodes that don't stutter?

macrumors member

macrumors 68000

macrumors member

macrumors 68000

Suspended

macrumors 68000

macrumors regular

macrumors 68000

macrumors 65816

macrumors 6502

macrumors member

macrumors 68000

macrumors 65816

macrumors 65816

macrumors member

macrumors 68000

macrumors member

macrumors 68040

macrumors 68000

macrumors 68040

Suspended

macrumors 65816

macrumors member

macrumors 65816

macrumors 68000

Our Staff