Using Handbrake to transcode to HEVC with hardware

[smithrh]

Preamble: I am no video expert by any means, and by extension the options available in Handbrake are mainly Greek to me. I'm sure there are further optimizations available and I'm happy to hear about them. But this is what I've found to work for me (really by accident vs. any plan) and so I thought I would share.

I've used - or tried to use - Handbrake in the past to transcode video with mixed results. Maybe the resulting quality was bad. Maybe the CPU usage went nuts and fans spun up. Maybe it took forever, etc etc etc, or worse, all the bad things all at the same time. I was puzzled by this, I figured my Macs should be eating this stuff up no problem.

Recently, somewhere - I've lost track of where, to be honest - there was a tip for setting Handbrake to use the built-in video hardware that the Apple Silicon chips have.

I tried it. Hmmmm... Almost no CPU used. Fans didn't spin up. There must have been a problem, the transcoded file is only 17% of the original. It must have died during the transcode or something like that.

Opened the transcoded file, and while it wasn't great, it was actually pretty good! Especially when it was only 17% of the size of the original file.

Decided to poke around more, and experimented with a handful of the settings to try to get the result a little better. What's below is what I came up with. AGAIN I am no expert. There are almost certainly further optimizations to be used, and again, I'm happy to hear about them!

Make sure you have the latest HandBrake version.

The key thing to get the video hardware involved is to use the Video Encoder setting of "H.265 (VideoToolbox)." I'm using that setting, not the 10-bit one but the results have been fine. If you have HDR content I'd try the 10-bit one. This setting is found in the Video section, top left. I set Framerate to "Same as source" as is Color Range. I set the Encoder Option Preset to speed (all the way to the left). Tune is none, Profile is auto as is Level. For Quality I selected Constant Quality, CQ of 60.

To make sure the video hardware is used, in the Filters section turn everything to Off. Otherwise you will get CPU ramping up pretty substantially.

Dimensions are up to you, you may or may not want to change the dimensions. I also did not change anything in Audio, Subtitles, Chapters.

While the improvements are generally seen without this, I also went into Handbrake Settings/Preferences in the Advanced section and selected to enable the Videotoolbox hardware decoders and encoders.

In the Queue section of settings, there's an option to encode N jobs simultaneously. While I was hoping to get better results with 4 or even 3, I believe that the video encoding/decoding hardware is the bottleneck and 2 is the sweet spot, being somewhat better than 1.

Does this result in extremely quick transcodes? Oh no. They still take time, but it's much better than before, and the CPU isn't getting killed at all with resulting heat. The quality is decent. And file size savings are anywhere from 90% to 60%. With hard drives and SSDs becoming expensive, this is a way to free up quite a bit of space at not much of a quality hit.

 

[!!!]

I'm using that setting, not the 10-bit one but the results have been fine. If you have HDR content I'd try the 10-bit one.

You should always use 10-bit for HEVC, it helps with the color banding, even if your source file is only 8-bit.

the CPU isn't getting killed at all with resulting heat

The CPU isn't going to get damaged from being used normally. All modern CPUs are designed to be capable of running at 100% for sustained periods.

As for using hardware encoding, if you think it's fine, then more power to you. But in general, even with Apple's encoders, the quality of software encodes is far, far better than hardware. I don't know what kind of videos you have, of course, but you could likely get down to about 5% of the original file size if you used a software encoder, and it'd surely still look better.

And file size savings are anywhere from 90% to 60%. With hard drives and SSDs becoming expensive, this is a way to free up quite a bit of space at not much of a quality hit.

You could probably get the same size savings with no quality hit using software.

 

[T'hain Esh Kelch]

I use Handbrakes AV1 encoding (RF18), and I encode a Blu-ray in 20-30 minutes on an M4 Pro, using software encoding. Compression is usually 83-60% over the original file, with no discernible loss of quality.

 

[smithrh]

You should always use 10-bit for HEVC, it helps with the color banding, even if your source file is only 8-bit.

The CPU isn't going to get damaged from being used normally. All modern CPUs are designed to be capable of running at 100% for sustained periods.

As for using hardware encoding, if you think it's fine, then more power to you. But in general, even with Apple's encoders, the quality of software encodes is far, far better than hardware. I don't know what kind of videos you have, of course, but you could likely get down to about 5% of the original file size if you used a software encoder, and it'd surely still look better.

You could probably get the same size savings with no quality hit using software.

Thanks!

I didn't delete any of the original videos yet, so I can try your suggestions out.

When I said "kill the CPU" I didn't mean in a literal sense, more in a heat and resulting fan noise way.

 

[smithrh]

Unfortunately, going to 10-bit (still hardware/VideoToolbox) results in much larger files.

Where I had been saving 60-70-80%+ on file sizes, 10-bit dropped that down to 20% or less, one file only saved 2%. Not worth it.

Trying the software route is a mess - now the options change in the video window so much it's not possible to know how to re-create the settings used in Hardware encode/decode. RF scale is different, for one, and now the Encoder Options have unhelpful settings like "Ultrafast" "Superfast" "Veryfast" and "Fast" (this is not all of the choices).

A few test encodes would up with horrible results, but using RF 22 and "Veryfast" yields a decent-looking result.

However the transcode time triples. And CPU is railed at 100%. File savings is on par with the hardware method, as is the resulting quality.

Lots of settings - probably too many, honestly - to look at still. But as it stand right now, software route is just slower, not by a small amount either.