HD Video Editing Workflow and Hardware Advice from the PROs

[Dr Strangelove]

Up until now, I have been using an external FW800 drive with my Macbook Pro and Final Cut Express to edit my weekly video podcast at http://mmanuts.com. I just decided to ditch the laptop in favor of a new i7 iMAC (en route) and ipad instead.

I am looking to optimize my editing workflow with this new machine and would love some advice from the pros on both hardware and software

If I plan to use the following programs, what would be the ideal hard drive set up? Does using multiple versions of RAID and separating the rendering, scratch etc storage make sense?

Programs to be used:

Final Cut X (and other programs)
iMovie
Adobe After Effects


Basically we shoot a 60-90 min show every week that gets releases on our website in various formats as well as on Youtube and itunes/Zune etc.


Current workflow:

-60-90min 720p .mp4 exported from Final Cut
-Same segment as above 640x360 itunes version exported from Final Cut
-Short Highlight clips cut up from the main show exported from Final Cut in 720P


Any advice in how I can cut down on time spent waiting for conversion and optimize speed using external hard drives would be appreciated greatly.

 

[Dr Strangelove]

Nobody?

 

[LethalWolfe]

My first suggestion is not to assume that FCP X is going to have the features or the reliability you need right out of the gate. No one knows anything about it and Apple reps have told people to think of it as a Version 1 piece of software.

My second suggestion is to use something other than Final Cu Express to do your exports. Both Compressor (comes with final cut studio) and MPEG Streamclip (free) allow for batch processing and will yield better quality than Final Cut.

As far as HDD setup goes, what kind of footage are you editing with? For what you guys are doing I don't think the HDD itself is the bottle neck but the CPU. I mean, for your podcast you wouldn't gain anything by buying a fast RAID and editing in Uncompressed or anything like that.


Lethal

 

[Richardthe4th]

Totally agree with Lethal.

You can speed up your workflow by using FCSuite (with compressor), so using FCP instead of FCE. No problem, it has the same user interface.

I think firewire800 disks are sufficient, no use for raids (make backup's on a seperate). If you buy an iMac with Thunderbolt (for quicker external HD's, but I don't think you need the speed, it's still nice though); and you have some money left you can buy a Matrox MXO2 with max. It speeds up compressor crunching MP4's/H264. Make a compressor droplet and you can speed up you're workflow. That's where I would put my money...

Oh, and FCS comes with motion included. So you don't need AE

 

[Dr Strangelove]

Thank you both for the response.

I am currently editing AVCHD files shot on a Canon camcorder (will likely upgrade this within the next year or so to something more PRO-sumer)

I import the files using FCE log&transfer and then edit the final version of the show.

Are you suggesting I don't wait and buy FCP X then? I have been waiting for this to release for some time now instead of upgrading my Final Cut Express to FINAL CUT PRO.

What does Compressor give me that exporting from FCE does not? I hear that its buggy no? Also for Streamclip, what would I do, just export a RAW version of the final show and then cut it up that way?

Thanks again for any more advice.

 

[Richardthe4th]

We all have to wait to see what FCP X will be.

But, if you -only- use AVCHD files, and want a show on air every week without to much hassle, perhaps then FCP X is right for you. Edit the native AVCHD, share to different formats and ready.

But wait if FCP X will offer this and then decide. (And you have to learn a new interface, but that can only be fun.)

 

[Dr Strangelove]

Thanks Richard.


What about the hard drive configuration, does it make any sense to have rendering drives on a RAID0 and the imported files on a RAID5?

 

[LethalWolfe]

I import the files using FCE log&transfer and then edit the final version of the show.

What codec do you import the AVCHD as?

Are you suggesting I don't wait and buy FCP X then? I have been waiting for this to release for some time now instead of upgrading my Final Cut Express to FINAL CUT PRO.

I'd wait. But I always wait. I want to edit, not be an unpaid beta tester/bug hunter for Apple.

What does Compressor give me that exporting from FCE does not? I hear that its buggy no? Also for Streamclip, what would I do, just export a RAW version of the final show and then cut it up that way?

It gives you significantly more options, batching processing, and a higher quality transcoding engine. The Qmaster component of Compressor (which allows you to network render, or 'trick' the program into using more CPU cores) is a notorious flake but you don't have to use it if you don't want to. Some people have also complained that sending your timeline directly to Compressor from FCP is flaky but I've rarely have a problem with and have used it extensively on long form projects.

For MPEGStream Clip you could export one, full-res master version of your show then drop it into the program and have it make the MP4 and iTunes exports.

What about the hard drive configuration, does it make any sense to have rendering drives on a RAID0 and the imported files on a RAID5?

I don't think you need to do that unless you are working a huge codec like Uncompressed or ProRes HQ (which are both massive overkill for your needs).


Lethal

 

[jettoblack]

The point of highly compressed codecs like AVCHD, MP4, H264, etc. (all basically the same thing) is to squeeze the most quality into very limited number of bytes. It's a trade off of needing more CPU power but less disk space or bandwidth. Hard drive speed will have almost no effect when editing and compressing to these formats, as most of the time is spent by the CPU doing heavy computation, and the actual disk access takes relatively little time.

Contrast this to a codec like ProRes, AIC, or uncompressed, which have relatively low CPU usage but massive disk and bandwidth requirements as the files are very large.

To speed up your final exports, I'd recommend you look into a turbo.264 HD from Elgato. It's a USB stick with a H.264 codec chip inside which can compress video to MP4/H.264 many times faster than the fastest CPUs can and with equal or better quality. Plus it has plugins for FCP/Compressor, and presets for different resolutions & bandwidth such as for iTunes or YouTube.
http://www.elgato.com/elgato/na/mainmenu/products/Turbo264HD/product1.en.html

 

[Dr Strangelove]

The point of highly compressed codecs like AVCHD, MP4, H264, etc. (all basically the same thing) is to squeeze the most quality into very limited number of bytes. It's a trade off of needing more CPU power but less disk space or bandwidth. Hard drive speed will have almost no effect when editing and compressing to these formats, as most of the time is spent by the CPU doing heavy computation, and the actual disk access takes relatively little time.

Contrast this to a codec like ProRes, AIC, or uncompressed, which have relatively low CPU usage but massive disk and bandwidth requirements as the files are very large.

To speed up your final exports, I'd recommend you look into a turbo.264 HD from Elgato. It's a USB stick with a H.264 codec chip inside which can compress video to MP4/H.264 many times faster than the fastest CPUs can and with equal or better quality. Plus it has plugins for FCP/Compressor, and presets for different resolutions & bandwidth such as for iTunes or YouTube.
http://www.elgato.com/elgato/na/mainmenu/products/Turbo264HD/product1.en.html

Whoa whoa whoa, this Elgato thing could be a gamechanger for me....is this the only thing of its kind or are there other brands also?

I really don't see having a need for PROres in the near future even if we upgrade cameras etc.

 

[Dr Strangelove]

What codec do you import the AVCHD as?


I'd wait. But I always wait. I want to edit, not be an unpaid beta tester/bug hunter for Apple.


It gives you significantly more options, batching processing, and a higher quality transcoding engine. The Qmaster component of Compressor (which allows you to network render, or 'trick' the program into using more CPU cores) is a notorious flake but you don't have to use it if you don't want to. Some people have also complained that sending your timeline directly to Compressor from FCP is flaky but I've rarely have a problem with and have used it extensively on long form projects.

For MPEGStream Clip you could export one, full-res master version of your show then drop it into the program and have it make the MP4 and iTunes exports.


I don't think you need to do that unless you are working a huge codec like Uncompressed or ProRes HQ (which are both massive overkill for your needs).


Lethal

Hmm, are there other programs out there like this? This may be the ticket for a workflow that makes more sense. Have never tried FCP, I should probably make a trip to the Apple store to play around with it. I understand the interface is similar to FCE which I am already using.

 

[wonderspark]

I'll add a couple things to consider:

1) Budget vs. value of your time
If you're doing a 90 minute show every week, I'd think you could benefit from a stronger system than using a laptop, but seeing as you've been doing this for a while already, you must have it together already. The things I edit are relatively complex 1080p, and I'm frequently finding tricks and workarounds to get the job done despite having 16GB of RAM in an '09 Mac Pro.

2) Will you be expanding?
Maybe you're planning to move from 720p to 1080p. Maybe you're going to incorporate more After Effects work, or more complex effects. If so, then I think stronger hardware and RAID / backups are going to be valuable.

I'm an advocate of RAID0 for renders and files that require fast read/write, RAID3 or RAID5 for media storage of working files, and external backups on single disks.

What I feel it comes down to is, how valuable is your time, and is it worth it to spend more to save that time? You have to make that call.

 

[LethalWolfe]

Hmm, are there other programs out there like this? This may be the ticket for a workflow that makes more sense. Have never tried FCP, I should probably make a trip to the Apple store to play around with it. I understand the interface is similar to FCE which I am already using.

There are other compression programs such as Telestream Episode and Sorenson Squeeze, but those programs run into the hundreds of dollars each and are probably overkill for your needs.

FCE and FCP have the exact same interface. The only difference is FCP has more features and it comes in the Final Cut Studio suite of apps.


Lethal

 

[mBox]

There are other compression programs such as Telestream Episode and Sorenson Squeeze, but those programs run into the hundreds of dollars each and are probably overkill for your needs...

We use MainConcept Reference for MPEG encoding. You can buy a module for the other codecs. Super fast and more robust than Squeeze. Im not sure if they have a hardware codec option, most of my work is clocked under 30 seconds each

 

[Dr Strangelove]

I'll add a couple things to consider:

1) Budget vs. value of your time
If you're doing a 90 minute show every week, I'd think you could benefit from a stronger system than using a laptop, but seeing as you've been doing this for a while already, you must have it together already. The things I edit are relatively complex 1080p, and I'm frequently finding tricks and workarounds to get the job done despite having 16GB of RAM in an '09 Mac Pro.

2) Will you be expanding?
Maybe you're planning to move from 720p to 1080p. Maybe you're going to incorporate more After Effects work, or more complex effects. If so, then I think stronger hardware and RAID / backups are going to be valuable.

I'm an advocate of RAID0 for renders and files that require fast read/write, RAID3 or RAID5 for media storage of working files, and external backups on single disks.

What I feel it comes down to is, how valuable is your time, and is it worth it to spend more to save that time? You have to make that call.

Thanks for the post, more horsepower is the reason I just sold my macbook and upgraded to an i7 iMac which is on the way. My camera shoots in 1080p but I only release the show in 720p and don't really plan to increase this. I may do a little more AE work down the road but nothing substantial at this time.

 

[Dr Strangelove]

We use MainConcept Reference for MPEG encoding. You can buy a module for the other codecs. Super fast and more robust than Squeeze. Im not sure if they have a hardware codec option, most of my work is clocked under 30 seconds each

This would be better than using the Elgato solution?

 

[mBox]

This would be better than using the Elgato solution?

Nope just other options under the 300 dollar mark.

 

[Dr Strangelove]

Nope just other options under the 300 dollar mark.

The USB key seems to make sense as long as it supports iMovie, FCE and FCP X I should be all set.

I've been searching around and it seems like there were some quality issues in the beginning, does anyone know if they have been solved?

Also, when using the Elgato setting in Final Cut, can I still choose resolution size when creating my MP4?

 

[digitalfurball]

The USB key seems to make sense as long as it supports iMovie, FCE and FCP X I should be all set.

I've been searching around and it seems like there were some quality issues in the beginning, does anyone know if they have been solved?

Also, when using the Elgato setting in Final Cut, can I still choose resolution size when creating my MP4?

I registered so I could answer this a little for you. I use it professionally on a daily basis for getting rough edit over to clients. It hands down beats compressor and mpeg streamclip for speed, by a massive margin. Quality is on a par with streamclip, but not as good as compressor (with settings on full)

There were some audio sync issues with earlier versions of the software, but I only ever found them with footage shot XDCAM EX. Now these have been sorted out for me. Quality does strangely seem to be higher when the key is disconnected, but it's marginal.

I have found the best workflow is to export a quicktime at sequence settings and then drop into the provided software. There are plenty of presets, but you can customise and create your own, which is excellent. You can also batch and batch the same source file to multiple different outputs too.

Now comes the interesting bit regarding your RAID question and the reason I found this thread.

Using ProRes, on a brand new MBP i7 quad core. the fps cap out at 60fps (2x realtime) What I'm finding though is that the drives are maxing out pushing that much data 60MB per second or more. This is limiting the encode speed. I have a feeling it could go much faster with an SSD or some sort of thunderbolt RAID, although i can't confirm it.

What I can confirm is that much lower bitrate movies (apple trailers for example) encode at 120fps

If it's something you have to do an awful lot, a smallish RAID connected with something which can deliver 80MB/s or more or SSD might be a good option

 

[Dr Strangelove]

I registered so I could answer this a little for you. I use it professionally on a daily basis for getting rough edit over to clients. It hands down beats compressor and mpeg streamclip for speed, by a massive margin. Quality is on a par with streamclip, but not as good as compressor (with settings on full)

There were some audio sync issues with earlier versions of the software, but I only ever found them with footage shot XDCAM EX. Now these have been sorted out for me. Quality does strangely seem to be higher when the key is disconnected, but it's marginal.

I have found the best workflow is to export a quicktime at sequence settings and then drop into the provided software. There are plenty of presets, but you can customise and create your own, which is excellent. You can also batch and batch the same source file to multiple different outputs too.

Now comes the interesting bit regarding your RAID question and the reason I found this thread.

Using ProRes, on a brand new MBP i7 quad core. the fps cap out at 60fps (2x realtime) What I'm finding though is that the drives are maxing out pushing that much data 60MB per second or more. This is limiting the encode speed. I have a feeling it could go much faster with an SSD or some sort of thunderbolt RAID, although i can't confirm it.

What I can confirm is that much lower bitrate movies (apple trailers for example) encode at 120fps

If it's something you have to do an awful lot, a smallish RAID connected with something which can deliver 80MB/s or more or SSD might be a good option

Thank you for the post on this. I am still debating what to do. Received my imac this week and it is much faster in comparison to my macbook pro, which I expected. I am mostly exporting AVCHD video to Youtube and itunes, do you think the Elgato stick's quality is good enough for those mediums?

As for the RAID suggestion, I will wait till next week to see what is and is not released and then likely pull the trigger on an enclosure. Waiting any longer is becoming increasingly difficult.

For my purposes ProRes LT should be sufficient, no? Are those limits you mentioned the same for that format?

 

[digitalfurball]

Thank you for the post on this. I am still debating what to do. Received my imac this week and it is much faster in comparison to my macbook pro, which I expected. I am mostly exporting AVCHD video to Youtube and itunes, do you think the Elgato stick's quality is good enough for those mediums?

As for the RAID suggestion, I will wait till next week to see what is and is not released and then likely pull the trigger on an enclosure. Waiting any longer is becoming increasingly difficult.

For my purposes ProRes LT should be sufficient, no? Are those limits you mentioned the same for that format?

I think the quality is fine for those mediums. And at the price the benefit in speed is definitely worth it. And at the kind of level of shooting where a much higher quality encode is needed, the higher end solutions would pay for themselves in a single shoot.

ProRes LT is still a relatively high bit rate, so I think there'd still be that "issue". But I'd say the imac will have a much faster drive than the laptop i'm using here, so you may find your encodes are going up to 3x realtime... if that seems like a good enough improvement on what you're doing currently then I say go for it!

 

[Dr Strangelove]

I think the quality is fine for those mediums. And at the price the benefit in speed is definitely worth it. And at the kind of level of shooting where a much higher quality encode is needed, the higher end solutions would pay for themselves in a single shoot.

ProRes LT is still a relatively high bit rate, so I think there'd still be that "issue". But I'd say the imac will have a much faster drive than the laptop i'm using here, so you may find your encodes are going up to 3x realtime... if that seems like a good enough improvement on what you're doing currently then I say go for it!

Hmm, so export from FCP to QT and then use the Elgato software to batch versus using MPEG Streamclip. Its not possible to use Elgato with MPEG streamclip correct?

 

[digitalfurball]

Hmm, so export from FCP to QT and then use the Elgato software to batch versus using MPEG Streamclip. Its not possible to use Elgato with MPEG streamclip correct?

You can use it with MPEG Streamclip if you wish, but i find the elgato software simpler.

You can also output directly from Final Cut Pro if you wish, but I find having a full quality export is handy for various reasons (if you want to go to WMV or FLV etc) it also frees up Final Cut again.