Hard drive speed and render times – where’s the bottleneck?

[Chris7]

Hard drive speed and render times – where’s the bottleneck?

I just got FCP and a Canon HF100.

I’m currently using a MacBook Pro, to upgrade to a MacPro hopefully in about a year.

From what little I know about FCP so far, it looks like I’ll be editing in ProRes SQ 1920 x1080, using the 384 x 216 Photo JPEG proxies. For most of my videos I’ll use no more than two streams, but I’d like to also do some projects with a montage of video images, perhaps up to three to five at one time. I’ll eventually export them to duel-layer DVD for playback on a Blue Ray player.

It looks like LaCie just came out with a four drive hardware RAID set up that has an eSATA connection, which will work with my MacBook Pro through the Experess card 34 slot. It has a claimed “burst” transfer rate of up to 200 MB/sec. on an empty drive, and the safety of RAID 5. I just bought a Caldigit FireWireVR, which would only do about 70-95 MB/sec in RAID 0 or half that that in RAID 1. I realized I probably have to use RAID 1 to protect the data. I packed it up for return when I heard of the LaCie.

My question is, if I’m using three to five streams of video, is my 2.2 GHz Core Duo going to be slower than the LaCie drive anyway when rendering? Or can I get a faster render with the LaCie than the Caldigit, even on my humble MacBook Pro? When I upgrade to a literally five times faster Mac Pro, will the faster HD effect render?

 

[-DH]

Rendering in FCP is primarily dependent on processor speed, with the exception of FXPlugs which are dependent on the power of the graphics card. As far as the number of streams that you'll be able to playback in RT, an eSATA RAID configuration would provide the best throughput on a laptop. But it will also depend on your RT and playback quality settings in FCP.

Since the HF100 records in AVCHD, capturing/editing in ProRes will help speed things up as AVCHD is very processor intensive to work with.

-DH

 

[LethalWolfe]

"Burst rate" is not a good indicator of speed for video editing because we need an consistent, uninterrupted flow of data (the video stream). Sustained transfer rate is a much more meaningful number. Also, LaCie's have a reputation of poorly engineered enclosures that are failure prone and that don't keep the drives cool enough leading to premature drive failure. I use a LaCie as one of my back-up drives w/o a problem, but it remains off save for a few hours a week. I'd never feel comfortable using one as an editing drive.

As -DH said, and as I've said before, your bottle neck for pulling multiple HD streams, resizing them and animating them around the the frame is going to be your CPU. If you want to get more RT performance you can lower the quality of your playback settings (basically putting FCP playback into 'draft quality') and the QuickView tool can be very handy as well (a 2-10 second RAM preview tool that live updates). When you move up to a Mac Pro your render times will definitely drop.

I don't think that using proxies is a good way to go as that will take up more drive space and it will unnecessarily complicate your workflow. No one ever uses the terms "robust" and "media management" together to describe FCP for a reason. The more simple your work flow the happy FCP will be and the happier you will be.


Lethal

 

[Chris7]

Thanks for the help. Some of these questions will seem obvious when I learn FCP, but I can’t learn how to work in HD until I have an external hard drive.

1) CalDigit also makes a dual-disk RAID 0/ RAID 1 box that will take eSATA, the S2VR Duo, and they’re coming out with a cheaper version without the PCIe or PCI-X card. CaptinChunk posted that it gets 150 MB/sec empty and 105 MB/sec 90% full in RAID 0. But RAID 0 makes me nervous as a HD could fail. In RAID 1, this would mean about 50-75 MB/sec. It would seem that this could only handle one or two streams of 20 MB/sec ProRes in real time playback. LaCie is the only company I know of that makes a 4 disk hard drive that will work with an ExpressCard 34 slot. Do you know of any others?

2) Proxies seemed like a pain to me, but I did not see any other way. If I do not use proxies, is there another way to adjust settings so that I can view in real time more streams than the hard drive will transfer at once? Intuitively, it would seem that this would not be possible.

3) Also, at what point does the hard drive speed/transport rate affect render after your done with the project (as apposed to real time render for playback)?

 

[KeithPratt]

I don't know who'd use RAID1 on the S2VR. You're paying a premium for its sustained speed and reliability (ventilation). It doesn't make sense to pay that price for a backup you could do to any old hard drive. How about using the S2VR in RAID0 as your scratch disk and clone it to a cheaper non-RAID disk when it's worthwhile.

And why do you need multiple streams of ProRes?

 

[RatVega]

Thanks for the help. Some of these questions will seem obvious when I learn FCP, but I can’t learn how to work in HD until I have an external hard drive.

1) CalDigit also makes a dual-disk RAID 0/ RAID 1 box that will take eSATA, the S2VR Duo, and they’re coming out with a cheaper version without the PCIe or PCI-X card. CaptinChunk posted that it gets 150 MB/sec empty and 105 MB/sec 90% full in RAID 0. But RAID 0 makes me nervous as a HD could fail. In RAID 1, this would mean about 50-75 MB/sec. It would seem that this could only handle one or two streams of 20 MB/sec ProRes in real time playback. LaCie is the only company I know of that makes a 4 disk hard drive that will work with an ExpressCard 34 slot. Do you know of any others?

2) Proxies seemed like a pain to me, but I did not see any other way. If I do not use proxies, is there another way to adjust settings so that I can view in real time more streams than the hard drive will transfer at once? Intuitively, it would seem that this would not be possible.

3) Also, at what point does the hard drive speed/transport rate affect render after your done with the project (as apposed to real time render for playback)?

Thing One: you don't want to sacrifice performance by using a RAID 1. If you are paranoid, run a RAID 5 on 4-5 spindles. I've got several RAID 0 arrays that have run for years (longest: 4.5 years, shortest: 9 months) without a problem. I need the speed and I'm not afraid to back up.

Thing Two: CalDigit and LaCie both make Express card that can handle four spindles. As far as I know, both do so using port multiplexing. At four drives, port muxing and "channel-per-drive" seem to behave about the same.

If you are determined to try to cut 4-5 streams with Multiclip, your best bet is to use Standard Definition. Five streams will require less than 30MB/s and you have enough processor power. If you can get on a RAID 0 or a hardware RAID 5, ProRes 422 will require about 70-100MB/s and the processing power is iffy... Efforts to render 5-6 layer compositions will be laughable. Although my signature doesn't reflect it yet, I'm rendering 1080 ProRes in Shake and 4-6 layer comps on a 8X 3.2GHz MacPro/10GB/2TB RAID 0 and even that isn't all that fast.

Yes, Proxies are a pain but they are a needed concession to a lack of processing power. You are trying to haul a house trailer with a VW Beetle... It's a cute little car but it's not a semi. We are using ProRes 422 as an intermediate more because we have to deal with HDV footage than for ease of cutting. Taking the HDV to a 10-bit color space as early as possible is the best way have to ensure a quality product on output. Having said that, transcoding to ProRes may be a smarter move than trying to process AVC HD directly.

As stated in the posts above, rendering is primarily a processor game. The one exception I've seen was back in the G4/FireWire days when transcoding SD to MPEG-2 was 25-30% faster if the footage was rendered to a RAID instead of a FW400 drive. I see this not as praise of the RAID but rather as an indictment of FW speed.

Bottom Line: You're asking a lot of a good (but not horrifically fast) laptop, but then it's your circus, and you can have as many clowns as you like...

 

[Chris7]

Thanks Lethal and RatVega. Ok, so the CPU is always going to be the bottleneck. The other info is really helpful, too.

I guess at this point, I suppose I need to decide how much hard drive I should even bother using with a 2.2 GHz MBP – a two disk hardware RAID 0 box (and just back it up to a regular drive), or a 4 disk hardware RAID 5 box.

I could probably make this decision if anyone could please answer a couple hypothetical questions…

A) Hypothetically, lets suppose I did have a hard drive that could read, write, and transfer up to 5 streams of 20MB/sec ProRes (1920 x 1080), and I did not use proxies, nor Motion (because it needs a hot graphics card for RT). If I just started layering plain supers (which again I would not do for a project – this is hypothetical), is there a way to adjust settings to be able to view in real time:
2 layers?
3 layers?
4 layers?
5 layers?

B) Same hypothetical question, except for after edit render times (editing is complete). If I were to layer supers of 20MB/sec ProRes, about how long would it take to render (after editing) a 5 minute video using:
2 layers?
3 layers?
4 layers?
5 layers?
(A very loose estimate here would of course be more than welcome).

Thanks in advance. If someone will answer this it will give me a huge head start.

 

[LethalWolfe]

Personally I'd go for RAID0 and then back up to separate FW HDDs that are kept on a shelf. RAID5 won't help you if you accidentally delete a file or if a power surge fries your gear. You can adjust the Real Time effects settings in FCP. For example, using "Unlimited RT" w/playback quality set to "low" will give you the most Real Time functionality but your video will appear low quality during playback.

I don't have a laptop so I can't give you estimates on render times.


Lethal