Setting up RAID for HD editing

[Mr Garbaggio]

Ok, I have a couple questions kinda related to each other so here it goes. . .

So I have a Canon Vixia HF S200 which records in 1920x1080p. I've been recording in the native 24p option a lot but I'll probably also record a little in the 60i option as well. I would like to Record 1080p -> Edit 1080p -> Watch 1080p. No point of buying HD TVs + cameras if i'm not going to watch in the same resolution. I have a iMac 27" 2.93 ghz, 4gb RAM (planning on adding more) and I have Final Cut Studio 3. My father-in-law works for Western Digital so he was able to get me a My Book Studion LX with FireWire 800 and that's what I use currently for my scratch disk. Currently I havent really done anything with the footage from my Canon yet, I've been editing xbox gameplay vids. I'm not a professional at all, I just want to do this for my own enjoyment and fun. My questions are:

Question #1
When I start to edit the Canon 1920x1080 footage, I'm planning on transcoding to ProRes, so will the WD My Book Studio FW800 be able to do this? I would really like an answer of yes or no.

Question #2
Would I have to setup a RAID to be able to edit my HD footage? Which RAID option would be able to do this? Since my father-in-law works for WD, he can get stuff on the super cheap level, would the WD My Book Studio ll (has RAID option) be sufficient?

Question #3
This one is a rookie question, will any RAID setup work over FW 800? I know eSata is faster but is there a way I can set that up with an iMac?

So that sums up my problems. I read a ton! I read articles, blogs, forum threads, and any other thing I can find to keep learning and I've come pretty far but obviously there are still problems that I run into that I just cant figure out on my own so I'm appreciative to whoever replies. Thanks in advance!!


Mr Garbaggio

 

[alph45]

Ok, I have a couple questions kinda related to each other so here it goes. . .

So I have a Canon Vixia HF S200 which records in 1920x1080p. I've been recording in the native 24p option a lot but I'll probably also record a little in the 60i option as well. I would like to Record 1080p -> Edit 1080p -> Watch 1080p. No point of buying HD TVs + cameras if i'm not going to watch in the same resolution. I have a iMac 27" 2.93 ghz, 4gb RAM (planning on adding more) and I have Final Cut Studio 3. My father-in-law works for Western Digital so he was able to get me a My Book Studion LX with FireWire 800 and that's what I use currently for my scratch disk. Currently I havent really done anything with the footage from my Canon yet, I've been editing xbox gameplay vids. I'm not a professional at all, I just want to do this for my own enjoyment and fun. My questions are:

Question #1
When I start to edit the Canon 1920x1080 footage, I'm planning on transcoding to ProRes, so will the WD My Book Studio FW800 be able to do this? I would really like an answer of yes or no.

Question #2
Would I have to setup a RAID to be able to edit my HD footage? Which RAID option would be able to do this? Since my father-in-law works for WD, he can get stuff on the super cheap level, would the WD My Book Studio ll (has RAID option) be sufficient?

Question #3
This one is a rookie question, will any RAID setup work over FW 800? I know eSata is faster but is there a way I can set that up with an iMac?

So that sums up my problems. I read a ton! I read articles, blogs, forum threads, and any other thing I can find to keep learning and I've come pretty far but obviously there are still problems that I run into that I just cant figure out on my own so I'm appreciative to whoever replies. Thanks in advance!!


Mr Garbaggio

1. fw800 will work for HD. ProRez really helps. I've edited HD on lesser computers with no prob. The larger the format, however, the longer your render times. You can use downsampled footage if you run into problems and have FCP source your (original) HD footage for output. How long a piece your making is also a big influence.

2. you don't need a RAID, but they are faster than a single drive.

3. eSata isn't available on iMac's unless you hack yours or have someone like OWC do it for you.

 

[martinX]

2. you don't need a RAID, but they are faster than a single drive.

RAID 0 is. Raid 1, non.

A single G-Technology drive gets (claimed) 75 MB/sec read over FW800, about 600 mbps. That's reaching the top of the practical FW800 speed. I'd be surprised to see 4 streams of ProRes coming across that uninterrupted. FW800 just doesn't go any faster.

 

[mBox]

RAID 0 is. Raid 1, non.

A single G-Technology drive gets (claimed) 75 MB/sec read over FW800, about 600 mbps. That's reaching the top of the practical FW800 speed. I'd be surprised to see 4 streams of ProRes coming across that uninterrupted. FW800 just doesn't go any faster.

We use a variety for G-Tech single drives e.g. GRAID2 4TB, GRAIDmini 1TB and they handle the highest codec from Avid (DNxHD220) on our MacBook Pros running Media Composer.
If your using ProRes, your fine with FW800.
Ive overlooked hour long docs using ProRes edited on MacBook Pros.
Some of my clients dont like the proxy process (stubborn).
Since the final outputs go to BD for viewing, no need to go crazy with proxies.

 

[firestarter]

All recent full sized hard disk drives can manage a sustained data throughput that's higher than FW800 can support. There's really no point in considering faster external disk technology (RAID, SSD) unless you're using a faster interface like eSATA.

 

[Arrowk127]

Op Might want to consider Raid for redundancy rather than speed. If you are shooting to flash memory rather than tape once the card is full you offload and erase. Then there is no other copy elsewhere. A good backup solution is a must for editing should the HDD with all your footage on it dies.

 

[alph45]

RAID 0 is. Raid 1, non.

A single G-Technology drive gets (claimed) 75 MB/sec read over FW800, about 600 mbps. That's reaching the top of the practical FW800 speed. I'd be surprised to see 4 streams of ProRes coming across that uninterrupted. FW800 just doesn't go any faster.

- he asked if he needed a Raid setup to edit. i stopped at "no". Raid flavors are irrelevant after "no".

- he didn't ask if he could run 4 streams of HD, he asked if he could edit with an external firewire scratch. If I was intending to run 4 streams, I would use a flavor of ProRez like 422 LT for the lower data rate (4 streams weights in at 408 Mbps).

 

[martinX]

- he asked if he needed a Raid setup to edit. i stopped at "no". Raid flavors are irrelevant after "no".

I was addressing the bit after the "no". RAID flavours are relevant with a blanket statement like "they are faster than a single drive". In addition to that, not everyone gets the RAID variations. Since this is a resource for more people than the OP, my comment is relevant.

- he didn't ask if he could run 4 streams of HD, he asked if he could edit with an external firewire scratch. If I was intending to run 4 streams, I would use a flavor of ProRez like 422 LT for the lower data rate (4 streams weights in at 408 Mbps).

Based on numbers alone, you could "edit" ProRes from a USB2 drive if you limited yourself to a single stream. However, I decided to semi-quantify the "fw800 will work for HD" because 4 streams is easy to start doing and easy to want to start doing. He has FCS3 so multicam edits are an option and it's important to know what a system is capable of. Why would you expect him to limit his editing to 4 streams or less? In addition, he said he's transcode to ProRes, not ProResLT.

None of what I posted is wrong so I don't know why you're posting a reply.

 

[Mr Garbaggio]

Thanks for replying everyone! Just a couple of follow up questions:

Why would you expect him to limit his editing to 4 streams or less? In addition, he said he's transcode to ProRes, not ProResLT.

When you say 4 streams, do you mean 4 clips on top of each other playing at the same time? like the pic shows in the attachment.

I did a little more reading and found out that ProRes (normal) runs at 145 Mbps and ProRes HQ runs at 220 Mbps. So i'm assuming that the 800 Mbps (i know it doesnt actually go this high but neither do the 145 Mbps or 220 Mbps) is plenty fast. . . I just read over your reply again and now I get what you mean by "numbers alone", ProRes will run off of USB2. Also, does it matter how fast the rpm is on the drive for it to achieve the max data rate speeds?

Also, do you recommend one drive over another (sticking with FW800)? I just got the WD My Book Studio LX pretty much just because I can get it for half the price as retail.

Last question, based on "numbers alone", if i add different effects, audio, and graphics to my FCP project, does this add anything to the data rate that the scratch disk has to handle?

Again, thanks in advance for helping a rook out


Mr. Garbaggio

 

[mBox]

When you say 4 streams, do you mean 4 clips on top of each other playing at the same time? like the pic shows in the attachment.

Take four different HD clips and stack em.
Then scale each one quarter size and have them showing at the same time like a video wall.

 

[martinX]

I did a little more reading and found out that ProRes (normal) runs at 145 Mbps and ProRes HQ runs at 220 Mbps. So i'm assuming that the 800 Mbps (i know it doesnt actually go this high but neither do the 145 Mbps or 220 Mbps) is plenty fast

It's fine for a couple of streams. I was only kidding about the USB drive

Also, does it matter how fast the rpm is on the drive for it to achieve the max data rate speeds?

Yes. 7200 rpm is better than 5400 rpm.

Also, do you recommend one drive over another (sticking with FW800)? I just got the WD My Book Studio LX pretty much just because I can get it for half the price as retail.

I like WD. I avoided Seagate for a while because they had ... troubles. Then when things settled down, I bought one. It didn't work, I returned it and bought a WD. And then I bought another WD because my first one was full. The professionals like things like G Technology systems which use Hitachi drives. Whatever hard drive you get, it will die. One day, when it's most inconvenient, it will die. Prepare.

Last question, based on "numbers alone", if i add different effects, audio, and graphics to my FCP project, does this add anything to the data rate that the scratch disk has to handle?

Yes. Don't forget effects and graphics will have to be rendered to playback, one way (in reality, in your Render Files folder) or the other (on the fly by the CPU) and this, too, will have some effect on how things play back.

Personally, I'm not a big fan of the "get a FW drive for your scratch disk cuz it goes faster" mentality. I can get 5 streams of AIC out of my overcrowded internal drive on a 2.5 year old iMac. FW800 would not go any faster, and by having stuff on my internal drive, it gets backed up to my Time Machine drive (a lil' ol' USB drive. WD of course). If your workflow precludes backing up the contents of your FCE/P Documents folder to a TM drive (and it might. You may have, say, a RAID 1 external drive) then using an external drive for FCP scratch disk would be perfectly valid.

IMO, it's not until you get to eSATA and USB3 that you can truly justify ditching the internal drive on the basis of speed alone. Of course Apple could have gone with FW3200, that was approved nearly 3 years ago. Apple has an annoying habit of letting things slide (by comparison) then coming out with something super-duper (compared to what was on the Macs until that point, but might be just incrementally better that what has been creeping along on PCs over that time) and we all go wild. Sometimes, I'd like to see the things creeping along on the PCs on my Mac too. Oh, well, I've nailed my colours to the mast. All 6 of them

 

[musique]

RAID is not necessarily the same as a back-up

Just my two cents on "back-up" as it was used earlier:

A RAID device is a great tool to maintain a copy of things that are on your local machine. It may even, on a small, fast network, be capable of acting as a work drive (though I don't know if FCS will permit it).

However, a RAID per se is not the same as a back-up. Even with dual redundancy, it can fail because of a faulty fan/power unit, the power can go out for an extended period, you can have a fire, or some other unexpected disaster might occur. RAID devices should have their data backed up on a regular basis.

 

[martinX]

However, a RAID per se is not the same as a back-up.

And a RAID 0 isn't even anything like a backup. It's like parachuting without a spare chute. In your underpants.
I love this vid.

Photographer/Director Chase Jarvis shares his bombproof workflow and backup for every image he shoots, stills and video alike. This in-depth look includes all the steps from capture to archive and gives you a method to ensure that you'll never lose a single image.

 

[alph45]

Personally, I'm not a big fan of the "get a FW drive for your scratch disk cuz it goes faster" mentality. I can get 5 streams of AIC out of my overcrowded internal drive on a 2.5 year old iMac. FW800 would not go any faster, and by having stuff on my internal drive, it gets backed up to my Time Machine drive (a lil' ol' USB drive. WD of course). If your workflow precludes backing up the contents of your FCE/P Documents folder to a TM drive (and it might. You may have, say, a RAID 1 external drive) then using an external drive for FCP scratch disk would be perfectly valid.

- it really isn't about speed other than needing enough to accomplish the task.
- fragmenting your system drive with the constant read/writes in a busy editing environment isn't so hot. If your on a MP with multiple internal drives and one is set up for scratch that's fine. If your on the road with a MBP running a SSD not so good.
- having projects on portable or external drives is useful if you're 1. managing multiple projects 2. swapping computers or locations 3. like to store the source and project files offsite once a FC has been made (drop in safety deposit box and call it good).

 

[martinX]

- it really isn't about speed other than needing enough to accomplish the task.
- fragmenting your system drive with the constant read/writes in a busy editing environment isn't so hot. If your on a MP with multiple internal drives and one is set up for scratch that's fine. If your on the road with a MBP running a SSD not so good.
- having projects on portable or external drives is useful if you're 1. managing multiple projects 2. swapping computers or locations 3. like to store the source and project files offsite once a FC has been made (drop in safety deposit box and call it good).

All good points. Fragmenting can be overcome by periodically wiping and restoring from a TM backup ... which would go faster over FW800 (My first backups over USB2 take about 12 hours)

I'd add that having projects on external drives scares me unless there is lots of copies and I'm really organised. Which I'm not. What I love about my Mac (amongst other things) is Time Machine. With everything on my HDD, it's backed up without even thinking about it. Once, my whole iMac got stolen but not my Time Machine.

If I was as organised as Chase Jarvis (see link above), things would be handled differently.

 

[alph45]

The Chase Jarvis link was a good watch as well as a good example of professional workflow. I'm totally stealing the MBP in a briefcase with dual externals for backup. A big difference between them and the hobbyist is Liability, which almost forces this kind of workflow. I would bet that they have to "prove" data integrity just to get certain kinds of production insurance. Not that what they're doing isn't a good idea. Most people can accomplish a simplified version with a couple of externals dasychained.

When it comes to projects, the most important thing is to keep the original data safe (ie duplicated and put elsewhere). The actual project files that FCP generates can be stashed in multiple locations easily since they're tiny. I put them on my gmail account and on my website server. Then if disaster hits locally (computer / backup / etc) the project can always be recompiled from the project file and source footage.

 

[martinX]

I'm totally stealing the MBP in a briefcase with dual externals for backup.

It is such a cool look. Combine it with one of these, and you get to be Mac Daddy On The Set.