Are striped disks overkill for me?

[Avery1]

I'm totally new to high-definition video (so go easy on me!)... just been taking little HD mini-movies on my cameras (some will shoot AVCHD-lite). However, I'd like to get into it more, and will likely buy a HD videocamera or one of the Canon SLRs that is HD capable. At best, I'll probably create a few short (< 15 minutes) amateur HD videos each month, and some longer ones a few times each year. This will be with Premiere Pro.

My current hardware is a 2009 Mac Pro with 2 quad-core 2.66 GHz CPUs, 8GB RAM, and the stock internal 640GB hard drive. Snow Leopard.

I am going to use an Intel 80GB SSD for the boot drive, and will have either:
a) two data/scratch disks in software RAID1
or
b) 4 data/scratch disks in software RAID10 (I believe this is possible with 10.6, but have read conflicting comments)

The disks are pretty fast -- Samsung FD HD103SJ 1TB 7200rpm disks.

I already have all 4 disks, but am considering option a) because:
a) power savings (about 15 watts... the machine already consumes 140 watts when idle)
b) immediate cost savings of about $170
c) decreased complexity of software raid solution / less likely for 'something' to go wrong immediately or after an OS patch/upgrade.
d) allows me to mount the 640GB internally as a backup drive

- If I go with 2 disks in RAID1 (redundant drives is a requirement for me) and the SSD boot disk, then is it likely I will see any performance issues?
- Under what scenarios would having the RAID10 setup show noticeable benefit?
- Any issues with having a 'single' RAID0 or RAID10 volume that is used for both video and all of my other data?
- Other thoughts?

 

[sl1200mk2]

I'm totally new to high-definition video (so go easy on me!)... just been taking little HD mini-movies on my cameras (some will shoot AVCHD-lite). However, I'd like to get into it more, and will likely buy a HD videocamera or one of the Canon SLRs that is HD capable. At best, I'll probably create a few short (< 15 minutes) amateur HD videos each month, and some longer ones a few times each year. This will be with Premiere Pro.

My current hardware is a 2009 Mac Pro with 2 quad-core 2.66 GHz CPUs, 8GB RAM, and the stock internal 640GB hard drive. Snow Leopard.

I am going to use an Intel 80GB SSD for the boot drive, and will have either:
a) two data/scratch disks in software RAID1
or
b) 4 data/scratch disks in software RAID10 (I believe this is possible with 10.6, but have read conflicting comments)

The disks are pretty fast -- Samsung FD HD103SJ 1TB 7200rpm disks.

I already have all 4 disks, but am considering option a) because:
a) power savings (about 15 watts... the machine already consumes 140 watts when idle)
b) immediate cost savings of about $170
c) decreased complexity of software raid solution / less likely for 'something' to go wrong immediately or after an OS patch/upgrade.
d) allows me to mount the 640GB internally as a backup drive

- If I go with 2 disks in RAID1 (redundant drives is a requirement for me) and the SSD boot disk, then is it likely I will see any performance issues?
- Under what scenarios would having the RAID10 setup show noticeable benefit?
- Any issues with having a 'single' RAID0 or RAID10 volume that is used for both video and all of my other data?
- Other thoughts?

I'm not a big video guy (tinkering amateur), but I do work with a lot of high end SAN storage arrays (RAID-10, 6 and 50) with anything from 16 to 48 drive setups. I think it really depends on how many streams of HD footage you plan on working with. You should benchmark your current workflow and see where if any potential bottlenecks are. I'd say RAID-10 is probably going to be overkill for you. You can always start with a single drive setup (in addition to your SSD boot) and progress from there as needed. RAID-1 will get you little performance difference other than data integrity. RAID-0 on the scratch would be just as fast as 10 since you're only spanning the data on two spindles (the other is a mirror) and much less expensive while protecting the boot device.

I would further say that (IMHO) software based RAID-10 isn't pointless, but if you're going to go that far it's a bad idea without using a full hardware based card. While 0 or 10 doesn't involve a lot of parity calculations you benefit greatly from the dedicated cache and write-through abilities. The fact that it's battery backed in many cases is very nice as well since software raid can be particularly vulnerable to corruption in sudden power loss when any sort of striping or parity is involved.

My personal setup (hopefully within the next year) will consist of an SSD boot, 10K scratch disk and 2TB storage / backup. I can always add another 10K drive in RAID-0, but doubt that would ever be needed (for me anyway).

Good luck -

 

[Avery1]

I'm not a big video guy (tinkering amateur), but I do work with a lot of high end SAN storage arrays (RAID-10, 6 and 50) with anything from 16 to 48 drive setups. I think it really depends on how many streams of HD footage you plan on working with. You should benchmark your current workflow and see where if any potential bottlenecks are. I'd say RAID-10 is probably going to be overkill for you. You can always start with a single drive setup (in addition to your SSD boot) and progress from there as needed. RAID-1 will get you little performance difference other than data integrity. RAID-0 on the scratch would be just as fast as 10 since you're only spanning the data on two spindles (the other is a mirror) and much less expensive while protecting the boot device.

I would further say that (IMHO) software based RAID-10 isn't pointless, but if you're going to go that far it's a bad idea without using a full hardware based card. While 0 or 10 doesn't involve a lot of parity calculations you benefit greatly from the dedicated cache and write-through abilities. The fact that it's battery backed in many cases is very nice as well since software raid can be particularly vulnerable to corruption in sudden power loss when any sort of striping or parity is involved.

My personal setup (hopefully within the next year) will consist of an SSD boot, 10K scratch disk and 2TB storage / backup. I can always add another 10K drive in RAID-0, but doubt that would ever be needed (for me anyway).

Good luck -

Thanks for the feedback. I sense a RAID10 setup is going to be overkill too, especially given I have the SSD for the boot drive. However, I just haven't done much video yet -- just a few short HD things on my 13" MBP -- which definitely slowed to a crawl. Based on a few comments I've read, coupled with the fact there are some folks editing HD on an external firewire... I'm guessing I will be find with an internal RAID1 setup.

It seems a regular occurrence on MR that folks are pointing out that RAID1 is not a backup. However, in my eyes -- it IS half of the backup solution to handle hardware failure.... and *for me*, hard drives failure is the riskiest part of ensuring data is there (and my non-media data will be there, too). If you work on SANS, I'm sure you see enterprise disks fail all the time. Sure, I'll have conventional backups, too.

CS5 won't arrive for another week or two (gotta love student pricing), so I can't really do any benchmarking yet And, I may be using different formats for raw footage, going forward. That IS the right approach, though.

Point taken on going to 10 without a hardware solution. Definitely not going there right now. Hopefully power loss won't be a regular issue with a good UPS, but yes -- still possible.

As to migrating from no-raid to RAID __ (fill in the blank) or from RAID __ to RAID __, I was under the impression that MacOS required you to completely wipe the drives for that activity???

I'm hoping that someone who has an octo with a SSD and a single (non-RAID) scratch disk can suggest if they have issues editing HD with no more than a couple streams.

 

[matteusclement]

Not the same but it is kinda close

Here's my set up:
quad core 2.83ghz with 8gigs ram and 4 x 500gig 7200RPM discs in RAID 0.

My bottleneck?
Still my CPU's

Always will be.

Also, you didn't mention which video software you would use. FCP i take it?

I used to edit HD on my imac 21" and it was pretty painful compared to now with the quad core. Then with the RAID 0, it's smooth sailing. The longest wait times now are in Color.

that help?

 

[Avery1]

Thanks.

That is another helpful datapoint, and kind of confirms my suspicion about the 'heavier on CPU side of things'.

I'll be using Premiere Pro.

 

[sl1200mk2]

Sorry, I don't have experience yet with software RAID on OS X, but generally it's not possible to migrate from different RAID levels or from no RAID to an array setup without wiping the disks and starting fresh. Some cards and storage devices support this, but it's usually an enterprise level feature not available in software or at the consumer/prosumer markets.

I would always advise you that RAID is never a substitute for a solid backup strategy. Although the data is mirrored, it's still at heart a single point of failure. Bad data can get written to both drives in milliseconds. My own backup strategy involves copies of data at home (a combination of online, offline USB drives and DVD's for some data), an offsite USB drive that gets rotated between home and work and then all data is backed up to 'cloud' service.

I work with hundreds of servers and see people lose data all the time. Having lost some very valuable things myself to sheer stupidity I'm very vigilant now that so much of my life and memories are all digital now.

 

[Avery1]

Sorry, I don't have experience yet with software RAID on OS X, but generally it's not possible to migrate from different RAID levels or from no RAID to an array setup without wiping the disks and starting fresh. Some cards and storage devices support this, but it's usually an enterprise level feature not available in software or at the consumer/prosumer markets.

I would always advise you that RAID is never a substitute for a solid backup strategy. Although the data is mirrored, it's still at heart a single point of failure. Bad data can get written to both drives in milliseconds. My own backup strategy involves copies of data at home (a combination of online, offline USB drives and DVD's for some data), an offsite USB drive that gets rotated between home and work and then all data is backed up to 'cloud' service.

I work with hundreds of servers and see people lose data all the time. Having lost some very valuable things myself to sheer stupidity I'm very vigilant now that so much of my life and memories are all digital now.

Yep, totally agree.

For ME, RAID1 is a necessary *part* of making the overall backup solution robust! I'll explain... My 'backups' become my second line-of defense. So many things can go wrong with backups and restores, that I'd rather not rely on them for hardware failures (which seem all-too common), not to mention the PITA factor. A second disk is cheap

Along the same lines, I think it is pretty risky for people to put anything they actually care about on RAID0, unless they doing at least a regular and *thorough* test of their backups (I'm not quite that diligent). If you are RAID0, hands down, you just know that at some point you will *have to be* entirely reliant on your backups. The first thought that crosses anyone's mind who works in the industry, when they realize they have to pull from a backup for a production system, is "I hope the backup is good". The second thought is: "I wonder how recent we can go"

Sadly, I can't tell you how many times I've had a DBA come back and say "we don't have the backup we thought we had" (!)... bad day, bad day.

Cheers

 

[sl1200mk2]

Yep, totally agree.

For ME, RAID1 is a necessary *part* of making the overall backup solution robust! I'll explain... My 'backups' become my second line-of defense. So many things can go wrong with backups and restores, that I'd rather not rely on them for hardware failures (which seem all-too common), not to mention the PITA factor. A second disk is cheap

Along the same lines, I think it is pretty risky for people to put anything they actually care about on RAID0, unless they doing at least a regular and *thorough* test of their backups (I'm not quite that diligent). If you are RAID0, hands down, you just know that at some point you will *have to be* entirely reliant on your backups. The first thought that crosses anyone's mind who works in the industry, when they realize they have to pull from a backup for a production system, is "I hope the backup is good". The second thought is: "I wonder how recent we can go"

Sadly, I can't tell you how many times I've had a DBA come back and say "we don't have the backup we thought we had" (!)... bad day, bad day.

Cheers

You understand and get it. There is this saying with backup: "Backups are nothing, restores are everything". My goal is to never have to restore either, but I do test the ability to do so. I use Chronosync to keep all the versions in order.

It more often than not that you'll see someone think they can stop backing up because they have RAID. They eventually learn some very hard and painful lessons.

Good luck on your setup -

 

[Avery1]

Good luck on your setup -

Thanks... installing the SSD and OS right now! This should be good.

You are very right, though - I think backup approach and implementation is misunderstood by many. It's more complicated than meets the eye.

One of my favorite questions to ask when someone doesn't want to approve money, allocate enough storage for backups, or spend the time testing restores: "How much data are you okay losing?" The response is pretty universal

Okay, I'll cease the thread-drift now...