RAID Cards, Configurations, and the Mac Pro

[Laslo Panaflex]

Caldigit HD one

I don't know if this would suit your needs, but I recently purchased a CalDigit HD One for HD video work. I have absolutely zero issues with it. I have the 2TB system, but CalDigit says you can put any drives you want into the trays, and a sales rep says they even offer the manufacturer and model #'s of drives they "test and approve" for the enclosure, which is nice.

You can find more info about it here; http://www.caldigit.com/HDOne/ I have attached my speed results for the RAID as tested in Blackmagic's speed test app, pretty impressive.

 

[nanofrog]

I don't know if this would suit your needs, but I recently purchased a CalDigit HD One for HD video work. I have absolutely zero issues with it. I have the 2TB system, but CalDigit says you can put any drives you want into the trays, and a sales rep says they even offer the manufacturer and model #'s of drives they "test and approve" for the enclosure, which is nice.

You can find more info about it here; http://www.caldigit.com/HDOne/ I have attached my speed results for the RAID as tested in Blackmagic's speed test app, pretty impressive.

It's an External port card + external enclosure, so it would likely work in the '09 MP. Easier perhaps, as it's nearly a full ready made solution (just add drives).

Some may prefer to buy the separate pieces for specific parts, features, and performance (highly budget dependent here ) though. To each his/her own.

 

[scaley]

Yes, unless you have the inclination and tools to either make one, or to extend the power connections. If so, you can solder in some additional wire, allowing the MiniSAS end to extend farther. Otherwise, the Maxconnect device is the only one ready-made I'm aware of.

Damn. Period. Guess I'm going to have to eat this one.

Highpoint's cards can only handle firmware (OS booting) for one OS (either OS X or Windows/Linux). So if you want to boot OS X, it couldn't boot windows or linux. (Not sure if this is an issue for you).

Yeah I did see that it was bootable under OS X and I am a OS X only user so dual booting / bootcamp isn't an issue. Given that this is the case would you suggest as earlier about moving the whole OS onto the array? I know there are speed and redundancy gains to be expected but how does this work with upgrading the OS, etc?

If you have an old CD or DVD ROM drive lying about, just strip it down, and use one of the metal enclosure plates to make a mount. 2x 3.5" HDD's will fit on one. Easy, and far cheaper. Faster too, as you already have what you need.

Sounds like the route to attempt first and that will be easy. I hope that running the cabling works.


So back to the post about the new 2009 Mac Pros and Raid cards: Does that really mean that NO 3rd party raid cards are going to work? And that effectively limits your options to ONLY the 4 internal bays? Did they get rid of the miniSAS ipass cable then?

 

[nanofrog]

Yeah I did see that it was bootable under OS X and I am a OS X only user so dual booting / bootcamp isn't an issue. Given that this is the case would you suggest as earlier about moving the whole OS onto the array? I know there are speed and redundancy gains to be expected but how does this work with upgrading the OS, etc?

It may not be a bad idea.

You'd need to keep the OS drive in, and do the physical installation. Then copy/clone the OS, and reload to the array. Make sure you download and install the EFI firmware from Highpoint.

Sounds like the route to attempt first and that will be easy. I hope that running the cabling works.

You do realize, that the '09 MP won't allow you to use the HDD bays with a 3rd party card?

If you use an external enclosure, a 3rd party (internal or external port) model will work with it.

It's easiest to do this with an external card and an external cable. One end would look like this (SFF-8088/MiniSAS), while the other end could be the same, or Multilane. It depends on the enclosure used.

It can be done with an internal card as well, but requires a different cable end attached internally to the card, and an open PCIe slot cover to physically be passed through. Looks like this (SFF-8087). The SFF-8088 on both ends is harder to find, and is more expensive than the MultiLane variant IIRC. Ultimately, more expensive, a little more of a PITA, and can even cost you a needed PCIe slot.

To me, this route would only be done in cases the card is already owned (and used past any return period), or if a very specific need is present, that there's no external equivalent.

You can find these cables here.

So, you'd be better off exchanging the card you have for the external version, RocketRAID 4322.

So back to the post about the new 2009 Mac Pros and Raid cards: Does that really mean that NO 3rd party raid cards are going to work? And that effectively limits your options to ONLY the 4 internal bays? Did they get rid of the miniSAS ipass cable then?

Yes, they elliminated the MiniSAS/iPass cable, and soldered the HDD ports directly to the logic board. Data is transmitted via traces on the board itself.

In the case of the internal bays, the only way to have them in a RAID configuration, is to either use OS X's software RAID utility, or purchase Apple's RAID card. No 3rd party card can use them this time.

3rd party leaves you with an external solution for the most part. The only other alternative, is remove the optical drive, and use both bays to install an internal system, and locate a 5.25" external (single HDD) enclosure to hold the Optical drive, paying close attention to the interface that enclosure uses. Otherwise, you'd need some sort of adapter, such as FW400, or eSATA.

The new design definitely complicates matters, and makes hardware RAID more expensive.

Sorry for the bad news , but it's better to discover it now while a return is possible.

 

[scaley]

Nanofrog,

thanks for the detailed post and the links to all the cabling. I was definitely unclear about what happened with the new 2009 Mac Pro models and it certainly seems like the internal option for the 2009 models is much more complicated and an external route would be advisable. But I won't have to worry about this since I purchased the 2008 2x2.8 version.

So what have have now just for clarity is:

2008 Mac Pro, 2x2.8 Ghz
6 gigs Memory
6, 1 TB Seagate Barracuda ES.2 drives
Rocket Raid 4320

I was planning on keeping the raid internal and hence me needing to get the dumb $90 cable (thanks for that).

It is interesting to hear that you also suggest it might not hurt to put the OS on the array since I thought I'd seen various people saying it was a bad idea. Now I'm confused. But just to clarify is the following procedure correct if I wanted to do that?

1.) Clone my OS to another external disk via carbon copy cloner
2.) Connect up the entire 6 drive array in my Mac Pro (I have the 4320 which will be connected to the 4 bays + I will have a miniSAS to 4x sata cable to connect to the 2 drives in the 2nd optical bay.
3.) Clone back to the Array and install the EFI update
4.) Select the array as the boot volume.

Seem right? I was heeding the warning off of Barefeats.com that says: "Having done that, we caution against using your RAID set for booting. If you are working with HD Uncompressed video, we recommend booting from a separate volume and dedicating the RAID set for such dedicated actions such as capture and playback."

What is the issue with booting from the array?
How does upgrading Mac OS X work if the cloned OS is then on the array when say, Snow leopard comes out?

Sorry about all the questions... just trying to make sure not to botch this all up. Information is good right?

 

[nanofrog]

Nanofrog,

thanks for the detailed post and the links to all the cabling. I was definitely unclear about what happened with the new 2009 Mac Pro models and it certainly seems like the internal option for the 2009 models is much more complicated and an external route would be advisable. But I won't have to worry about this since I purchased the 2008 2x2.8 version.

So what have have now just for clarity is:

2008 Mac Pro, 2x2.8 Ghz
6 gigs Memory
6, 1 TB Seagate Barracuda ES.2 drives
Rocket Raid 4320

I was planning on keeping the raid internal and hence me needing to get the dumb $90 cable (thanks for that).

It is interesting to hear that you also suggest it might not hurt to put the OS on the array since I thought I'd seen various people saying it was a bad idea. Now I'm confused. But just to clarify is the following procedure correct if I wanted to do that?

1.) Clone my OS to another external disk via carbon copy cloner
2.) Connect up the entire 6 drive array in my Mac Pro (I have the 4320 which will be connected to the 4 bays + I will have a miniSAS to 4x sata cable to connect to the 2 drives in the 2nd optical bay.
3.) Clone back to the Array and install the EFI update
4.) Select the array as the boot volume.

Seem right? I was heeding the warning off of Barefeats.com that says: "Having done that, we caution against using your RAID set for booting. If you are working with HD Uncompressed video, we recommend booting from a separate volume and dedicating the RAID set for such dedicated actions such as capture and playback."

What is the issue with booting from the array?
How does upgrading Mac OS X work if the cloned OS is then on the array when say, Snow leopard comes out?

Sorry about all the questions... just trying to make sure not to botch this all up. Information is good right?

In your case, the '08 model is much easier to work with, particularly as the internal HDD bays can be used.

Your basics seem good, just read the documentation first, as the order might be different.

1. Install physical components, including drivers
2. Make a clone of the OS, as you need the drivers on it.
3. Create the array
4. Set it as the boot drive
5. Install the clone onto the array

The order of 2-4 isn't quite so critical, so long as they all are done prior to step 5.

If you have access, an external HDD via USB/FW400/800... would be handy.

You could always use the original HDD as well, but leaving it unmounted and locating power may be more of a pain, depending on you're take for this method.

 

[scaley]

In your case, the '08 model is much easier to work with, particularly as the internal HDD bays can be used.

Your basics seem good, just read the documentation first, as the order might be different.

1. Install physical components, including drivers
2. Make a clone of the OS, as you need the drivers on it.
3. Create the array
4. Set it as the boot drive
5. Install the clone onto the array

The order of 2-4 isn't quite so critical, so long as they all are done prior to step 5.

If you have access, an external HDD via USB/FW400/800... would be handy.

You could always use the original HDD as well, but leaving it unmounted and locating power may be more of a pain, depending on you're take for this method.

Excellent, now I guess my question still remains: How will that work when an OS upgrade is released? Since the OS is already bootable and installed on the array will the upgrade (when snow leopard comes out) already recognize it as a partition? Or is it gonna toast the whole mess and hence everyone suggesting NOT putting the system on the Array?

For that matter and having not done this, what do people normally do? Make one large journaled partition? 3 gigs in my case with either RAID 5+HS or RAID 6? Advice on that?

Thanks again, this has been extremely informative!!!

 

[nanofrog]

Excellent, now I guess my question still remains: How will that work when an OS upgrade is released? Since the OS is already bootable and installed on the array will the upgrade (when snow leopard comes out) already recognize it as a partition? Or is it gonna toast the whole mess and hence everyone suggesting NOT putting the system on the Array?

For that matter and having not done this, what do people normally do? Make one large journaled partition? 3 gigs in my case with either RAID 5+HS or RAID 6? Advice on that?

Thanks again, this has been extremely informative!!!

As for the eventual release of SL, I'm not sure if it would destroy the data or not.

I wouldn't think so, but in any case, its HIGHLY recommended that you make fresh backups just prior to any OS installation, as things can go wrong. Better to loose time only, than time and the data. If you haven't realized, data recovery services are VERY expensive.

As for not placing the OS on the array, it's likely more of convienience for their way of thinking, or to help split the load. Slow OS as it's on a single drive, while only using the array for data only, or data + applications.

It's up to you, but if the array has the bandwidth, I'd put the OS on it. You can benifit from the redundancy at least. It likely won't boot any faster however, as additional time is required for the card to initialize first, and then load the OS. So boot times can actually increase. (Perhaps another reason they leave it off--Impatience.)

As for 5+HS or 6, I'm not sure what your priorities are. If you value speed most, 5 is faster, though adding an HS drive will slow it down a little. RAID 6 trades a little speed for additional fault tolerance. If you're paranoid about data, this may be the best way to go.

Speed differences between the two, may be quite small. Once the array has been created, it's best to test it out. Power failures, etc. to see how the card behaves BEFORE you trust your data to it. Quite a learning expereince, and really needs to be done with every single RAID card you ever buy, as they aren't necessarily the same. (Even small quirks can make a BIG difference). So while testing, create a 5+HS and test the speed, then again in 6, testing it's speed. Then make up your mind on what's the best balance to fit you're needs.

Sorry if it's not quite so direct, but not everyone has the same needs, nor have the exact same opinions on how to balance capacity/speed/redundancy. Everyone, and situation is different.

As you plan to use enterprise drives, you'll have a little more stability and reliability just from the drives. HS drives are handy to have installed if you're away from the system for long periods of time, no matter if you run 5 or 6. If you'll be infront of the machine daily, and for long periods of time, you might be able to forgo it, as the HDD space is already limited (otherwise I'd be more inclined to say use it anyway, just in case it poops out while you're in the bathroom ).

You also want to keep a spare drive handy, and be sure to implement a good backup system. Both can really save your hide when the **** hits the fan.

 

[scaley]

Nanofrog, That is EXACTLY what I needed to know. I'm feeling like I have a pretty good hold on this whole thing after all these posts. Though I'm sure that could all go out with the bathwater once I'm getting the thing up and ready. Thanks (to everyone)!

So if people can tolerate it I'll keep asking questions:

You also want to keep a spare drive handy, and be sure to implement a good backup system. Both can really save your hide when the **** hits the fan.

So, in practice what is the best way to back up a 4 Terrabyte raid? Short of another RAID array? I know tape backups were an option but what is a fairly inexpensive way to do this? I'm a poor researcher and don't have infinite money...for that matter - don't have too much at all! I'd welcome suggestions on that too.

 

[nanofrog]

So, in practice what is the best way to back up a 4 Terrabyte raid? Short of another RAID array? I know tape backups were an option but what is a fairly inexpensive way to do this? I'm a poor researcher and don't have infinite money...for that matter - don't have too much at all! I'd welcome suggestions on that too.

Yes, another RAID is the best option.

But it doesn't have to be fast. Look into something like NAS,iSCSI, or AoE based gear.
(Don't worry, a search engine will find plenty).

Cost effective, as you won't need special cables, etc. Just the unit, and some Ethernet cables to attach to your switch/router. Not expensive if you need one.

In this instance, as it's not "On"/being used constantly, you can get away with consumer drives to save on cost.

 

[ncc1701d]

Yes, another RAID is the best option.

But it doesn't have to be fast. Look into something like NAS,iSCSI, or AoE based gear.
(Don't worry, a search engine will find plenty).

Cost effective, as you won't need special cables, etc. Just the unit, and some Ethernet cables to attach to your switch/router. Not expensive if you need one.

In this instance, as it's not "On"/being used constantly, you can get away with consumer drives to save on cost.

Drobo? From the research I've done it seems pretty easy. However, it looks as though if you had 4 TB to back up (Mac Pro with 4 x 1TB) you'd need to put in at least 3 x 2TB drives (assuming your 4TB on th epro were full). If they're not full, you can buy enough drives as required.

 

[scaley]

Damn! I hate being broke at times like these because sounds like I need to afford more drives to be sure I have a backup. I wish there was a cheaper solution (how many times do people say that) - like a multi-terabyte tape backup or something but those are pricy too!

 

[Virtuoso]

Drobo? More like Slowbo! It only manages 16MB/s, which means it would take 70 hours to backup a 4TB RAID.

Personally I use a Windows Home Server with a 500GB boot drive and 6x 1TB storage drives. Data transfer performance is around 90-100MB/s which is pretty good for a gigabit network.

 

[nanofrog]

Drobo? More like Slowbo! It only manages 16MB/s, which means it would take 70 hours to backup a 4TB RAID.

Personally I use a Windows Home Server with a 500GB boot drive and 6x 1TB storage drives. Data transfer performance is around 90-100MB/s which is pretty good for a gigabit network.

As you've pointed out, there's other options besides Drobo. Though speed isn't the absolute primary concern, 70 hours is unacceptable.

Thecus might be another model to look at. (I haven't put much into looking at it though).

Personally, I just use a card with enough ports to have a separate array of the same size or larger, and viola. Load the Backup software, set it, and hopefully forget it.

 

[ncc1701d]

Drobo? More like Slowbo! It only manages 16MB/s, which means it would take 70 hours to backup a 4TB RAID.

Personally I use a Windows Home Server with a 500GB boot drive and 6x 1TB storage drives. Data transfer performance is around 90-100MB/s which is pretty good for a gigabit network.

Yes... That does sound like a lot of time While the initial back up (for me at least) may be a while, it's more like 1TB to start with. From then on, it would be just normal additions of a rip here and there etc. Using "Time Machine", I was thinking it wouldn't be too long to let it do it's thing. Only over a long time would it be a full 4TB and it woudl be incremental. Any thoughts?

 

[scaley]

So since it seems like backing up an internal array to a secondary external array seems to be the preferred route, and since I'll be needing to clone my OS disk to copy it back to the array, I am thinking that it makes a lot more sense to just pick up another drive or two and slap them in the external enclosure that I have now but was trying to replace with the internal array. It all comes full circle!

 

[rylin]

Thecus might be another model to look at. (I haven't put much into looking at it though).

I've looked at Thecus, and would probably go with QNAP if I wanted a ready-made NAS.
Instead, for $500 you get an infinitely better machine that holds plenty of disk and can still do AoE, iSCSI, NFS, SMB etc.

It does require a bit of know-how though, but it's cheaper than a drobo, cheaper than a QNAP etc. and it will cream them all when it comes to performance.

 

[nanofrog]

I've looked at Thecus, and would probably go with QNAP if I wanted a ready-made NAS.
Instead, for $500 you get an infinitely better machine that holds plenty of disk and can still do AoE, iSCSI, NFS, SMB etc.

It does require a bit of know-how though, but it's cheaper than a drobo, cheaper than a QNAP etc. and it will cream them all when it comes to performance.

I hadn't even considered the skill level needed to use it. I thinking in terms of low cost, good performance, and would meet his needs.

 

[rylin]

I hadn't even considered the skill level needed to use it. I thinking in terms of low cost, good performance, and would meet his needs.

Re-reading what I wrote, it probably wasn't too clear I advocated a DIY storage box.
$500 will get you an amazing machine which you can fill with disks, but then the problem becomes management, which can be tricky.

For anyone looking for an archival system to use at home or in a SMB, the DIY is well worth it, even if it means having someone else do the initial setup.

I'm working on a few things related to making this easy.
I just wish there was a mfg out there providing good storage-server enclosures.

I mean come on, is 8+ 5.25" slots and a well-engineered layout too much to ask?!

 

[nanofrog]

Re-reading what I wrote, it probably wasn't too clear I advocated a DIY storage box.
$500 will get you an amazing machine which you can fill with disks, but then the problem becomes management, which can be tricky.

For anyone looking for an archival system to use at home or in a SMB, the DIY is well worth it, even if it means having someone else do the initial setup.

I'm working on a few things related to making this easy.
I just wish there was a mfg out there providing good storage-server enclosures.

I mean come on, is 8+ 5.25" slots and a well-engineered layout too much to ask?!

I understand exactly where you're coming from.

I'm a big proponent of DIY. You get exactly what you want/need, and can usually do it for less $$$ as well. I see it as the ideal method for individuals, SMB's, or any other company on a very tight budget. My guess, this might become more common.

Enclosures do tend to be expensive for anything I'd like to have (decent appearance, very well laid out, and 8+ drives) in a non-rackmount design.
(Say $ 550-600 for an Enhance E8MS, s/h not included just for an enclosure). Yikes!

Some good rackmount cases as well, but finding space for them can be a pain. Not to mention too industrial (ugly) in appearance for a home. Then add loud, as quiet isn't part of the design criteria.

So I try to get a case large enough to contain 2x what's needed, and make a duplicate array for backup purposes. Not separate, but tends to be cost effective, as it's essentially just the additional drives and the necessary cabling, once the basic system is built. (I usually use 12 - 16 port cards anyway, just to find all the features I'm after).

 

[scaley]

Rylin and Nanofrog,

All of the options that you posted seem like great routes to go and I do really like the look and simplicity of the Enhance or QNAP systems. As you know by now I purchased the RocketRaid 4320 (internal ports) to do the 6 drive configuration that I wanted to do internally.

But, to back that array up with something like the Enhance cases would it be advisable to get a 2nd raid card that has external ports or can you run it using an internal to external connector. It seems like one benefit of the NAS route would be to have that thing hiding in a closet somewhere attached to the network but I wouldn't have to see it or hear it. The QNAP devices look pretty dang sick I have to say.

Think I'm going to have to wait until I pay off the internal array first but will definitely go one of these routes. Meanwhile I just hope that the redundancy of RAID 6 will serve me fine.

ADDITION:

I was thinking about the prior discussion too about putting the OS, home directories etc all onto the internal 6 drive array and then thought I'd ask one question again:

If I go the route of moving the OS onto the array as well do people usually put this all onto ONE partition (in my case it would be 4 gig in RAID 6)? Or is it better to partition the array into a system partition, a user partition, etc? I would think a single partition no?

 

[nanofrog]

Rylin and Nanofrog,

All of the options that you posted seem like great routes to go and I do really like the look and simplicity of the Enhance or QNAP systems. As you know by now I purchased the RocketRaid 4320 (internal ports) to do the 6 drive configuration that I wanted to do internally.

But, to back that array up with something like the Enhance cases would it be advisable to get a 2nd raid card that has external ports or can you run it using an internal to external connector. It seems like one benefit of the NAS route would be to have that thing hiding in a closet somewhere attached to the network but I wouldn't have to see it or hear it. The QNAP devices look pretty dang sick I have to say.

Think I'm going to have to wait until I pay off the internal array first but will definitely go one of these routes. Meanwhile I just hope that the redundancy of RAID 6 will serve me fine.

ADDITION:

I was thinking about the prior discussion too about putting the OS, home directories etc all onto the internal 6 drive array and then thought I'd ask one question again:

If I go the route of moving the OS onto the array as well do people usually put this all onto ONE partition (in my case it would be 4 gig in RAID 6)? Or is it better to partition the array into a system partition, a user partition, etc? I would think a single partition no?

Check the $$$ numbers to help decide between something like a QNAP/Thecus (or similar product), or Enhance enclosure + 2nd RAID card, as the addtional card and enclosure cost may be the more expensive method in your case.

If you're set on the Enhance/external enclosure, you might be better off exchanging your RocketRAID for a different model, containing additional ports. Might be cheaper in the long run, but check the numbers first to be sure.

Your idea of two different cards is easy enough, but it burns up an additional slot, and the arrays would need to kept separate. (Don't mix the internal and external drives in an array, no matter the gear used). Carries additional risk, and isn't advisable for someone not already very familiar with RAID, unless you have the time to experiment and truly learn what those risks are first hand.

I really don't recommend running on zero backups. Just too dangerous, and the added difficulty in testing the array prior to adding data, can be a problem. (Testing is far more likely to be skipped this way, and if something goes wrong, you may not understand the card's behavior, and be able to recover without that experience).

Get a cheap single ~1TB drive and use that as a bare minimum, as optical isn't practical. Cheapest thing I can think of (data not hitting the array's capacity limit in the begining), and is certainly better than nothing. Then upgrade to a better method as soon as you can.

 

[scaley]

I really don't recommend running on zero backups. Just too dangerous, and the added difficulty in testing the array prior to adding data, can be a problem. (Testing is far more likely to be skipped this way, and if something goes wrong, you may not understand the card's behavior, and be able to recover without that experience).

Well, it seems like this is ever the tough decision. Internal RAID allows for a lot of the performance and space requirements I need, and while I am at my computer daily, if RAID 5+HS/6 don't provide enough fault tolerance for considering this also a good enough backup solution I'm going to have to think about this. And I say that knowing that RAID isn't a backup but if sufficient redundancy is built in with careful system administration then you can have confidence no?

Since you have a LOT more experience here. If you were setting this up with wanting performance and the safety of your data how would you do this with six 1 Tb drives, the internal card I have, and a need for backup considering I already have about 1.5 Terabytes of data that will need to go ON to the array?

I have thought of a few options:

Options with my current Rocket Raid Card 4320 (8 internal drives supported)
1.) 6 disc (1 Tb Ea), RAID 6 = 4 Tb, 2 drives failure tolerance
2.) 6 disc (1 Tb Ea), RAID 5 + HS = 4 Tb, 1 drive failure with auto rebuild
3.) 6 disc (1 Tb Ea), RAID 50 = 4 Tb, 2-3 drive failures?
4.) 4 disc (1 Tb Ea), RAID 5 (no HS) = 3 Tb as system disc on one channel of the RAID card backed up by 2 disc (1.5 Tb Ea), RAID 0 = 3 Tb on 2nd 4 port channel but the fact that it is RAID 0 wouldn't matter too much since it would just be a backup.
5.) 6 disc (1 Tb Ea), RAID 10 = 3 Tb, 3 drive failure (seems unlikely) GOOD ENOUGH? 3 Tb would definitely suffice for me.

I definitely see the need for an external backup but if I move a bunch of things external then I lose the benefits of the performance increases for the OS and would have to replace my 4320 card. Damn this is confusing, thanks for the patience, and I would really welcome configuration suggestions.

 

[rylin]

5.) 6 disc (1 Tb Ea), RAID 10 = 3 Tb, 3 drive failure (seems unlikely) GOOD ENOUGH? 3 Tb would definitely suffice for me.

I definitely see the need for an external backup but if I move a bunch of things external then I lose the benefits of the performance increases for the OS and would have to replace my 4320 card. Damn this is confusing, thanks for the patience, and I would really welcome configuration suggestions.

Just a quick note as I'm in the process of trying to wake up (i.e., can't think straight).

A 6 disk R10 would let you lose 1 - 3 drives.
The R1s are pairs (d0, d1. d2, d3. d4, d5), and are striped together.

If a disk from one of the mirrors die, you're fine.
You can in fact lose a disk in each of the mirrors and be fine.

If, on the other hand, you lose d0 and d1, the array is shot.

Personally, I dislike using external cards (but as I've said, I'm a DIYer).
Putting 8+ disks in a case isn't a problem, nor is hooking 'em up cheaply (I've started seeing mobos with 12 SATA ports).

At this point, AoE or iSCSI are terrific options -- disk traffic over the network.
Pretty much everyone would get a few hundred megabit per second straight off (100Mbps = 12.5MB/sec). iSCSI would have a well-known limit around 700-800Mbps, and AoE should be able to go a bit higher.

That's the low-cost option.
If you want more speed, you can get a bunch of extra network cards for the MP and the storage box and do "bonding". This mens that you effectively get a few gigabit links between you and your storage, often crushing the performance given by the big vendors.

If you have plenty of disk with fast caches, using 10Gbps cards would be an option.

Anyway, I'm tired enough that I've probably strayed more off topic now , but I'll make sure to re-read the thread during the day and see if I can come up with something.

 

[barefeats]

Rocketraid 3520, Rocketraid 4320, and the Caldigit raid card

Can anyone give me some testimonials about the relative performance of these cards? Or for that matter how much better internal raid will be relative to using the external eSATA RAID box I have now (which btw doesn't do anything other than RAIDs 0,1,10)

The RR3520 (IOP341) and RR4320 (IOP348) have a more powerful processor than the CalDigit (IOP333). That means CalDigit is limited to 500MB/s while both RocketRAIDs have tested over 1000MB/s in our lab.

Are you doing internal drives? There are also versions of the RocketRAID that have external mini-SAS ports that can connect to enclosures with either SAS or SATA drives (RR3522 and RR4322).

I found that the RocketRAID cards are happiest with either Seagate or Hitachi drives (both SAS and SATA).