RAID Cards, Configurations, and the Mac Pro

Discussion in 'Mac Pro' started by scaley, Mar 10, 2009.

  1. scaley macrumors member

    Joined:
    Mar 3, 2009
    #1
    Hello everyone, I have read through the posts on this forum numerous times about RAID and RAID configurations in my search for the perfect RAID solution for my setup and I'm looking for some more advice and experience from people who have used the various cards.

    So, what I currently have is a Mac Pro 2x2.8 Ghz, 6gigs memory, with a 1 Tb drive internal and a second 1.5 Tb drive backing that up with time machine. Attached to the mac pro via an eSATA card I am running 4, 1Tb drives operating in a RAID 1+0 configuration in a Venus T5 enclosure with a Sil 4726 hardware raid card inside it.

    I have a few (albeit possibly lame) gripes about this configuration. For one, the Venus Case is a loud bugger and I am not to found of large fan noises. Second I have over 6 Tb of purchased hard drive space and was looking to migrate this to a RAID 5 configuration. Currently all my drives are SATA drives and I don't think I will be affording SAS ones anytime soon.

    I don't know what the performance gain is, but I've been looking at putting 2 drives in the 2nd optical bay, getting rid of the Venus and migrating to an internal storage solution. I would have a separate system disk and then I will just run the other 5 drives in a RAID 5 config. After looking at all of the card options I've sort of settled on a few that are more or less in my price range:

    Rocketraid 3520, Rocketraid 4320, and the Caldigit raid card

    Can anyone give me some testimonials about the relative performance of these cards? Or for that matter how much better internal raid will be relative to using the external eSATA RAID box I have now (which btw doesn't do anything other than RAIDs 0,1,10)

    Thanks.
     
  2. bigbossbmb macrumors 68000

    bigbossbmb

    Joined:
    Jul 1, 2004
    Location:
    Pasadena/Hollywood
    #2
    You can't run 5 internal drives on the Caldigit card, only 4. Even if you are using the 2 ODD SATA ports on the motherboard (these aren't seen by the card).

    So you could run 4 internal drives in RAID5 (4.5TB using 4x1.5TB) along with an additional external RAID with the HDElement (it is not recommended to use internal and external in the same RAID).

    You could do 2x HDElements in the same RAID (total of 8 drives).

    Other RAID cards have mini-SAS as well and can support 5-8bay enclosures with RAID5. You could look into these as well.
     
  3. rylin macrumors 6502

    Joined:
    Aug 18, 2006
    #3
    RAID-5 on several large disks = bad idea.
    Odds are you'll end up with a broken array if you have to rebuild.

    Don't do it, cap'n!
     
  4. scaley thread starter macrumors member

    Joined:
    Mar 3, 2009
    #4
    Yeah sorry I should have mentioned that I was aware of that and in the event that I went with the Caldigit card I would just use the two extra ports off of the motherboard and run it as a 4 drive raid. The Caldigit card has the advantage of being full length and not having to use connectors, however I am discouraged by the proprietary external ports and the limitation to 4 internal drives.

    Given the price of HDElements I don't think I'll ever use their enclosures and it is actually part of the reason that I am discouraged by the Caldigit card. However, it seems to get good reviews.

    Yeah, so I've done a lot of research on this and that is where I'm leaning. Again the Rocketraid cards seem like very solid options and at a reasonable price. I know that in order to connect to them you need to use a product such as the MaxConnect and so I'm curious if anyone has experience with these cards. In particular, what performance difference I could expect from the PCIe-4x vs. 8x and more importantly if it is going to be substantially faster that the external Sil 4726 Box that my drives are in now. It seems like it should be.

    I'm curious why you say this as it seems like many people have successfully build RAID 5 arrays that give both good throughput and redundancy (particularly with the use of a hot spare) and it seems like a good solution. Are you speaking from experience with the configuration software of one of the Rocketraid cards or the Caldigit?
     
  5. belvdr macrumors 603

    Joined:
    Aug 15, 2005
    #5
    Wrong. The rebuild process is done by the controller, not the disks. If the controller stinks at rebuilding, it won't matter what size disks you use.
     
  6. rylin macrumors 6502

    Joined:
    Aug 18, 2006
    #6
    When a drive dies or is replaced, all the remaining disks are read and the controller calculates the "correct" bits to write to the new disk.

    With UREs roughly every 10^14 bits, it's just a question of time before you hit an URE.
    If it happens during rebuild, the rebuild fails, and since you've just had another disk failure, your data is poo-pooed.

    It's not the controller, it's maths, and it's the medium.

    If you have a array consisting of 250GB disks, the odds are lower that you'll hit an URE simply because there's less bits to read in order to rebuild a failed array.
     
  7. belvdr macrumors 603

    Joined:
    Aug 15, 2005
    #7
    I see. Mathematically, it looks that way, but I've never seen that in any of the arrays I maintained and they were all 300GB disks or better.
     
  8. rylin macrumors 6502

    Joined:
    Aug 18, 2006
    #8
    Using a hot spare or two is definitely best practice, as is using enterprise class drives, and most importantly having a solid backup strategy.

    I strongly recommend you read http://subnetmask255x4.wordpress.com/2008/10/28/sata-unrecoverable-errors-and-how-that-impacts-raid/ and the linked articles if you want a more in-depth analysis of the problem.

    Basically, if you're looking at using the R5 as a work/scratch disk, that's most likely fine. The big question is how you and your system handles catastrophic failure.

    Personally, I'd hate having a few TB of data going missing ;)
     
  9. rylin macrumors 6502

    Joined:
    Aug 18, 2006
    #9
    The problem is it's a relatively new "problem".
    Disk sizes increase faster than our ability to improve the medium.

    There's also a chance people won't get affected by the "problem" even if they're running a massive array based on large consumer drives, since you need a primary failure followed by a secondary failure during the rebuild.

    In the end though, it's all just maths.
    It's just think that if people want to gable with their data, they should do so knowing the risks :)

    If a 10-disk array helps me work faster and be more productive, and I have a backup strategy which means I can restore from backup quickly, it might be a reasonable risk/reward strategy.

    In other words, it's as always up to individual needs.
     
  10. scaley thread starter macrumors member

    Joined:
    Mar 3, 2009
    #10
    Thanks to everyone thus far for the responses and I am definitely up on many of the pros and cons of the different RAID setups which is less of my concern. If I could attempt to shift the conversation back again to my original questions what I most need to know are:

    1.) User experience with the Rocketraid vs. Caldigit vs. other RAID cards in implementing a RAID array on a Mac Pro (and additionally how the configuration software is - seems like Caldigits is nice, don't know about others)
    2.) The performance differences to be expected between moving to an internal hardware RAID with one of the above mentioned cards (or others you may suggest) and my current setup which is and external Venus t5 (with hardware Sil 4726 controller) interfacing through eSATA
    3.) How much does PCIe4x vs. PCIe8x matter in performance / throughput since this seems to be a difference in many cards?
     
  11. subnetmask255x4 macrumors newbie

    Joined:
    Mar 11, 2009
    #11
    First, let me say that I am sorry about the train wreck my article has caused to your original question. While I have not used any of these cards, I have some information that might be helpful.

    From the User Manuals, the Rocketraids both use a web browser interface to control them, while the CalDigit has an OS X native program called RAID Shield. From what I can tell, the RAID shield software has more options, such as emailing alerts.

    You will see a larger hit by going to RAID-5 from RAID-10 than from the external to internal move. Something around a 25% performance hit.

    Not that much. PCIe4x supports up to 1000MB/s and PCIe8x supports up to 2000MB/s, each way. Individual SATA 3Gbit/s drives can move only 300MB/s at max, and it would take 4 drives in RAID-0 to fill PCIe4x. Therefore, there is simply no way any RAID controller will be filling that kind of bandwidth.

    As for the content of my article, Unrecoverable Read Errors only become a probability when you start reaching RAID sizes over 10TB.

    My biggest question is do you want to go to RAID-5 over RAID-10 because of reliability, or because RAID-10 only has 1/2 of your drive space available? If it is the first one, then remember that straight-up RAID-5 does not include a hot spare, so you only get one dead drive free, the next one kills the data. With RAID-10, you can lose one drive free, and then one other drive if it is one of the other mirrored pair without losing data. If it is the second, do you need 4.5TB of space (in RAID-5) or would 3TB do it (in RAID-10)? While you lose more space in RAID-10, it has a definite performance improvement.

    Just as a side note, performance improvement of RAID-10 over RAID-5 starts to diminish as the number of drives increase. At 3 drives, the difference is around 10% per drive, at 4 drives it is 6%, at 5 it is around 4%. That is why at 6 to 8 drives, the performance difference is marginal, so you get more space, and guaranteed reliability with RAID-5 with a hot spare, or RAID-6 with or without a hot spare.

    Just my couple of cents worth.
     
  12. scaley thread starter macrumors member

    Joined:
    Mar 3, 2009
    #12
    Thanks for your post! This is exactly what I needed to know and so I managed to make my decision pretty quick after hearing this and so I went with the Rocketraid 4320 since I don't have to worry about dealing with any of the proprietary business with Caldigit and it has a much faster processor and not to mention support for SAS drives.

    Ok but onto your comments. I am still surprised (mostly because of my lack of knowledge) that cramming all the external raid through 1 eSATA port wouldn't make that big of a difference because it would seem like the CPU of the Mac Pro then still has to deal with all copy operations through that port and then off to the external case Raid controller and the harddrives are bottlenecked through that interface. But no worries I purchased my Raid card and so now it is on to the next thing... the configuration.

    My absolute biggest concern is balancing space with reliability. I do deal with a lot of large files (numerical modeling, images, etc) and so space is always at a premium. However, after having gone through a hard drive failure before that cost $1700 to recover professionally I absolutely need redundancy and hence that is why I moved to RAID 10 originally.

    So where I am now is wanting to expand that over the 5 available 1 terrabyte hard drives I have. So being it isn't an even number of discs Raid 10 seems like it's out and Raid 5 or 6 seem better. So now I'm debating between the following and welcome advice as to which is better:

    RAID 5 + Hot spare (3 Tb effective)
    RAID 6 (also 3 Tb effective)

    Either way I gain 1 Tb back over my current Raid 10 setup. My feeling is RAID 6 is better since it can handle 2 failures. But then again Raid 5 with the hot spare is essentially the same unless a drive fails during rebuild. Thoughts? Is there anyway that you would risk RAID 5 without a hot spare?
     
  13. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #13
    Given the use of consumer drives, RAID 6 would be better suit your needs from what I can tell. ;)
     
  14. subnetmask255x4 macrumors newbie

    Joined:
    Mar 11, 2009
    #14
    Exactly. I would go with the RAID 5 with the hot spare because there is a distinct performance hit with RAID 6. I would only use RAID 6 with a very large array (>10TB) and then you are getting into hardware systems that can handle that without the performance hit.

    Only if I had a backup, be it online or offline (tape, etc...).

    Honestly, RAID 5 with the Hot Spare will give you better performance than RAID-6, and still give you the redundancy that you need.

    Good Luck.
     
  15. rylin macrumors 6502

    Joined:
    Aug 18, 2006
    #15
    Personally I'd go the RAID-6 route (or ZFS RAID-Z2 despite the issues that still exist), as the hot spare is only "hot" in the sense that it'll automatically start the rebuild when a failure is detected.

    In reality, it's not that different from removing the failed drive and plugging a new one in.
    At this point, the system would be in such a degraded state that I wouldn't want to "work" with it at all (especially considering it'd increase the time it'd take to rebuild the array, giving more room for failure).

    Of course, the better thing to do would be to just make sure to have an even number of disks and still do RAID-10, but we all make our own choices depending on our own usage scenarios :)

    All in all, I'm somewhat glad the OP has had a catastrophic failure before (despite it being costly), as it's more or less the only way people will ever bother with redundancy and fault tolerance.

    I've seen way too many people here post about how much they like their 4-disk R0 setups and how "it's RAID so it must be good" :(

    Bottom line?
    Disk is cheap.

    Memories, hobbies and work lost isn't.
     
  16. scaley thread starter macrumors member

    Joined:
    Mar 3, 2009
    #16
    Oh I should have mentioned that I actually am using Seagate Barracuda ES.2 drives with the exception of one (which is the consumer Barracuda) since I had heard to stay away from the consumer drives as well. I appreciate the advice.

    Now the only remaining issue is how to connect all this up. I really don't want to pay $90 bucks for the maxConnect miniSAS-miniSAS cable to reach my rocketraid card. There has to be a better way that doesn't involve $90.
     
  17. scaley thread starter macrumors member

    Joined:
    Mar 3, 2009
    #17
    Yeah, I had a Big Lacie drive (RAID 0) where one of the discs failed but luckily by that time I had it backed up on something else. What cost me $1700 was when my MacBook pro MELTED (literally the screen was destroyed) and consequently the 2.5 drive became a pile of trash that had to be recovered. That is what did it for me. But yes. Point taken.

    I don't think I'll manage to get to Raid 10 though given that I can't configure this thing with 6 drives. There is no way that I want to make my system on the array so that is staying on a separate drive of the 6, hence 5.

    I am debating moving the home user directories onto the RAID or alternatively will just make a time machine partition on it that I can backup the system disk with the user directories to.
     
  18. scaley thread starter macrumors member

    Joined:
    Mar 3, 2009
    #18
    Based on an earlier comment about the performance gains to be expected from going from the external eSATA Venus 3 raid box with the 4726 controller in it versus moving to an internal Raid I am now convinced I HAVE to do better by a LOT than the external case. I just ran AJA system test on that external array that bottlenecks through the eSATA and it is limited to a read / write of 100 MB/S. Shouldn't I get 3-4 x that with a 5 disk RAID 5 with hot spare on a Rocket Raid 4320? Below are the test results.
     

    Attached Files:

  19. Rtech macrumors newbie

    Joined:
    Mar 12, 2009
    #19
    Scaley,

    I've had very good experience with the Areca RAID controller. I'm using the ARC-1212X controller, with 4x 1TB drives in a CineRAID 4 bay tower. I'm getting over 310MB/s read and write speeds in RAID 0 using the AJA Kona test. Even with my storage capacity filled at about 30% full.

    Also, if your still looking for the MiniSAS cables, check out http://www.tekramonline.com/Accessories_Cables2.html, they have the external MiniSAS cables for $59
     
  20. subnetmask255x4 macrumors newbie

    Joined:
    Mar 11, 2009
    #20
    Yeah, but then again if a RAID-6 drive fails and you plug a new one in, it still has to start a rebuild of data onto the new drive. While a "good" controller will try to do that rebuild during idle times or at least only when you are not trashing the disks yourself. However, the performance hit from going to RAID-6 can be big. This benchmark from HighPoint http://www.highpoint-tech.com/PDF/RR4320/RocketRAID_4320_Performance_Benchmark.pdf shows that there is around 12% performance loss in sequential read/writes, and over 25% loss in "real world performance I/Os." While this is a bit lacking in actual information, it is enough to see the difference.

    Now, I will say that some systems won't see a hit. We have a pair of Promise VTraks here at work, and I don't see a difference in performance between 5 and 6. But then again, a basic 8 drive setup costs over $11,000, so there better not be a performance loss!

    Wow, something must really be limiting the performance there. You will probably see a 3x performance gain on burst read/write, and a 2 to 2.5x gain on full throughput. The really limiting factor to RAID-5 or 6 performance is the fact that the controller has to read data from all drives, and then verify the parity, before sending it on. RAID-1 allows the controller to read data from any of the mirrors, so it is as fast as the fastest of the drives. RAID-0 allows the controller to read 1/n of the data from each drive, so you can "theoretically" add the speeds of the drives together.

    If I were you, I would be seeing about moving the entire system to it. If not, then any failure of the system drive means that you have to rebuild your system to the way it was, and then you can start using the data from the RAID system. Since the performance of this setup will not be all that different from a good internal drive (maybe even faster), then you have protection for EVERYTHING, rather than just your data files. See the Tao of Backup http://www.taobackup.com/ , especially the first part "Coverage." Much like the novice, if you only protect user data, then you can spend days reinstalling software to get your machine back to where it was.
     
  21. Virtuoso macrumors regular

    Joined:
    Feb 21, 2008
    Location:
    Seattle
    #21
    I've had the Apple RAID Card and I've been using the Caldigit Card for about a year. I got off to a pretty rocky start - the battery in the Apple card wouldn't charge so I returned it. I'm glad I did, the reconditioning cycle is ridiculous - every 3 months your performance is reduced while it discharges and recharges the battery, which some people found can take a few days to complete. The Apple card also doesn't support Boot Camp.

    As for the Caldigit Card, the early drivers had some major issues, but it's been running fine since those issues were sorted out. It did however take them almost a year of hollow promises and missed deadlines to get the battery and HD Element options onto the market.

    I run 4x750GB Western Digital RE2 drives in a RAID 5 array which gives just over 2TB of usable space, plus a 1TB WD Caviar Black boot drive. Performance from the RAID (tested with the Aja util) is 268MB/s for read or write. If you need more capacity or speed, there are three additional ports on the back of the card, each of which support an extra 4 drives. The downside is that you can only use Caldigit's external arrays which are extremely expensive - 4x1TB drives currently costs around $400, but the 4TB Caldigit box is $2,300!
     
  22. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #22
    What you describe is what soured me on CalDigit's products. :(

    It may not even be much of an option with the '09 models anyway, since Apple did away with the internal MiniSAS wiring, and went to soldered HDD ports on the logic board.
     
  23. Rick Here macrumors member

    Joined:
    Oct 9, 2007
    #23

    Yep, in the 2009 MAC Pro, looks like the Apple new RAID card is the only solution for internal drive bay and external RAID for 3rd party stuff. But when USB3.0 appears, which is 5Gbps, the world will change for external storage.
     
  24. scaley thread starter macrumors member

    Joined:
    Mar 3, 2009
    #24
    Yes it was a bit of a shock to me as well and I think that it has something to do with that particular case and controller (the Sil 4726) but I can't verify this yet. Either way the point is mute because I'm going internal now :D I got a screaming deal on the Highpoint RocketRaid 4320 for $350 bucks on newegg this week which is the card I'd been pretty much set on anyway so it should be here on Monday.

    This is something that I'm still struggling to come to grips with and believe me I see your point in the performance increases and possibly the backups but I have a few concerns about this based on reading these forums and posts by Nanofrog and others about this. It sounds like it is a bad idea to boot from the array from the perspective of upgrading the OS - bugs with drivers, etc in that you may have no way to get back to your system and data if this goes wrong. I don't know if that is actually the concern but I thought maybe pulling the user data off onto the array would at least assuage some of those concerns. At a minimum the data would be safe and I would not have to worry about cloning the OS onto the Array.

    Maybe I have that all wrong - I'd appreciate insights into that too since there seems to be a wealth of knowledge, and perhaps more important, experience with these issues.

    WOW! This is sad and I guess is just one more reason I'm glad I got the 2x2.8 Ghz 2008 MacPro and not the 2x2.26 2009 MacPro.

    So the picture now is:
    Purchased the Highpoint RocketRaid 4320 still have the 6, 1 terrabyte Seagate Barracuda ES.2 drives, and now just need to know:

    1.) Is the MaxUpgrades MiniSAS-MiniSAS extender cable the only option for reaching the card? No one seems to make a female to male internal miniSAS other than them but the $89 price tag is near highway robbery.
    2.) Why is booting from the array not the best option? I've come to this conclusions based on previous posts (see points above) but anymore information is great.
    3.) I'm thinking of machining my own case for the optical bay housing. We have a machine shop here and I don't really want to pay $89-$139 bucks just to put 2 extra drives in the optical bay. I think I could do this for less than $30 bucks but I have to look at the design of that thing. Anyone else have a DIY mod for these? Pics / advice?
     
  25. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #25
    1. Yes, unless you have the inclination and tools to either make one, or to extend the power connections. If so, you can solder in some additional wire, allowing the MiniSAS end to extend farther. Otherwise, the Maxconnect device is the only one ready-made I'm aware of.

    2. Highpoint's cards can only handle firmware (OS booting) for one OS (either OS X or Windows/Linux). So if you want to boot OS X, it couldn't boot windows or linux. (Not sure if this is an issue for you).

    3. If you have an old CD or DVD ROM drive lying about, just strip it down, and use one of the metal enclosure plates to make a mount. 2x 3.5" HDD's will fit on one. Easy, and far cheaper. Faster too, as you already have what you need. :D
     

Share This Page