using RAID with a MacPro

Discussion in 'Mac Pro' started by rassoodock, Apr 21, 2012.

  1. rassoodock macrumors member

    Joined:
    Jan 22, 2007
    #1
    After years of using a mixture of internal and external drives with my MacPro, I have finally decided to streamline things and get a desktop RAID array to protect my data. I currently only have about 5TB, but want to put together something that I can grow into if possible. As average users accumulate large amounts of data, audio and video and move away from removable disc media, I think this subject will be of growing interest to many.

    This area is new to me (especially a RAID array, which I have never dealt with before). So, with that in mind, I’m going to layout what I’ve come up with so far (along with some questions) and see what people think. As always, any help, suggestions or warnings are most appreciated.

    Of course, if this topic has already been discussed or if someone knows of a source of information that would be useful, please be so kind as to let me know about it. I like to do as much research as I can on my own and only ask questions where needed.

    Here we go:

    MY COMPUTER
    MacPro 2 x 2.66 GHz Dual-Core Intel Xeon
    Memory: 2 GB 667 MHz DDR2 FB-DIMM
    OS X 10.6.8

    INTERNAL DRIVES
    I plan to fill three internal drive slots with 3TB drives and transfer the data off my other drives. The ones I think I want to get are Western Digital 3TB Caviar Green drives. I know they spin at 5400RPM, as opposed to 7200, but these drives are mostly for storage and a lightning quick loading time is not essential for when I do need to access them.

    DESKTOP RAID ARRAY
    Here is where I get somewhat lost. I know that I want to have the capacity to backup 12TB using at least RAID5 and have it work with Time Machine. I am aware of Thunderbolt, but it is too new to be of any use to me now. Ideally, I’d like to get a RAID enclosure that has both a FireWire AND a Thunderbolt connection so I have the opportunity to take advantage of Thunderbolt in the future. However, since my RAID array is for backup only with Time Machine, lighting fast transfer rates are not crucial.

    I have a number of questions regarding RAID and how to do this:

    1. I believe that what I want is RAID5, but perhaps something else would be better suited for my situation. I want to be able to survive one, perhaps two, drive failures.

    2. I would like to get an array that can backup a max of 12TB from the drives in my MacPro. I know that the RAID level I choose determines how much capacity I will need the array to have. Does such a large capacity exist for desktops?

    3. In general, do you have to use certain types/brands/capacity drives in a RAID enclosure or can you drop in anything and have it work? I’ve read contradicting information, so I’m a bit confused on this one.


    Well, that’s about all I can think of right now. I’m sure I’m missing some crucial question or leaving out a vital piece of information. However, I hope I’ve provided enough detail to let you know what I’m trying to accomplish.

    Any help, suggestions or warnings are most appreciated. Thank you.
     
  2. ashman70 macrumors 6502a

    Joined:
    Dec 20, 2010
    #2
    Rather then do something internally, which will require a hardware raid controller, why not consider doing something external, such as a firewire raid enclosure, or get an eSata card and go eSata instead.

    External enclosures are better for future upgrades and expandability.
     
  3. ssgbryan, Apr 21, 2012
    Last edited: Apr 21, 2012

    ssgbryan macrumors 6502a

    ssgbryan

    Joined:
    Jul 18, 2002
    #3
    I have a setup that is almost exactly what you are asking for.

    Your computer:

    Get more ram. 2GB has to be incredibly painful to use. In the 64-bit world, ram is more important that raw clock speed. I am at 22Gb & crossing the 16Gb barrier made a great deal of difference. Just going to 4 gigs would be a major increase in system responsiveness. If you are going to keep a 1,1 going for a while, get 8Gb (2x4Gb) kits (On ebay, they can be had for about $180 per kit.) otherwise, just go with 2Gb sticks.

    INTERNAL DRIVES
    You can actually install 6 hard drives internally to a MacPro 1,1 or 2,1 without needing an additional card.

    There are 2 unused SATA II ports on the motherboard. I purchased a Pro Caddy 2 from Transintl.com ($89)

    http://www.transintl.com/store/category.cfm?category=2790

    I also purchased 2 OWC 240 SSDs and put them in a raid 0 (480Gb).
    http://eshop.macsales.com/shop/internal_storage/SSD/Mercury_Electra_3G_Solid_State

    I went with SATA II because I don't have an internal SATA III card. The Mercury 6G Electra's are $20 more & are SSD III. (They weren't available when I bought my SSDs.) The 6g SATA III's can be moved to a newer system when the time comes.

    Internal bay 1 & 2 hold 2 Hitachi 2TB drives in a RAID 0 (4TB) for my data. Bay 3 & 4 hold 2 additional Hitachi 2TB drives in a different raid 0 (4TB) for my iTunes library. I could actually fit everything on the first RAID, but I got lucky on a sale for Hitachis. (This was pre-Thailand flood.)

    You can't do an internal RAID 5 without a number of additional items (RAID Card & cabling).

    My system's total internal capacity - 1 480GB SSD Raid 0 boot drive and 2 4TB data drives.

    I could go & get a DX4 & an internal RAID Card:

    http://www.transintl.com/store/category.cfm?Category=2799

    and add an additional 4 2.5 hard drives (ssd or notebook drives). As to whether or not it would be cost effective is up to the individual user and their needs.

    EXTERNAL DRIVES
    To run a raid 5 you need 4 hard drives. You will have the storage space of 3 drives (the 4th drive is the redundancy made through parity stripping).

    If you want 12TB in a RAID 5, you are looking at buying 5 4TB drives (Hitachi's are the only one I have seen so far - they go for about $320 - 350 each) - so right there you are looking at over $1,500 right there.

    Why 5 drives when you are looking at a 4 enclosure box? Because when (not if) one of those drives die, you want to insure that you replace a bad drive with one from the same run - you can run into real issues with drives that have differing firmware levels.

    I have a OWC Mercury Elite Pro Qx2

    http://eshop.macsales.com/item/Other World Computing/MEQX2KIT0GB/

    That will put you back anywhere from 175 - 275 depending on where you buy it from.

    In my Qx2 I have 4 2TB Western Digital Green hard drives in a RAID 5 (6TB capability). It backs up my internal system. The fact that they are 5400 drives is irrelevant once you put them in a RAID. Time Machine doesn't saturate the FW 800 connection.

    So far, no fuss, no muss.

    One thing to keep in mind with a 1st or 2nd Generation MacPro is that we have reached the end of the line. If Lion & Mountain Lion are not your cup of tea, then it isn't a problem. If you buy anything to go inside the MacPro, I would recommend that you ensure it can move to a newer machine (Drives, Video Cards, SATA Cards - memory won't move).

    There is a lot of things you can do to make a 1st Gen Mac Pro be very, very responsive, but at some point you will reach diminishing returns.

    I have gone all in with my system. It was cost effective for me because I shopped around extensively, and $100 here and there was easier than dropping $3,500 and then starting to upgrade it, although some items I didn't have a choice on (I am on my 3rd video card - ATI 5770 from Apple.)

    I haven't replaced the CPU's just yet, but for $90 each & some elbow grease I'll have the speed of the latest iMac - with all of the pluses of a MacPro.

    I find it a better deal than dropping $3,500+ on a new MacPro right now. I will be getting a new Mac Pro, but not until next year.
     
  4. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #4
    There are a lot of RAID threads here in MR if you search. The specifics may not be identical, but there is still a lot of good information in there that you need to understand.

    Keep in mind, that the power management settings in Green drives mean they like to spin down. So access will be slower initially than you may realize, as they have to spin up first.

    Forget about ThunderBolt.

    Now let's get something really basic, but crucial about RAID5 (parity based array). RAID can be done two ways; software and hardware. Software based implementations are fine for stripe sets (RAID0), and can be used for mirrors (RAID1), and JBOD (concatenation) as it offers a lower cost to implement it (by skipping specialized hardware needed to do the job).

    When dealing with parity levels however, a software implementation is NOT the way to go, as it does not address the write hole issue associated with it (RAID5/6/50/60/51/61). A hardware implementation is the only way to really go, as it actually addresses this issue (only way to do so).

    Without it, your data is in serious peril (write hole information can be found in Wiki).

    Due to cost reasons, RAID5 is likely the best solution (fewer drives needed to get the desired usable capacity).

    Yes, its possible to get a solution that will hold 12TB in a RAID5. Not exactly cheap though.

    Not only do you need the external enclosure, you'll need a RAID card if the enclosure does not include the RAID hardware, and you must use Enterprise grade HDD's (not an option, as they have the correct firmware to be used with such cards/enclosures).

    Consumer drives are not stable in this sort of equipment (wrong firmware timings), so do not try to use them. You'll go through absolute hell if you try.

    Assuming you want all 12TB in a single volume, you'll need a single controller tied to all members (disks) in the set. Now to get 12TB in a RAID5, let's say using 2TB disks, you'll need n+1, or 7 * 2TB = 14TB of total capacity. Usable capacity = single disk capacity * (n-1), or 12TB in this particular case.

    This means you'll need 8 HDD bays (can be more than one enclosure if they're tied to the same controller), or a single 8 bay controller that has the RAID hardware included (http://www.newegg.com/Product/Product.aspx?Item=N82E16816151070).

    Now looking at the link, you'll see that this gets expensive fast, as you don't even have disks yet (can use either FW800 or eSATA if you've an eSATA card installed in the system, which would be the faster way to transfer data).

    2TB disks that are suitable (example). So tack on another $2340 for disks....

    You're over $3300.

    Now since you're after a backup solution, there may be ways to do this cheaper IF you're willing to use 2x separate volumes to do so. Specifically, the Qx2 (here), which can be used with consumer grade HDD"s would give you 12TB of RAID5 at much less money. The hardware isn't nearly as good, but it is a hardware solution.

    Type, Yes. They must use an interface that will work with the equipment (i.e. SAS or SATA), but you do need to pay attention if you're using SAS. Where it does get critical however, is you must use Enterprise drives as already mentioned.
     
  5. deconstruct60, Apr 21, 2012
    Last edited: Apr 21, 2012

    deconstruct60 macrumors 604

    Joined:
    Mar 10, 2009
    #5
    If these are 3 separate volumes then can back them to separate images. You're likely not going to need some fancy RAID config to do that. If you have 5TB now then 3 2TB drives would bump storage up 1TB. (or two 3TB drives and one 2TB drive if they are separate volumes they don't have to match. )

    If you are going to glob these together into one volume with RAID 1 then the rebuild time is going to be large in the event of a failure. Even as a JBOD has rebuild time issues.


    Maybe this is time machine driven but backing up to one volume brings drama. One problem with TimeMachine backup volume matching the same size as the volume being backed up is that you may not get much "history" in your backup.

    For example, if you had two 2TB and one 3TB. drives internally and backup up to two 3TB and one 4TB drive externally on each there would be room for:

    a. the data
    b. about 500GB of versions of the "old version" data
    c. 500GB of free space. [ pragmatically you don't want to fill the file system up to the brim.]


    There is no RAID here at all (could use RAID-1 for the back-up volumes to add some additional safety if not gong to do rotation). You can backup to external drives. Even put a rotation of external drives into the mix so that could secure offsite backups (tree falls on your Mac Pro could also wipe out your direct attached storage "back-up" box too. ) .


    If you back-up three different 3TB volumes onto one 12TB RAID-5 volume, you will have competing back-ups vying for the same "extra space" for the "old version" data. You'd quickly get to the disk full. There is a buffer you'd need to add above and beyond the original source size.


    RAID-5 typically requires that you add battery back-up to your system set up. Sudden power failure can wipe out the array. Something that would power the Mac Pro and enclosure box until a shutdown command completed would work.

    P.S. While HFS+ now goes up to 8EB (http://support.apple.com/kb/HT2422) max volume size, good luck trying to get a repair done on the file system as you creep 10TB and higher. The question becomes what back-ups the back-up.
     
  6. rassoodock thread starter macrumors member

    Joined:
    Jan 22, 2007
    #6
    Wow! Thank you all for such detailed and informed information. I must say that a lot of it goes a bit over my head, so I know I have some research and learning to do. There are a lot of options to consider thanks to everyone's helpful posts. I will get to work on going through all your tips and suggestions, but might need to come back with additional questions. Again, thank you for taking the time to help a longtime Mac user, but a complete RAID newbie.
     
  7. throAU, Apr 22, 2012
    Last edited: Apr 22, 2012

    throAU macrumors 601

    throAU

    Joined:
    Feb 13, 2012
    Location:
    Perth, Western Australia
    #7
    Heads up: 3 drives will only get you RAID5 or RAID0 and those give you 1 drive failure or 0 drive failures before data loss respectively. You sacrifice 1 drive's worth of space.

    If you can stump the cash for a 4th drive, you can go to RAID10 (stripe across 2 mirrors), which will give you up to 2 drive failures (one in each mirror - 2 in one mirror of the stripe is still data loss). You sacrifice half your space.

    RAID5 has good read speed, but relatively poor write speed when doing "non full stripe writes" (e.g., if your stripe size is 64k, and you need to write 32k of data your controller needs to do a read from all 3 drives first, recalculate parity then re-write the data).

    RAID5 has very poor performance whilst under a failure condition and slow rebuild time.

    RAID10 has minimal (like 6%) performance penalty whilst under a drive failure condition and rebuilds are much faster. Write speed is faster especially if the writes are not "full stripe" - there is no read necessary to calculate the parity for the rest of the stripe (as its just a simple mirror - the controller just writes the same data at the same time to the other side of the mirror).

    It definitely depends what your needs are, but IMHO raid 5 is evil and to be avoided.

    see:

    http://www.miracleas.com/BAARF/BAARF2.html

    In particular, consider this:
    http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt
     
  8. wonderspark macrumors 68030

    wonderspark

    Joined:
    Feb 4, 2010
    Location:
    Oregon
    #8
    I run a RAID6 and couldn't be happier. 12TB, any two of the eight can fail without data loss and rad performance, even during drive failure.
     
  9. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #9
    For a software implementation, absolutely. But not so with a good hardware controller.

    Newer RAID cards are faster at writes than 10 these days. Rebuilds are faster as well, but as always, it depends on the actual components used on the card. For example, and older IOP processor from Intel (i.e. IOP348, circa 2006) is faster for RAID5 rebuilds and initializations than the much more recent 6.0Gb/s LSI RAID chips available from what I've experienced.
     
  10. throAU, Apr 24, 2012
    Last edited: Apr 24, 2012

    throAU macrumors 601

    throAU

    Joined:
    Feb 13, 2012
    Location:
    Perth, Western Australia
    #10
    The controller can only do so much. There is an inescapable fact that to do a partial stripe re-write (as opposed to a full-stripe write to a clean part of the disk), all disks need to be read from first. This is true for all RAID variants that use parity - RAID4, RAID5, RAID6...

    Put RAID10 on the same controller and the write, rebuild and degraded performance will be far superior.

    The same is true even for enterprise storage arrays. Netapp get equivalent to RAID10 performance with their RAID6 WAFL variant (essentially, they avoid the problem by bundling writes so they are all full stripe and write only to unused blocks if possible), but this depends on sufficient free space.

    Once the array is >90% full, performance degrades as re-writes need to happen, which means that for a partial stripe re-write, all disks in the stripe need to be read before the write can occur - which delays the write and slows down concurrent reads as well... while the disks are reading for the write they aren't reading what you want read.

    Cache can help, but once your hot data is bigger than your cache, you're back to physical disk performance.
     
  11. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #11
    I'd actually figured the "evil" comment was in regard to the write hole issue, not rebuild times, which is why I mentioned the software aspect (RAID5 via software is a matter of when, not if a failure will occur).

    What I was getting at in terms of controllers and speed, is the old adage that "10 trumps 5 in all conditions is no longer true" (I still see a lot of this). Even rebuilds of parity sets are decent these days, and for mission critical enterprise systems, there are ways to mitigate it (split up into multiple sets using multiple controllers for example), and possibly nested in order to reduce the rebuild load per parity set/controller.

    I do realize what you're getting at, but that's part of designing the correct storage system for the specific conditions from my POV (i.e. what level/s, how many controllers, what degree of redundancy, ... down to the stripe size in order to get the best overall balance of performance, uptime, and cost) rather than just make a blanket statement.

    In the case of a backup system for a single user (not the primary array), cost is almost certainly more critical than rebuild times, and given that the OP means to use Time Machine, 5 would be suitable for that task.

    I'm not saying 10 is useless in this case (as this fact may not be clear in the rest of the post), just not so sure if it's the most cost effective way to go without further specifics (namely due to usable capacity = n*2 disks; as the member count increases, the cost advantage to a software based 10 implementation disappears vs. 5 or 6 for a given usable capacity).
     
  12. rassoodock thread starter macrumors member

    Joined:
    Jan 22, 2007
    #12
    You make a good point. This backup would be for one person, backing up one computer with a 12TB capacity. While a quick rebuild after a drive failure would be nice, I would still have all the data on my primary computer. Cost definitely does play a part in this. I'm not against paying for quality, but at a certain point, I have to ask myself if I'm bringing a bazooka to a bow & arrow fight. Ya know what I mean?

    I'm still sifting through all the excellent suggestions, but one thing has become clear: if I go with RAID5, do it with hardware and not software.
     
  13. pprior macrumors 65816

    Joined:
    Aug 1, 2007
    #13
    I also run raid6 on my main data array. I back that data up to a raid-5 array in the basement for time machine. I've had to rebuild both arrays due to bad disks over the last couple years and it's VERY comforting to know that another disk could go bad during the rebuild and I'd still not lose data on the raid-6.
     
  14. wonderspark macrumors 68030

    wonderspark

    Joined:
    Feb 4, 2010
    Location:
    Oregon
    #14
    I just had my first failed disk... sort of. After shutting down for the night and starting it up the next morning, one disk failed to spin up and join the RAID. I went to pull it and reinsert it, and ended up pulling the wrong disk, so now I showed two failed disks, of course. Whoops!

    I put the good one back and it started a rebuild of the RAID, yet still no data loss. I found the correct failed disk (not gonna pull a third one by mistake!) and pulled it. After maybe three or four reinserts, it rejoined the RAID and was happy.

    I kept editing while it rebuilt, and that night I shut it down from a normal state. The next morning, the same drive failed on startup. Western Digital sent me a new disk (thank you 5-year warranty!) and all is well again, but it goes to show that RAID 6 can be extremely handy if you happen to pull the wrong disk in a failure. If I was in a RAID 5 and did that, I would have been reloading the RAID from my backup instead of pulling and plugging disks and being able to keep working.
     
  15. rassoodock thread starter macrumors member

    Joined:
    Jan 22, 2007
    #15
    OK! Finally made a decision and have a few more questions:

    I’ve decided to get a 16TB OWC Mercury Elite Pro Qx2 running RAID10. I know it won’t give me the 12TB I wanted, but I’ll deal with it because RAID10 seems like the way to go. I like this unit because it has been well reviewed and I don’t have to add a controller card to my Mac Pro — just connect via FireWire.

    http://eshop.macsales.com/item/Other%20World%20Computing/MEQX2T16.0V/


    However, I have noticed that they also offer a 12TB with Enterprise drives:

    http://eshop.macsales.com/item/Other World Computing/MEQX2T12.0E/


    Obviously, using consumer grade drives is MUCH cheaper than Enterprise drives, so my question is this: is it worth the enormous cost to use Enterprise drives? Are consumer grade drives so much more unreliable that it’s worth it? A 3TB Enterprise drive is almost $600 and I assume I’d need to buy at least one extra to have on hand to replace a failed drive — one with the same firmware. That is what you’re supposed to do, right?


    QUESTION: I’ve noticed how limited Time Machine actually is and am wondering if there is a better backup utility for Mac that allows you more versatility and, most importantly, allows for your backups to be encrypted or at least password protected. Time Machine seems rather basic, so there must be something I can buy with more features and options. Can anybody suggest or recommend one?

    QUESTION: If I do end up going with Time Machine, would it be a good idea to initially partition 95% - 5% (or something along those lines) so that Time Machine is unable to MAX out the RAID with historical copies. I would say that 80% of what I’m backing up will not have changes and will just be sitting there.

    My sincere thanks to everyone who has contributed or will contribute to this thread. Your help and suggestions are most appreciated.

    P.S. I followed a suggestion made here and bought another 4GB of RAM for a total of 6GB. Thanks!
     
  16. nanofrog, May 17, 2012
    Last edited: May 17, 2012

    nanofrog macrumors G4

    Joined:
    May 6, 2008
    #16
    It depends on the specifics.

    If you intend to use this as a backup only (another copy of the data on a primary array/volume), then consumer grade drives will be suitable so long as you run it in 10 (or 0/1 only; RAID 5 would be better off on Enterprise disks).

    Please do not interpret this as an indication of me being a proponent of a single copy of data, unless it's temporary data (see past posts for further details as to what levels do/do not require Enterprise grade disks).

    Now if this is actually your primary array, then you will need a sufficient backup system of some kind (under the impression this is not the case, but if it is, take the necessary steps to insure your data).

    You'd be better off running a 3rd party backup application IMO, but I've not used a Mac version in some time. So others would be better suited to recommend current versions available.
     
  17. rassoodock thread starter macrumors member

    Joined:
    Jan 22, 2007
    #17
    Yes, this would be for backup only. All the data would be present on the drives in my Mac Pro and backed up to the RAID10 array.

    QUESTION: should I buy the RAID enclosure and drives separately or buy their stocked enclosure? Also, I assume I should also get at least one extra drive from the same run as a replacement in case of drive failure. Correct?


    Yeah, the more I think about what I want Time Machine to do, the more I realize that it does almost none of it.

    So... has anybody out there been using a 3rd party backup app that they can recommend?

    Thanks
     
  18. wonderspark macrumors 68030

    wonderspark

    Joined:
    Feb 4, 2010
    Location:
    Oregon
    #18
    I don't use time machine at all. What I do is back up working data on separate single disks, and keep two clones of my operating system disk. The RAID-6 is my working volume, as it provides very fast data with two-disk failure tolerance. Single external disks as backup work for me because they are only powered on long enough to run a backup after data is changed, so I expect they will last much longer than if they were always on like a Time Machine disk would be. I use SuperDuper! for my clones, and just drag/drop data for the other backups.

    Clone 1 gets updated every one to seven days, and clone 2 is left alone until well after verification of major updates, like 10.6.7 to 10.6.8 for example. This ensures that I have a known working clone in case some update screws up my system.

    Other data that is on my RAID is backed up on single external disks. Depending on how critical that data is, it might be backed up twice (three copies) on single external disks. Disks are so big and cheap that it's not a big deal for me to have a few external disks on the shelf.

    From a failure standpoint, I've had one fail on me. It was one of my enterprise disks in the RAID6, and Western Digital gave me an advanced replacement without any hassle. (I took an advanced replacement because the disk was actually still working, so long as I didn't shut down the RAID. It was failing during cold start-ups only.)

    This is the logic behind my method, but you may find a better way of handling your data. Good luck!
     
  19. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #19
    Run the numbers, and see which is cheaper (generally speaking, I find it cheaper to buy the parts and assemble compared to as close to an equivalent system/configuration as possible). YMMV though.

    Another thing to consider, is you have more control over what you're using when you buy the components and assemble it yourself (i.e. you get good disks, while OWC is likely opting for whatever fits the minimum requirements at the lowest cost).

    As per an extra drive, it's a good idea to have one on hand, so in the event of a failure, it reduces the time the array is in a degraded state (degraded state increases the stress on the remaining members, potentially leading to a catastrophic failure <all data lost>). So long as the primary data is still in tact (internal in the MP), then this isn't quite as critical. But it's still not a bad idea.
     

Share This Page