How do I restore a failed RAID 5?

Discussion in 'Mac Pro' started by dpavid, Oct 21, 2016.

  1. dpavid macrumors regular

    Joined:
    Feb 17, 2004
    Location:
    Mililani, Hawaii
    #1
    I have a ARECA ARC-8050T2. 8 total drives, all WD Red 3TB in each bay, I have 1 parity in the RAID 5 so 24 TB total but only 21TB usable. 2 drives failed within a day of each other before I could get to rebuild one.

    I removed the failed drives from the RAID and can mount the drives on my macpro fine. Disk Utility sees them, unrecognizable of course and ask to initialize because of the way MRAID sets up the RAID.

    The Areca RAID is in degraded status within MRAID since only 6 of the 8 drives are ok. Is there a process I can use to clone the two failed drives to two new ones and restore the RAID? They spin up fine and show on my MacPro.

    Thanks in advance.

    Moderator, please move to another forum category if not in the right place.
     
  2. pastrychef macrumors 601

    pastrychef

    Joined:
    Sep 15, 2006
    Location:
    New York City, NY
    #2
    If you only have single drive redundancy and two drives failed, you are screwed.
     
  3. Phil A. Moderator

    Phil A.

    Staff Member

    Joined:
    Apr 2, 2006
    Location:
    Shropshire, UK
    #3
    I'm pretty sure your only option is to replace the disks, re-initialise and restore from backup
     
  4. chrfr macrumors 603

    Joined:
    Jul 11, 2009
    #4
    You're at the point where you need to restore from backup. I would also expect that your other drives are in imminent danger of failing too.
     
  5. ITguy2016 Suspended

    Joined:
    May 25, 2016
    #5
    If you have another 3GB drive which matches the configuration of the failed drives you could try using dd to make an exact copy of the failed drives. If they're not recognized by the RAID controller then this is unlikely to succeed but it might be worth a try. For a RAID 5 configuration you would only need to recover one of them.

    From a command prompt:

    dd if=/dev/<source drive> of=/dev/<destination drive> bs=4m

    For example:

    dd if=/dev/disk2 of=/dev/disk3 bs=4m

    This example assumes the failed drive has been given the disk2 assignment and the destination drive the disk3 assignment by the OS. Disk Utility should be helpful in determining the drive number assignments.
     
  6. AidenShaw macrumors P6

    AidenShaw

    Joined:
    Feb 8, 2003
    Location:
    The Peninsula
    #6
    Also look at "ddrescue" on Linux. It tries hard to retry any errors, and failing that if fills the unreadable parts with zeroes.

    If you then verify the RAID, you'll see errors on the stripes that have missing data - and perhaps you can determine which files are bad.
     
  7. ActionableMango macrumors 604

    ActionableMango

    Joined:
    Sep 21, 2010
    #7
    Two drives coincidentally failing on their own within a day of each other seems monumentally improbable. And they didn't die, because they both still mount correctly?

    I can't help but wonder if the drives are fine, something else is actually failing, and the symptom is showing as data corruption on the two drives (just those two so far). Something like the RAID controller or power supply.

    Back when I used to do RAID, RAID-related forums were filled with complaints about the RAID controllers failing or causing problems themselves.

    I hope you are able to clone your drives somehow and plop the new ones back in, but I worry that the two drives aren't really the problem in the first place.
     
  8. ITguy2016 Suspended

    Joined:
    May 25, 2016
    #8
    In a situation like this it might be preferable to have it be something other than the drives. Ultimately the data is on the drives and if they're OK the possibility of recovering the data increases dramatically. For instance if the RAID controller itself is failing then a new RAID (same one) controller would probably provide access to his data.
     
  9. monkeybagel macrumors 65816

    Joined:
    Jul 24, 2011
    Location:
    United States
    #9
    It is more likely than you may think, considering the remaining drives must be able to read every sector on every disk to replace the missing data. If it comes across a bad sector during the rebuild phase, it could easily fail. 21TB is a lot of data to read, particularly considering they are not enterprise-class drives.
     
  10. dpavid thread starter macrumors regular

    Joined:
    Feb 17, 2004
    Location:
    Mililani, Hawaii
    #10
    I have exact model identical spare drives. All WD Red NAS 3 TB. That is my 1st thought. To clone the failed drives that the RAID no longer recognizes. I can see both of them on my MacPro via USB host adapters in Data Rescue 3 and Disk Utility. Both drives spin up fine and are recognized in an "unknown" format, assuming it's a structure for the RAID. I can even see the used space.

    Is that the terminal command above to clone the drive?
    --- Post Merged, Oct 21, 2016 ---
    I know right? What are the chances? They are WD RED NAS drives too. Yes, they spin up fine when in a USB adapter unmounted from the RAID so they're not dead. Disk Utility and Data Rescue sees them both. It's a Hardware RAID controller unit that uses MRAID software, not a JBOD striped in Disk Utility. I might have bigger problems.
     
  11. ibgb macrumors member

    Joined:
    Jun 19, 2012
    Location:
    usa
  12. ITguy2016 Suspended

    Joined:
    May 25, 2016
    #12
    It's close. The difference being the specific drive information. If you do not understand the command I would highly recommend consulting with someone who does. Otherwise you could risk overwriting the incorrect disk. If you have a backup of the data that's the preferred method of recovering the information. The advice I've given you is more of a last resort. If you do not have a backup I can try to assist via PM.
     
  13. prvt.donut macrumors 6502a

    prvt.donut

    Joined:
    Jan 1, 2008
    #13
    I have gotten a raid 5 array rebuilt.

    It was expensive though, I sent the NAS and all the drives to a data recovery company that repaired the drives and then restored the array and sent the data back to me on a USB drive. Cost about $10,000 USD.
     
  14. dpavid thread starter macrumors regular

    Joined:
    Feb 17, 2004
    Location:
    Mililani, Hawaii
    #14
    Yes I do.
    --- Post Merged, Oct 21, 2016 ---
    I have a backup of 90% of the data. It's the new recent data that is not backed up. If I had an identical clone, I would just restore from backup and call it the day. Also, I was told the RAID drives also store the SN of the hard drive as part of the RAID build so cloning the failed drive probably won't solve the problem unless you can edit the new drive to match the SN of the failed drive. Is this true? Here is more details about the RAID if it helps from the log.

    The System Information
    ===========================================
    Main Processor : 800MHz
    CPU ICache Size : 32KB
    CPU DCache Size : 32KB
    CPU SCache Size : 1024KB
    System Memory : 1024MB/1333MHz/ECC
    Firmware Version : V1.52 2014-02-07
    BOOT ROM Version : V1.52 2014-02-07
    Serial Number : W410AHELAR200324
    Controller Name : ARC-8050
    Current IP Address : 192.168.1.100


    Raid Set Information
    ===========================================
    Raid Set Name : Raid Set # 000
    Member Disks : 8
    Total Raw Capacity : 24000.0GB
    Free Raw Capacity : 0.1GB
    Min Member Disk Size : 3000.0GB
    Supported Volumes : 128
    Raid Set State : Failed
    Member Disk Channels : xE1S2.xE1S4.E1S5.E1S6.E1S7.E1S8.


    Volume Set Information
    ===========================================
    Volume Set Name : ARC-8050-VOL#000
    Raid Set Name : Raid Set # 000
    Volume Capacity : 20999.9GB
    SCSI Ch/Id/Lun : 00/00/00
    Raid Level : Raid5
    Stripe Size : 128K
    Member Disks : 8
    Cache Mode : Write Back
    Tagged Queuing : Enabled
    Volume State : Failed


    Drives Information
    ===============================================================
    Device Type : SATA(4433221104000000)
    Device Location : Enclosure#1 Slot#1
    Model Name : WDC WD30EFRX-68EUZN0
    Serial Number : WD-WCC4N0894714
    Firmware Rev. : 80.00A80
    Disk Capacity : 3000.6GB
    Device State : FAILED
    Timeout Count : 0
    Media Error Count : 0
    Device Temperature : 255 C
    SMART Read Error Rate : 255(51)
    SMART Spinup Time : 255(21)
    SMART Reallocation Count : 255(140)
    SMART Seek Error Rate : 255(0)
    SMART Spinup Retries : 255(0)
    SMART Calibration Retries : N.A.(N.A.)


    Device Type : SATA(4433221105000000)
    Device Location : Enclosure#1 Slot#2
    Model Name : WDC WD30EFRX-68EUZN0
    Serial Number : WD-WCC4N4DF81LA
    Firmware Rev. : 82.00A82
    Disk Capacity : 3000.6GB
    Device State : NORMAL
    Timeout Count : 0
    Media Error Count : 0
    Device Temperature : 38 C
    SMART Read Error Rate : 200(51)
    SMART Spinup Time : 194(21)
    SMART Reallocation Count : 200(140)
    SMART Seek Error Rate : 100(0)
    SMART Spinup Retries : 100(0)
    SMART Calibration Retries : 100(0)


    Device Type : SATA(4433221106000000)
    Device Location : Enclosure#1 Slot#3
    Model Name : WDC WD30EFRX-68EUZN0
    Serial Number : WD-WMC4N2681497
    Firmware Rev. : 80.00A80
    Disk Capacity : 3000.6GB
    Device State : FAILED
    Timeout Count : 1
    Media Error Count : 0
    Device Temperature : 255 C
    SMART Read Error Rate : 255(51)
    SMART Spinup Time : 255(21)
    SMART Reallocation Count : 255(140)
    SMART Seek Error Rate : 255(0)
    SMART Spinup Retries : 255(0)
    SMART Calibration Retries : N.A.(N.A.)


    Device Type : SATA(4433221107000000)
    Device Location : Enclosure#1 Slot#4
    Model Name : WDC WD30EFRX-68EUZN0
    Serial Number : WD-WMC4N2501694
    Firmware Rev. : 80.00A80
    Disk Capacity : 3000.6GB
    Device State : NORMAL
    Timeout Count : 0
    Media Error Count : 0
    Device Temperature : 39 C
    SMART Read Error Rate : 200(51)
    SMART Spinup Time : 196(21)
    SMART Reallocation Count : 200(140)
    SMART Seek Error Rate : 100(0)
    SMART Spinup Retries : 100(0)
    SMART Calibration Retries : 100(0)


    Device Type : SATA(4433221100000000)
    Device Location : Enclosure#1 Slot#5
    Model Name : WDC WD30EFRX-68EUZN0
    Serial Number : WD-WMC4N2597379
    Firmware Rev. : 80.00A80
    Disk Capacity : 3000.6GB
    Device State : NORMAL
    Timeout Count : 0
    Media Error Count : 0
    Device Temperature : 37 C
    SMART Read Error Rate : 200(51)
    SMART Spinup Time : 195(21)
    SMART Reallocation Count : 200(140)
    SMART Seek Error Rate : 100(0)
    SMART Spinup Retries : 100(0)
    SMART Calibration Retries : 100(0)


    Device Type : SATA(4433221101000000)
    Device Location : Enclosure#1 Slot#6
    Model Name : WDC WD30EFRX-68EUZN0
    Serial Number : WD-WCC4N0842787
    Firmware Rev. : 80.00A80
    Disk Capacity : 3000.6GB
    Device State : NORMAL
    Timeout Count : 0
    Media Error Count : 0
    Device Temperature : 39 C
    SMART Read Error Rate : 200(51)
    SMART Spinup Time : 176(21)
    SMART Reallocation Count : 200(140)
    SMART Seek Error Rate : 100(0)
    SMART Spinup Retries : 100(0)
    SMART Calibration Retries : 100(0)


    Device Type : SATA(4433221102000000)
    Device Location : Enclosure#1 Slot#7
    Model Name : WDC WD30EFRX-68EUZN0
    Serial Number : WD-WMC4N1562580
    Firmware Rev. : 80.00A80
    Disk Capacity : 3000.6GB
    Device State : NORMAL
    Timeout Count : 0
    Media Error Count : 0
    Device Temperature : 39 C
    SMART Read Error Rate : 200(51)
    SMART Spinup Time : 190(21)
    SMART Reallocation Count : 200(140)
    SMART Seek Error Rate : 100(0)
    SMART Spinup Retries : 100(0)
    SMART Calibration Retries : 100(0)


    Device Type : SATA(4433221103000000)
    Device Location : Enclosure#1 Slot#8
    Model Name : WDC WD30EFRX-68EUZN0
    Serial Number : WD-WCC4N0816114
    Firmware Rev. : 80.00A80
    Disk Capacity : 3000.6GB
    Device State : NORMAL
    Timeout Count : 0
    Media Error Count : 0
    Device Temperature : 39 C
    SMART Read Error Rate : 200(51)
    SMART Spinup Time : 192(21)
    SMART Reallocation Count : 200(140)
    SMART Seek Error Rate : 200(0)
    SMART Spinup Retries : 100(0)
    SMART Calibration Retries : 100(0)




    The Hardware Monitor Information
    =====================================================
    [Controller H/W Monitor]
    CPU Temperature : 64 C
    Controller Temp. : 41 C
    12V : 12.220 V
    5V : 5.080 V
    3.3V : 3.344 V
    IO Voltage +1.8V : 1.792 V
    DDR3 +1.5V : 1.488 V
    CPU VCore +1.0V : 1.056 V
    Analog +1.0V : 0.992 V
    DDR3 +0.75V : 0.736 V
    Battery Status : Not Installed
    [Enclosure#1 : ARECA SAS RAID AdapterV1.0]
    Fan#1 : 1147 RPM
    Fan#2 : 1250 RPM
     
  15. prvt.donut macrumors 6502a

    prvt.donut

    Joined:
    Jan 1, 2008
    #15
    You are correct. Try to find a cheap data recovery company, maybe they can give you a decent quote.
     
  16. ITguy2016, Oct 22, 2016
    Last edited: Oct 22, 2016

    ITguy2016 Suspended

    Joined:
    May 25, 2016
    #16
    It all depends on the RAID implementation. Whether it would work or not can only be determined by trying to do it. However if you have important data on this RAID and are thinking of sending it to a data recovery company then I would recommend you not do anything with it. You don't want to make a bad situation worse by attempting repairs through other means. They're very knowledgeable and can most likely recover most if not all of your data.

    EDIT: My advice is merely meant as something to try as a last resort. If, at such time you've determined you have nothing to lose, then I can advise you how to close the disk with dd (or ddrescue) There are other tools you can use as well. Something like DataRescue's DD:

    http://www.datarescue.com/photorescue/v3/drdd.htm

    However I want to stress you should only try these methods (though they're not inherently risky but you don't know how the RAID controller with react to a cloned drive) if you've resigned yourself to losing all of the data on the RAID. IOW, you have nothing to lose.
     
  17. dpavid thread starter macrumors regular

    Joined:
    Feb 17, 2004
    Location:
    Mililani, Hawaii
    #17
    Thank you! I got a quote for $8000. Sound about right for a RAID5 with 24 TB?
     
  18. OS6-OSX macrumors 6502a

    OS6-OSX

    Joined:
    Jun 13, 2004
    Location:
    California
    #18
    I'm sure you are searching the web in between checking this thread. Out of curiosity I did the same and have probably come to the same conclusion. The internet has convincing arguments for what ever direction you want to head.

    1. Some say if 2 drives in RAID 5 fail you are done!
    2. Others claim to have gotten things back even with 2 drives down!
    3. Some took the professional data recovery route
    4. Some claimed to have fixed it with a controller firmware update
    5. A guy with an ATTO R360 card said ATTO tech support got him up and running. (Creative Cow)
    6. Most vowed to switch to RAID 6 or 10!

    This coming from a RAID 0 user!
    Saw one post where someone claimed to have spent $19.5 on recovery! WOW!
    (And yes that is 19K!)
     
  19. prvt.donut macrumors 6502a

    prvt.donut

    Joined:
    Jan 1, 2008
    #19
    I hate to say it but this yet again shows us RAID is not a backup. (Yes, I have co-workers who have told me that their servers do not need backups because they are RAID 1 *facepalm*).

    I recommend people to read these articles, the first was written way back in 2007 http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

    And this one in 2010 http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/.

    As I am sure everyone noticed, we are way past 2009, the main argument was that the chance an unreadable sector occurs in an array increases as drives get bigger and the single parity of RAID 5 will not be able to cope. The same argument is for RAID 6 and the cut off date is going to be around 2019. It is great that dpavid had most data backed up, but the fact that it all wasn't backed up on a daily basis (in a professional environment) is a ticking time bomb.

    This makes me wonder, what do all of you Mac based professionals do for enterprise grade storage and backups?
     
  20. ITguy2016 Suspended

    Joined:
    May 25, 2016
    #20
    I really couldn't say as I've never had to use a data recovery service.
    --- Post Merged, Oct 23, 2016 ---
    Ironically all of those solutions are probably correct. The protection offered by RAID is against drive failures and nothing more. It doesn't offer protection from a controller failure. It doesn't offer protection from a connection failure. It doesn't offer protection from a data corruption issue. Thus depending on the nature of the failure would depend on the solution.
     

Share This Page