[Disaster] Areca ARC-1880LP all RAID-5 volumes destroyed by Mac OS X 10.7 Lion

Discussion in 'Mac Pro' started by solaris8x86, Jun 21, 2011.

  1. solaris8x86, Jun 21, 2011
    Last edited: Jun 22, 2011

    solaris8x86 macrumors member

    Joined:
    Nov 24, 2007
    #1
    Today, is a doomsday of my data on my Mac.:(

    The most destructive Operating System - Mac OS X 10.7 Lion

    I got 3 x RAID-5 volumes (12 x 2TB) on my Mac Pro OS X 10.6 formed by a Areca ARC-1880LP HW RAID Controller and they were running fine. I did something stupid to upgrade my Mac to 10.7 Lion Developer Preview-4 beta build on last night for a trial run. After finishing installation. All of my existing RAID-5 volumes crashed. I fallback to 10.6 immediately and realized all of RAID-5 volumes are crash in a disaster manner. Snow Leopard prompts up after boot up: "To back up all your data on all disks immediately, and to reformat them asap", and tried that Disk Utility couldn't fix them. Each RAID-set is having at least 1 disk crashed on RAID configuration shown in Acrea's configuration page; and all arrays are running in "Degraded" mode. If any one more disk is crash. All data would long gone.

    I am now spending my best effort to rescue the RAID-sets with Areca Support. I am here to share this doom-feel experience that I have with all Mac Pro users. Do NOT upgrade to 10.7 Lion without a proper RAID card driver installed. Otherwise, disaster like this may come true without warning.

    Doomsday man... my doomsday in my I.T. life.....:( 18TB data stored for 10 years now may long gone......
     
  2. bplein macrumors 6502

    bplein

    Joined:
    Jul 21, 2007
    Location:
    Austin, TX USA
    #2

    I feel for you, I really do. I've lost large (for me at the time) RAID sets before.

    That being said, RAID is not a substitute for backups or secondary copies. What would happen if your system were stolen or lost in a fire? Or if a ARECA bug happened to crash your data?

    Worse yet, "bit rot" occurs on disks, and RAID will not discover or rebuild that for you. Only systems that store CRCs with data (such as high end disk arrays, or ZFS file system for example) can protect against bit rot.

    Sitting with several TB of data that is not backed up is a disaster in itself.

    Again, I'm not trying to be mean, and I do feel for you. Been there, done that, but not with irreplaceable data.
     
  3. solaris8x86 thread starter macrumors member

    Joined:
    Nov 24, 2007
    #3

    I would like to commit suiside today. Great depression. My mood is now N times poor than the economics situation of United State of America. Any prefer ways?:( (which way should I die for)
     
  4. DesmoPilot macrumors 65816

    Joined:
    Feb 18, 2008
  5. solaris8x86, Jun 21, 2011
    Last edited by a moderator: Jun 22, 2011

    solaris8x86 thread starter macrumors member

    Joined:
    Nov 24, 2007
    #5
    They are Primary data bank. Not all of them having a backup snapshot in Time Machine. My life is expendable. My data is not.:(

    **** me to trust HW RAID. **** me!
     
  6. blueroom macrumors 603

    blueroom

    Joined:
    Feb 15, 2009
    Location:
    Toronto, Canada
    #6
    Not trying to be mean but why would anyone run beta software on their production or primary computer.
    If it were me I'd pile those drives into an external NAS or two.
     
  7. fizzwinkus macrumors 6502a

    Joined:
    Jan 27, 2008
    #7
    I would add running a raid without backups to beta software on a mission critical workstation...

    I'm truly sorry, that's horrible. :(
     
  8. blueroom macrumors 603

    blueroom

    Joined:
    Feb 15, 2009
    Location:
    Toronto, Canada
    #8
    I wouldn't blame the OS at this point especially since the drivers (Dec 2010) from Areca were obviously never tested on Lion. I think the warning should be
    a RAID is not a backup.
    Doesn't matter what the OS is.
     
  9. goMac macrumors 603

    Joined:
    Apr 15, 2004
    #9
    Why on earth were you running a beta on a system with production data?

    Where is that Picard facepalm...
     
  10. nanofrog macrumors G4

    Joined:
    May 6, 2008
    #10
    Ouch. :(

    As I and others have stated many times over, running a RAID, no matter how redundant it's designed to be, is foolish without a proper backup system. Life happens, and you're learning this the hard way.

    Worse yet, is you tried out a Beta version of an OS. No IT dept. would do this, as it can totally bork a system, which you've discovered first hand.

    Even with a backup system in place, getting it all back up and fully operational takes time, and that tends to translate into money (think of a commerce site, or worse, a bank that can't process a single transaction for days). Careers, and even entire companies, can be ruined by something like this.

    Unfortunately, this was the result of very poor judgment on your part, not the hardware. :(

    At least it should be a learning experience (should make you remember to never do it again). ;)

    Now given what's happened, let's assume for the moment the disk the card has indicated as Failed is fine. Removing it and replacing it may be enough to bring the array back online (it could begin a rebuild process). Another bit of hope (just in case it doesn't show up as healthy or begin to rebuild), is that Arecas keep a copy of the partition tables in the firmware, which allows them to recover some situations other cards cannot do (cards that don't keep copies of the PTs in it's ROM; just the disks). This requires a command to the card (it's a hidden feature), but Areca will walk you through this if necessary.

    Now if the disk actually is bad, then you'll need to replace it. As the array is still operational (currently running in degraded mode), replacing the disk would cause the card to automatically begin rebuilding the array. I'd suggest letting it finish this process before trying to do anything else on the computer (keeps you from adding any further stress to the disks during the rebuild process). If this doesn't work, this is where the hidden recovery feature mentioned above would have to be attempted (last chance to get the data back).
     
  11. davidwarren macrumors 6502a

    Joined:
    Aug 28, 2007
  12. Moof1904 macrumors 65816

    Joined:
    May 20, 2004
    #12
    DriveSavers

    With this much data, though, selling a kidney probably wouldn't cover it.
     
  13. AlphaDogg macrumors 68040

    AlphaDogg

    Joined:
    May 20, 2010
    Location:
    Boulder, CO
    #13
    I'm not sure why nobody has recommended this yet... Get 9 2TB hard drives and start backing up!
     
  14. ActionableMango macrumors 604

    ActionableMango

    Joined:
    Sep 21, 2010
    #14
    I do feel for you. I've lost data twice before, not having backups in place. Some of it was very personal, important to me, and irreplaceable.

    1) The first time I lost data, I had no back ups. That taught me to back up important data.

    2) The second time I lost data, I had actually been backing it up. So I went to the backups only to find that not a damn one of them worked, so nothing could actually be recovered. So that taught me that I'm not really backing up unless I occasionally test the recovery process to verify it works.

    We've all seen many stories about #1, but I've also seen a few about #2. TEST YOUR BACKUPS.

    P.S. From my own experience as well as years of reading forums, I feel RAID is a bag of hurt. Yes there are some pros, but they come with a lot of cons as well, some of which are unacceptable to me.
     
  15. goMac macrumors 603

    Joined:
    Apr 15, 2004
    #15
    Another thing to keep in mind is OS upgrades frequently kill disks that were on the verge. An OS upgrade is stressful. The boot disk is writing TONS of data, moving blocks, Spotlight indexes are being rebuilt, etc etc.

    In IT I frequently saw disks die right after an upgrade/re-image. People would usually blame it on a bug in the new system, but honestly, it was always on older disks that were likely in borderline condition, and the stress of that I/O was enough to finally do them in.
     
  16. derbothaus macrumors 601

    derbothaus

    Joined:
    Jul 17, 2010
    #16
    I'm glad I did not upgrade to Lion seeing as it is not even out yet.
    I crashed a SAN at work with Lion beta 1. That was the end of my "Hey this is super OK and stable" foolery. I feel your pain but no backups and beta software is pebkac of the highest order.
    Telling people on a forum full of geeks takes some balls though.
     
  17. al2o3cr macrumors regular

    Joined:
    Oct 14, 2009
    #17
    ftfy.
     
  18. philipma1957 macrumors 603

    philipma1957

    Joined:
    Apr 13, 2010
    Location:
    Howell, New Jersey
    #18
    Yeah he does have courage. Good luck with the salvage of data.
     
  19. solaris8x86, Jun 22, 2011
    Last edited: Jun 22, 2011

    solaris8x86 thread starter macrumors member

    Joined:
    Nov 24, 2007
    #19
    After spending my best effort overnight. I have successfully rescued all data. Since all RAID-5 volumes are in "handicap-condition" but they are still allowing me to copy-out the data. All data are copied to another provisional storage. Then I used to recreate the RAID-5 diskgroup from scratch and restore the data back to the newly created RAID-5 disk groups and volumes. All are ok after a "Die Hard like" overnight rescue operation. Jesus christ!:D

    The conclusion is Mac OS X 10.7 Lion Developer Preview 4 will destroy all Areca RAID-5 volumes with its default build-in OS driver (Which means Areca RAID card is now bootable in Lion environment. If they are RAID-disks, Lion will split them off unexpectedly and breaking the RAID configuration in the hard drive sectors - The situation like mine). To rescue these RAID-5 volumes destroyed by Lion. All RAID-5 disk groups MUST BE RECREATED thru Areca's configuration page that to erase all damaged RAID information on the disks. So users who install the Lion with a Acrea RAID volumes attached during the installation process will suffer the same disaster like mine. This is a global disaster because many users have the same hardware configurations like mine and pending to upgrade to Lion. They will step on the same trap during the Lion's installation process. This is unavoidable because it is a build-in OS driver. Who proceed the OS installation. Who will be trapped. Damn it, Steve Job shouldn't distribute the disaster in the WWDC forum this month.:mad:

    I am now working with Areca official Support to test and report the bug to Apple priors to Lion release. Otherwise, million of users will suffer this disaster on next month. I am now testing Areca's latest beta driver for Lion (They just provide me this morning).:D

    I am reborn now...!!:apple:
     
  20. CrAkD macrumors 68040

    CrAkD

    Joined:
    Feb 15, 2010
    Location:
    Boston, MA
    #20
    good to hear you saved it all. I once screwed up the filesystem on one drive and I was ready to commit suicide. I rescued it using a drive rescue in windows thankfully but I cant imagine that happening with 18tb! Im currently backing up to a 1.5tb drive in my system but I might get a 2tb TC and add a 3tb usb drive to it.
     
  21. Tucom macrumors 65816

    Tucom

    Joined:
    Jul 29, 2006
    #21
    Good to hear you got the data back.

    As foolish (not judging) as using beta software w/ a RAID setup man, I think there's a silver lining here, and that would be that the OP potentially saved potentially thousands of other users of the same potential fate - or worse - and it's one more bug that's being ironed out to ensure Lions release is proper.

    Kinda mind blowing to know that the OP is helping Apple and the community on a beta software level getting Lion all ironed out you know like giving Apple an even better, more complete OS to be released, and on an issue that Apple may have not even looked at before.

    But yeah I think it's always just risky/dumb move - and I've made plenty of dumb moves in my life etc. - to try pre-release software w/ sensitive data etc. I know I've done risky/dumb moves like that but that's why back ups are important lol :D


    Let us know how the drivers work out yeah?
     
  22. solaris8x86, Jun 22, 2011
    Last edited: Jun 22, 2011

    solaris8x86 thread starter macrumors member

    Joined:
    Nov 24, 2007
    #22

    The latest Areca Beta Driver for Mac OS X 10.7 Lion (not yet available on their website) will be tested a couple hours later after I finish to restore all data into the newly created RAID-5 disk groups (18TB = 2TB x 12 HDDs = RAID-5 volumes x 3 (4 x HDDs per RAID-5 set)). It is top priority. After that, I will test the driver and report the bugs to Apple and Areca.:D

    But I do believe users who haven't see this thread will kick into the same trap. It is still a global scale disaster (Doomsday is still coming to look for you:D).
     
  23. blueroom macrumors 603

    blueroom

    Joined:
    Feb 15, 2009
    Location:
    Toronto, Canada
    #23
    Glad you got it back. I'd stick with Snow Leopard till Areca gets their drivers straightened out and Lion is well out of beta. After all what's the rush?
     
  24. chrono1081 macrumors 604

    chrono1081

    Joined:
    Jan 26, 2008
    Location:
    Isla Nublar
    #24
    Its stories like these that I show my friends who tell me I waste money on having so many backups. I'd be devastated if something happened to my data. I have so much invested in software its unreal.
     
  25. tamvly macrumors 6502a

    tamvly

    Joined:
    Nov 11, 2007
    #25
    Many, many moons ago, through my own idiocy I over-wrote the IPL volume (for those who know that that is) on a $10M IBM mainframe. I spent the next several hours restoring the volume through a rather painful process ... only to start the over-writing program all over again as I was finishing up.

    After another multi-hour recovery, identical to the one I had just finished, I went for a long walk. Lord help us from ourselves.

    Lesson here ... I had a backup.
     

Share This Page