[Disaster] Areca ARC-1880LP all RAID-5 volumes destroyed by Mac OS X 10.7 Lion

solaris8x86 said:

Doomsday man... my doomsday in my I.T. life..... 18TB data stored for 10 years now may long gone......

Click to expand...

I feel for you, I really do. I've lost large (for me at the time) RAID sets before.

That being said, RAID is not a substitute for backups or secondary copies. What would happen if your system were stolen or lost in a fire? Or if a ARECA bug happened to crash your data?

Worse yet, "bit rot" occurs on disks, and RAID will not discover or rebuild that for you. Only systems that store CRCs with data (such as high end disk arrays, or ZFS file system for example) can protect against bit rot.

Sitting with several TB of data that is not backed up is a disaster in itself.

Again, I'm not trying to be mean, and I do feel for you. Been there, done that, but not with irreplaceable data.

Wait, so you were using RAID as a backup?

Not trying to be mean but why would anyone run beta software on their production or primary computer.
If it were me I'd pile those drives into an external NAS or two.

I would add running a raid without backups to beta software on a mission critical workstation...

I'm truly sorry, that's horrible.

solaris8x86 said:

The most destructive Operating System - Mac OS X 10.7 Lion

Do NOT upgrade to 10.7 Lion without a proper RAID card driver installed

Click to expand...

I wouldn't blame the OS at this point especially since the drivers (Dec 2010) from Areca were obviously never tested on Lion. I think the warning should be Doesn't matter what the OS is.

Why on earth were you running a beta on a system with production data?

Where is that Picard facepalm...

solaris8x86 said:

Today, is a doomsday of my data on my Mac.

Click to expand...

Ouch.

As I and others have stated many times over, running a RAID, no matter how redundant it's designed to be, is foolish without a proper backup system. Life happens, and you're learning this the hard way.

Worse yet, is you tried out a Beta version of an OS. No IT dept. would do this, as it can totally bork a system, which you've discovered first hand.

Even with a backup system in place, getting it all back up and fully operational takes time, and that tends to translate into money (think of a commerce site, or worse, a bank that can't process a single transaction for days). Careers, and even entire companies, can be ruined by something like this.

solaris8x86 said:

Fucx me to trust HW RAID. Fucx me!

Click to expand...

Unfortunately, this was the result of very poor judgment on your part, not the hardware.

At least it should be a learning experience (should make you remember to never do it again).

Now given what's happened, let's assume for the moment the disk the card has indicated as Failed is fine. Removing it and replacing it may be enough to bring the array back online (it could begin a rebuild process). Another bit of hope (just in case it doesn't show up as healthy or begin to rebuild), is that Arecas keep a copy of the partition tables in the firmware, which allows them to recover some situations other cards cannot do (cards that don't keep copies of the PTs in it's ROM; just the disks). This requires a command to the card (it's a hidden feature), but Areca will walk you through this if necessary.

Now if the disk actually is bad, then you'll need to replace it. As the array is still operational (currently running in degraded mode), replacing the disk would cause the card to automatically begin rebuilding the array. I'd suggest letting it finish this process before trying to do anything else on the computer (keeps you from adding any further stress to the disks during the rebuild process). If this doesn't work, this is where the hidden recovery feature mentioned above would have to be attempted (last chance to get the data back).

so... what was on the drives?

DriveSavers

With this much data, though, selling a kidney probably wouldn't cover it.

solaris8x86 said:

Each RAID-set is having at least 1 disk crashed on RAID configuration shown in Acrea's configuration page; and all arrays are running in "Degraded" mode. If any one more disk is crash. All data would long gone.

18TB data stored for 10 years now may long gone......

Click to expand...

I'm not sure why nobody has recommended this yet... Get 9 2TB hard drives and start backing up!

I do feel for you. I've lost data twice before, not having backups in place. Some of it was very personal, important to me, and irreplaceable.

1) The first time I lost data, I had no back ups. That taught me to back up important data.

2) The second time I lost data, I had actually been backing it up. So I went to the backups only to find that not a damn one of them worked, so nothing could actually be recovered. So that taught me that I'm not really backing up unless I occasionally test the recovery process to verify it works.

We've all seen many stories about #1, but I've also seen a few about #2. TEST YOUR BACKUPS.

P.S. From my own experience as well as years of reading forums, I feel RAID is a bag of hurt. Yes there are some pros, but they come with a lot of cons as well, some of which are unacceptable to me.

Another thing to keep in mind is OS upgrades frequently kill disks that were on the verge. An OS upgrade is stressful. The boot disk is writing TONS of data, moving blocks, Spotlight indexes are being rebuilt, etc etc.

In IT I frequently saw disks die right after an upgrade/re-image. People would usually blame it on a bug in the new system, but honestly, it was always on older disks that were likely in borderline condition, and the stress of that I/O was enough to finally do them in.

I'm glad I did not upgrade to Lion seeing as it is not even out yet.
I crashed a SAN at work with Lion beta 1. That was the end of my "Hey this is super OK and stable" foolery. I feel your pain but no backups and beta software is pebkac of the highest order.
Telling people on a forum full of geeks takes some balls though.

solaris8x86 said:

do not upgrade to prerelease software without a proper backup. Otherwise, disaster like this may come true without warning.

Click to expand...

ftfy.

derbothaus said:

...I feel your pain but no backups and beta software is pebkac of the highest order.
Telling people on a forum full of geeks takes some balls though.

Click to expand...

Yeah he does have courage. Good luck with the salvage of data.

good to hear you saved it all. I once screwed up the filesystem on one drive and I was ready to commit suicide. I rescued it using a drive rescue in windows thankfully but I cant imagine that happening with 18tb! Im currently backing up to a 1.5tb drive in my system but I might get a 2tb TC and add a 3tb usb drive to it.

Good to hear you got the data back.

As foolish (not judging) as using beta software w/ a RAID setup man, I think there's a silver lining here, and that would be that the OP potentially saved potentially thousands of other users of the same potential fate - or worse - and it's one more bug that's being ironed out to ensure Lions release is proper.

Kinda mind blowing to know that the OP is helping Apple and the community on a beta software level getting Lion all ironed out you know like giving Apple an even better, more complete OS to be released, and on an issue that Apple may have not even looked at before.

But yeah I think it's always just risky/dumb move - and I've made plenty of dumb moves in my life etc. - to try pre-release software w/ sensitive data etc. I know I've done risky/dumb moves like that but that's why back ups are important lol


Let us know how the drivers work out yeah?

Glad you got it back. I'd stick with Snow Leopard till Areca gets their drivers straightened out and Lion is well out of beta. After all what's the rush?

Its stories like these that I show my friends who tell me I waste money on having so many backups. I'd be devastated if something happened to my data. I have so much invested in software its unreal.

Many, many moons ago, through my own idiocy I over-wrote the IPL volume (for those who know that that is) on a $10M IBM mainframe. I spent the next several hours restoring the volume through a rather painful process ... only to start the over-writing program all over again as I was finishing up.

After another multi-hour recovery, identical to the one I had just finished, I went for a long walk. Lord help us from ourselves.

Lesson here ... I had a backup.