A Reminder to Backup...

[MacRumors]

According to this ars thread (http://arstechnica.infopop.net/OpenTopic/page?a=tpc&s=50009562&f=8300945231&m=8700918145), it appears MacUpdate has suffered a hard drive failure.


MacUpdate's hard drive took a dive this morning and will no longer boot. This has caused unexpected down time for the MacUpdate site.

We are going to attempt to make the drive a secondary drive in the machine and copy the data over. If this doesn't work, we are in a very bad position, as the MacUpdate site and database are not backed up outside of this drive.

Keep us in your prayers if you would please.,

MacUpdate.com (http://www.macupdate.com) currently resolves to an Ensim server admin page.

[arn]

Not good news...

It looks like they are hosting with Rackshack.net - which has a notorious reputation for being cheap - but providing very little support.

During the server growth a few months back, I tried out a RackShack.net server and temporarily moved the forums over to it... but due to bad support and unexplainable failures, I canceled the service.

arn

[edesignuk]

Man, that's very bad. You would think any big website would have at least one good backup :rolleyes:

[Future Man]

Aren't they almost as big as versiontracker? Or am I totally off?

[arn]

Originally posted by Future Man
Aren't they almost as big as versiontracker? Or am I totally off?

they weren't quite as big as Versiontracker. I think versiontracker is one of the largest Mac-related sites out there.

arn

[whfsdude]

Ouch, they should just get an xserve heh :) send it to a local server farm :)

Edit: Old site if you didn't know - http://216.239.51.100/search?q=cache:xzbpor1iSfgC:www.macupdate.com/+mac+update&hl=en&ie=UTF-8

[medea]

uh gee a site like that which doesnt practice proper backing up of info, thats sad, hopefully they will pull it back together though and will learn a lesson from this. I hope something like this doesnt happen to macrumors, but Im sure arn wouldnt let that happen.;)

[arn]

Originally posted by medea
I hope something like this doesnt happen to macrumors, but Im sure arn wouldnt let that happen.;)

we've got nightly backups to a second drive. So, unless both drives fail simultaneously.... but I guess I should have a routine off-site backup plan. I'll setup something.

arn

[xelterran]

that is bad news :( i hope they dont go and host it on the same server again.. http://www.ipshosting.com/ is a good php host ive heard...

[idkew]

Originally posted by arn


we've got nightly backups to a second drive. So, unless both drives fail simultaneously.... but I guess I should have a routine off-site backup plan. I'll setup something.

arn

just curious arn- about how large is the database for MR now?

[arn]

Originally posted by idkew


just curious arn- about how large is the database for MR now?

About 420MB.

arn

[guifa]

According to index.html on MacUpdate right now, they were able to retrieve the data for the site and are currently copying it over to a new server.

I dunno about others, but I like the layout of MacRumors better than VersionTracker, it just seems cleaner. Maybe it's the color scheme.

[dricci]

I hope they get everything up and running again. I prefer MacUpdate over Version Tracker, much more friendly than VT. Also, I use them for my site's (http://www.pctomac.com/how/software.htm) Software Updates Syndication.

[Choppaface]

does anybody know if myacen is a good host? I think they do backups for you too...

Originally posted by arn


About 420MB.

arn

O_O_O_O_O_O_O_O

and yet searches and things go pretty fast...wow

[daRAT]

BTDT... I lost a lot of Bryce work that I did...

Solution, Firewire harddrive! Love it.

My condolences to Macupdate :(

[brossow]

Holy crap. That's just unbelievably stupid. $100 for a second drive + $50 for Retrospect = virtually no down time. Or skip Retrospect and use Carbon Copy Cloner in the wee hours of the morning every day to have virtually no down time. Or get a real server setup using RAID. In 2002, there's no excuse for more than a few minutes of downtime in any situation.

[Les Kern]

Originally posted by brossow
... there's no excuse for more than a few minutes of downtime in any situation.

But it happens. I feel for them. I have been doing this for almost 20 years, and in my surrent job I DO say to the users BACKUP,BACKUP,BACKUP AGAIN!. Some genius I am: Last year I lost 3 GB of files because the whole setup failed when I was transfering them to two external drives before I backed up to the server. For some reason my mac died and took two drives with it! $1,700.00 to DriveSavers later I had MOST of my files, and thankfull all of another user's. The lesson: Now I backup to 4 locations on FW drives and servers on alternating days AND to a FW portable daily for off-site. As a matter of fact this protocol is in place for most of the 23 servers. But know what? It can happen again. So pat them on the back and offer condolenses.

[Mr. Anderson]

Originally posted by arn


About 420MB.

arn

ha, that's it - all that spamming, too, who would have thought? Do you have a breakdown on the types of info? How much is imagery? Its amazing to think you can fit all of MacRumors on one CD :D

D

[Choppaface]

Originally posted by dukestreet
Do you have a breakdown on the types of info? How much is imagery?

I think he was only talking about the mysql database, which is all text, but i dunno

[rhett121]

So where did you find that 450GB harddrive for $100? And retrospect for $50, I don't guess you meant the professional version huh?

Maybe they were using IBM harddrives, I had 4 IBM "Deathstars" die on me unwarned, so my backup drive failed too. It happened in 2 sets of 2 so you can pretty much guarantee I will never buy another IBM drive. The 3 year warranty is not of any use when you still have to buy new drives to hold you over for the 3-5 weeks it takes to replace them.

I wish them the very best in recovery.

[brossow]

Originally posted by rhett121
So where did you find that 450GB harddrive for $100? And retrospect for $50, I don't guess you meant the professional version huh?

Maybe they were using IBM harddrives, I had 4 IBM "Deathstars" die on me unwarned, so my backup drive failed too. It happened in 2 sets of 2 so you can pretty much guarantee I will never buy another IBM drive. The 3 year warranty is not of any use when you still have to buy new drives to hold you over for the 3-5 weeks it takes to replace them.

I wish them the very best in recovery.

1. Why a 450GB drive? There's no WAY that MacUpdate was storing 450 gigs of data on their web server. Maybe you were mistakenly referring to the reported 450 MEGAbyte size of the MacRumors database mentioned elsewhere in this thread.

2. No, I wasn't talking about Retrospect Workgroup or Server. You don't need those for backing up a single drive; that would be extreme overkill. At $50, Retrospect Express would be adequate for the one drive and would have allowed an almost immediate return to functionality. Only way you'd need Workgroup is if the server was actually running OS X Server; lesser versions of Retrospect won't back up ASIP or OS X Server.

Whatever the case, this could easily have been averted for under $500. What's the downtime worth? Ultimately it's not my problem, but don't expect me to feel sympathetic. Sheesh.

[arn]

Originally posted by brossow

Whatever the case, this could easily have been averted for under $500. What's the downtime worth? Ultimately it's not my problem, but don't expect me to feel sympathetic. Sheesh.

It's actually not quite that easy...

Web server hosting tends to be more costly... and you can't just translate "one time" costs of a home machine over to a server.

You can co-locate a server... in that you can purchase a full machine and have it hosted at a location, but then you have to maintain it yourself, and the burden of software maintainence and hardware failures is on you. Plus, you're still paying hundreds of dollars a month on colocation/bandwidth.

Leasing a dedicated server is more managable... but hosting costs for dedicated servers start at $99 -- but once you start adding bandwidth and backup capacity, it quickly elevates. MacRumors hosting was close to $600-700/month at one point -- and that was without a backup solution at all.

Plus - many of these sites -- like MacUpdate are started as individually run sites rather than company funded sites. As bandwidth and traffic increases, costs tend to be coming out of pocket for many webmasters. MacMusic.org, for example, is planning on closing due to too much popularity... they can't cover their hosting costs. Adding another $100/month or so for a good backup system on top of losing money is not something that some people can do.

In the end - yes, "real" businesses should spend what it takes... but with websites, many can be individually funded sites which make additional expenses more problematic.

arn

[cubist]

1. Backups.
2. Backups.
3. Backups.
The way I interpret this is:
First, have a "live" backup in the form of a RAID or mirror drive (because hard drives inevitably fail; it is only a question of when).
Second, have a backup on a rotating tape or CD-RW media set which is produced and maintained on-site.
Third, have an off-site tape or CD-R (or WORM) backup set which is produced weekly and maintained off-site.
A site of such national strategic significance as MacRumors :cool: ought to have a "warm" off-site backup server ready to go, possibly at reduced bandwidth, should a total site outage occur.

In telephony, we believe in multiple levels of protection. At one company, we had two sites with enormous UPS systems consisting of a roomful of lead-acid batteries, plus diesel generators. The generators were tested every month. Suddenly a massive windstorm destroyed the electical supply to one of the two sites. The generator came on - and failed immediately. Backup tapes were produced as the systems ran on battery, the tapes were taken to the other site, and then the switches were moved to the backup commlinks to the other site. Double faults occur. Even triple faults can occur.

[brossow]

Originally posted by arn


It's actually not quite that easy...

Web server hosting tends to be more costly... and you can't just translate "one time" costs of a home machine over to a server.

You can co-locate a server... in that you can purchase a full machine and have it hosted at a location, but then you have to maintain it yourself, and the burden of software maintainence and hardware failures is on you. Plus, you're still paying hundreds of dollars a month on colocation/bandwidth.

Leasing a dedicated server is more managable... but hosting costs for dedicated servers start at $99 -- but once you start adding bandwidth and backup capacity, it quickly elevates. MacRumors hosting was close to $600-700/month at one point -- and that was without a backup solution at all.

Plus - many of these sites -- like MacUpdate are started as individually run sites rather than company funded sites. As bandwidth and traffic increases, costs tend to be coming out of pocket for many webmasters. MacMusic.org, for example, is planning on closing due to too much popularity... they can't cover their hosting costs. Adding another $100/month or so for a good backup system on top of losing money is not something that some people can do.

In the end - yes, "real" businesses should spend what it takes... but with websites, many can be individually funded sites which make additional expenses more problematic.

arn

No need to tell me about the costs involved with managing servers; it's part of what I do for a living. (I manage several servers -- more than six but fewer than a dozen -- in multiple locations. My budgets for the various locations are SEVERELY limited, yet in not one case would we lose more than a few hours' data and we'd have the affected server back up in minutes. Backup and recovery are simply part of the cost of having a server; it's not an option.) What I also know quite well is the cost of not properly backing up servers. Regardless of the cost of a proper backup system, what is the cost of rebuilding an entire site from scratch? From the limited information I've seen about this specific failure, an additional drive with some sort of mirroring would have saved hours of downtime and presumably considerable human time and effort, not to mention the bad image it projects when a simple drive failure takes down your service for an inordinate amount of time -- a bad image not just to your users but also to your advertisers. I don't know what their hosting situation is, but despite everything you've said, I don't think the downtime was necessary:

1. CoLo -- follow one of my previous suggestions and you're back up and running within five minutes of getting to the CoLo.

2. Leasing -- who would consider leasing without a backup solution in place? If you can't guarantee the security of all your time and labor, why bother?

Either way, if the business model doesn't include a provision for data backup and disaster recovery, there's a fundamental flaw in the plan. To me, this is the same thing as typing up your doctoral dissertation without ever saving the document, just leaving the computer running 24/7 until you're done and hoping that the power doesn't go out and that the computer doesn't crash before you can print it out. Sure, it might work for awhile but odds are eventually you're going to be sorry.

I won't belabor this further. I've said all I can say. It doesn't matter. For their sake, I hope the folks at MacUpdate learn something from this and come up with a better system.

[pantagruel]

ok I may still be pretty new here, but arn is kind of the top dog of macrumors and as a newbie man you probably shouldnt argue with him, not that your shouldnt be entitled to your own opinion because that is definitely not the case (i myself am a victim of this right now on a different thread) but maybe you should just word your replies more carefully, ive seen so far that arn is pretty smart and when he replies to a thread he knows what he is talking about.

[dricci]

Originally posted by arn
MacRumors hosting was close to $600-700/month at one point -- and that was without a backup solution at all.

Just wondering, arn, how much does it cost to run this site now? I'd imagine around expo time it'd be very costly!

[brossow]

Originally posted by pantagruel
ok I may still be pretty new here, but arn is kind of the top dog of macrumors and as a newbie man you probably shouldnt argue with him, not that your shouldnt be entitled to your own opinion because that is definitely not the case (i myself am a victim of this right now on a different thread) but maybe you should just word your replies more carefully, ive seen so far that arn is pretty smart and when he replies to a thread he knows what he is talking about.

Wow. I'm almost speechless. Because I'm recently registered here, I necessarily have less experience than arn and should bow to him? With all due respect to arn, who has provided an unquestionably masterful site, I don't think that's a fair assumption on your part. Talk about shortsighted! Ouch.

[rugby]

I know all about losing data. I've done it, twice. Our fileserver at work got flaky and while verifying the boot drive the Users RAID disappeared. I got it back, but all the resource forks were gone from all the files. I hadn't backed up that users partition in 2 weeks (I know, stupid). Now I have a tape drive backing up every night. And every weekend I use CCC to make an asr image just in case.

I also recently lost 80gb worth of info on my home computer. About 50gb worth of mp3's and 30gb worth of DV footage/video projects I was working on. NOT HAPPY. So now I'm re-importing all my cd's and even on a DP867 it's going slow.

lesson here, I'm backing up religiously. I'm backing up to dvd here at home because I don't have a tape drive. I also back up to a Mac OS X Server here at home that does nothing but serve files.

[mike3k]

After I had a problem like that with BounceWeb (which I use to host both MacMegasite & WorldBeatPlanet), I'm now running a script on my own machine to do nightly backups.

I have a cron job run at midnight that does a mysqldump on both site databases and saves it on my own machine, rotating the nightly backups to keep the last 5 days.

[BenderBot1138]

I kinda thought something wonky was going on that morning... any other news about its cause?

:cool:

[wdlove]

Thank you arn for your labor of love, hosting the Mac Forums. What is the total cost with backups for running Mac Forums. Thanks to your sponsors also, they must have a soft spot for Mac people!

I backup every other day, it has proved invaluable more than once, currently using an internal Zip 100.

[arn]

Originally posted by pantagruel
ok I may still be pretty new here, but arn is kind of the top dog of macrumors and as a newbie man you probably shouldnt argue with him, not that your shouldnt be entitled to your own opinion because that is definitely not the case (i myself am a victim of this right now on a different thread) but maybe you should just word your replies more carefully, ive seen so far that arn is pretty smart and when he replies to a thread he knows what he is talking about.

heh... while I'm flattered... everyone is certainly welcome to their opinion regardless of whether or not they agree with me.

I'm am certainly no expert in many areas... and web/server hosting is one of those areas I had to pick up on the go.

arn