OS X Server + Time machine = SCAM!!!

[handheldgames]

I thought going from a history of over priced windows servers to OS X Server would be a walk in paradise... What a walk in fantasy land!!

I made some adjustments to DNS this morning, making the server unusable... then decided to rely upon time machine to bring me back before the adjustments were made...

What a mistake trusting Apple on this one.. Ive run into a cascading failure of all my web services, from wiki, to blog, to ical.. Anything that relies upon apache after the restore is basically dead. Even my custom SVN repository is dead

HUH?!?! In my history... I've never seen a windows server PUKE and roll over like this...... I know this is a rant... but WTF?? Time machine is supposed to be the be all end all of backups and its turning out to be a huge pile os useless code....

My server is dead. Great.. Just wonderful...

---------------------------------------------------------------
updated from post below:
---------------------------------------------------------------

Luckily I had a mirror of SL Server on a drive that started to generate SMART errors... Was able to restore everything with SuperDuper.

Lets have this as a lesson to everyone. Time Machine on SL Server is broken and should not be relied upon for disaster recovery of your operating system.

Perhaps it works in the desktop world - but it is not even close to being in a shippable state. It is clearly untested and anyone who relies upon this technology, will surely get a rude awakening...

 

[SilentPanda]

I'm not sure how that's a "scam". But anyway...

Did you do a selective file restore or a full restore from Time Machine?

 

[belvdr]

I thought Time Machine was just a good backup for the user data. If that's true, I would have thought you would have needed to backup the entire filesystem.

 

[handheldgames]

I did a full restore from Time Machine.... Have tried several backups - going from the most recent to the oldest with the same effects.

Perhaps it has something to do with the fact that my ServiceData is on a separate partition?

In my "limited" understanding of backing up a server with TM. If something goes wrong, we should be able to go back to an earlier version without incident. Hence the idea of backups, and the 2TB drive I have sitting there for this "Oh crap!" moment...

My Kerio Email server is happy as a clam.. It's just the apple bungled components that are whining....

I'd be more than happy to just reinstall, but don't want to loose my mail server in the mix...

 

[belvdr]

I did a full restore from Time Machine.... Have tried several backups - going from the most recent to the oldest with the same effects.

Perhaps it has something to do with the fact that my ServiceData is on a separate partition?

In my "limited" understanding of backing up a server with TM. If something goes wrong, we should be able to go back to an earlier version without incident. Hence the idea of backups, and the 2TB drive I have sitting there for this "Oh crap!" moment...

My Kerio Email server is happy as a clam.. It's just the apple bungled components that are whining....

I'd be more than happy to just reinstall, but don't want to loose my mail server in the mix...

When browsing the Time Machine, do you see that second partition?

 

[handheldgames]

When browsing the Time Machine, do you see that second partition?

Well there are two steps... 1st... I had to boot from the Install CD to recover the Server Installation. It would not allow me to re install the server data with that step.

Once restored, I was able to bring that data back through TM or by browsing the TM disk... Both resulted with the correct data going back to where it belongs.

All the data is in the right place... It's just generating a TON of errors in the console every 10 seconds. Kind of annoying....

1st step - even though I am not running Apples Mail Server, I had to run through the wizard to disable CLAMAV, as the log file its looking for is no longer available. This caused an error every second or so.

2nd step - I had to shut down Wiki, Blog, Calendar, etc, from Web Services as they generated about 15 errors every 10 seconds.

3rd step - my svn server on port 443 is no longer known and cannot be accessed and generated erroneous errors and also had to be shut down....

By shutting down all web related services, my server is happy... But I just lost a lot of user data and my svn repository...

Have been doing Google searches on the console messages. It appears that there are other SL Server folks out there that are doing the same thing.. Restoring a SL Server with TM and hitting the same brick wall...

Shall we say... 40Billion in the bank and not enough QA to support the products??? Hmmmmmmmm k!

 

[Consultant]

Got screenshots of the errors?

 

[Koinu]

Time Machine does not save/restore Apache Folders

I have seen a similar problem trying to do a full restore from Time Machine. The local web services would not work at all. If you check the Console, there are many messages about certain Apache directories (or log files) not being present. If I recall correctly, one of the key directories is /var/log/apache2.

Why this directory would be created and delivered with a full / new installation of Leopard but not properly backed up and restored by a full Time Machine image is a bit of a mystery. It's clear that TM does not do a complete image of the disk (and this is well documented), but it does seem rather poor (even if documented) that a complete restore does not make the machine operable. I can accept the fact that certain cached files would be lost, and I can even understand the notion of doing a fresh install BEFORE doing the TM restore, but when there is an option to "Restore from TM Backup", it should restore the system to a working state.

Try re-creating the /var/log/apache2 directory -- this should allow the local httpd services to restart. This may not be your only problem, but might get you one step closer.

Regards,
Scott Hurd

 

[handheldgames]

Got screenshots of the errors?

Here is the output that appears immediately after enabling the service....

12/13/09 12:54:26 AM servermgrd[98] servermgr_web: htcleancache disabled
12/13/09 12:54:26 AM servermgrd[98] servermgr_web: Multi instance key 'SSLLog' has a single value=''!
12/13/09 12:54:26 AM servermgrd[98] servermgr_web: Multi instance key 'SpotlightIndexing' has a single value='0'!
12/13/09 12:54:26 AM servermgrd[98] servermgr_web: Multi instance key 'PerfCache' has a single value='1'!
12/13/09 12:54:26 AM servermgrd[98] servermgr_web: Multi instance key 'SSLLog' has a single value=''!
12/13/09 12:54:26 AM servermgrd[98] servermgr_web: Multi instance key 'SpotlightIndexing' has a single value='0'!
12/13/09 12:54:26 AM servermgrd[98] servermgr_web: Multi instance key 'PerfCache' has a single value='1'!
12/13/09 12:54:26 AM servermgrd[98] servermgr_web: Disabling service com.apple.wikid
12/13/09 12:54:26 AM servermgrd[98] servermgr_web: Enabling service com.apple.wikid
12/13/09 12:54:26 AM com.apple.wikid[8777] /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/zope/__init__.py:1: UserWarning: Module twisted was already imported from /usr/share/caldavd/lib/python/twisted/__init__.pyc, but /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python is being added to sys.path
12/13/09 12:54:26 AM com.apple.wikid[8777] __import__('pkg_resources').declare_namespace(__name__)
12/13/09 12:54:27 AM com.apple.wikid[8777] /usr/share/caldavd/lib/python/twisted/mail/smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the email package instead
12/13/09 12:54:27 AM com.apple.wikid[8777] import MimeWriter, tempfile, rfc822
12/13/09 12:54:27 AM com.apple.wikid[8777] /usr/share/caldavd/lib/python/calendarserver/tap/caldav.py:49: DeprecationWarning: mktap and related support modules are deprecated as of Twisted 8.0. Use Twisted Application Plugins with the 'twistd' command directly, as described in 'Writing a Twisted Application Plugin for twistd' chapter of the Developer Guide.
12/13/09 12:54:27 AM com.apple.wikid[8777] from twisted.scripts.mktap import getid
12/13/09 12:54:28 AM com.apple.wikid[8777] Traceback (most recent call last):
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/bin/twistd", line 19, in <module>
12/13/09 12:54:28 AM com.apple.wikid[8777] run()
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/scripts/twistd.py", line 27, in run
12/13/09 12:54:28 AM com.apple.wikid[8777] app.run(runApp, ServerOptions)
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/application/app.py", line 694, in run
12/13/09 12:54:28 AM com.apple.wikid[8777] runApp(config)
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/scripts/twistd.py", line 23, in runApp
12/13/09 12:54:28 AM com.apple.wikid[8777] _SomeApplicationRunner(config).run()
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/application/app.py", line 423, in run
12/13/09 12:54:28 AM com.apple.wikid[8777] self.logger.start(None)
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/application/app.py", line 251, in start
12/13/09 12:54:28 AM com.apple.wikid[8777] observer = self._getLogObserver()
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/scripts/_twistd_unix.py", line 140, in _getLogObserver
12/13/09 12:54:28 AM com.apple.wikid[8777] logFile = logfile.LogFile.fromFullPath(self._logfilename)
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/python/logfile.py", line 47, in fromFullPath
12/13/09 12:54:28 AM com.apple.wikid[8777] os.path.dirname(logPath), *args, **kwargs)
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/python/logfile.py", line 147, in __init__
12/13/09 12:54:28 AM com.apple.wikid[8777] BaseLogFile.__init__(self, name, directory, defaultMode)
12/13/09 12:54:28 AM com.apple.wikid[8777] File "/usr/share/caldavd/lib/python/twisted/python/logfile.py", line 32, in __init__
12/13/09 12:54:28 AM com.apple.wikid[8777] assert os.path.isdir(self.directory)
12/13/09 12:54:28 AM com.apple.wikid[8777] AssertionError
12/13/09 12:54:28 AM com.apple.launchd[1] (com.apple.wikid[8777]) Exited with exit code: 1
12/13/09 12:54:28 AM com.apple.launchd[1] (com.apple.wikid) Throttling respawn: Will start in 9 seconds
12/13/09 12:54:29 AM servermgrd[98] servermgr_web: htcleancache disabled
12/13/09 12:54:29 AM servermgrd[98] servermgr_web: Multi instance key 'SSLLog' has a single value=''!
12/13/09 12:54:29 AM servermgrd[98] servermgr_web: Multi instance key 'SpotlightIndexing' has a single value='0'!
12/13/09 12:54:29 AM servermgrd[98] servermgr_web: Multi instance key 'PerfCache' has a single value='1'!
12/13/09 12:54:29 AM servermgrd[98] servermgr_web: Multi instance key 'SSLLog' has a single value=''!
12/13/09 12:54:29 AM servermgrd[98] servermgr_web: Multi instance key 'SpotlightIndexing' has a single value='0'!
12/13/09 12:54:29 AM servermgrd[98] servermgr_web: Multi instance key 'PerfCache' has a single value='1'!
12/13/09 12:54:29 AM servermgrd[98] servermgr_web: Disabling service com.apple.wikid
12/13/09 12:54:29 AM com.apple.ReportCrash.Root[8784] 2009-12-13 00:54:29.892 ReportCrash[8784:2a03] Saved crash report for httpd[8783] version ??? (???) to /Library/Logs/DiagnosticReports/httpd_2009-12-13-005429_localhost.crash

 

[handheldgames]

Luckily I had a mirror of SL Server on a drive that started to generate SMART errors... Was able to restore everything with SuperDuper.

Lets have this as a lesson to everyone. Time Machine on SL Server is broken and should not be relied upon for disaster recovery of your operating system.

Perhaps it works in the desktop world - but it is not even close to being in a shippable state. It is clearly untested and anyone who relies upon this technology, will surely get a rude awakening...

 

[pianoman181]

If you don't know how to properly administer and backup a server, you probably shouldn't be doing it..

 

[Cromulent]

Perhaps it has something to do with the fact that my ServiceData is on a separate partition?

Of course. If you didn't tell Time Machine to back it up then it wouldn't have done.

Time Machine really shouldn't be relied upon for mission critical backups though. Use something like rsync for that.

 

[handheldgames]

If you don't know how to properly administer and backup a server, you probably shouldn't be doing it..

Have been running/administering windows servers for 11 years. This is my 1st OS X server. It's been an interesting trip to say the least.

My fault is that I relied on clearly untested Apple technologies - that should work as they are marketed and integrated into the operating system.

If Time Machine is not intended to recover SL Server, that feature should not be available for use by end users. It's a shame that 3rd party apps can survive a time machine recover, but the Apple bundled web technologies don't.

 

[chown33]

If Time Machine is not intended to recover SL Server, that feature should not be available for use by end users. It's a shame that 3rd party apps can survive a time machine recover, but the Apple bundled web technologies don't.

File a bug-report with Apple:
http://developer.apple.com/bugreporter/

 

[RedTomato]

I use TM all the time. I also work with servers (Windows, not OS X, but am no expert).

I only rely on TM for user data, or client macs. I wouldn't rely on TM for backing up the server. It seems me to be pretty obvious that TM does not back up the subtle settings that servers rely on.

If I ever had to back up an OSX server, I'd use something like CCC or SuperDuper.

Accurate server-grade backing up of OSX is much more complex than it appears - see

http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/

and

http://blog.plasticsfuture.org/2006/04/23/mac-backup-software-harmful/

and some updates at

http://blog.plasticsfuture.org/

The original articles are from 3 years ago, but they were very widely read, and it seems the situation is not much better today.

 

[Deanster]

Sorry you got sucked in by the marketing.

Time Machine does NOT make complete backups. It's best for user-data or other similar items. There's no reason NOT to use it on a server, but given how inter-connected all the bits of a *nix server are, it seems a bit much to hope for that all the links, etc will back up when you restore just one section.

For a server, a minimum approach, IMHO, is mirrored or Raid 5 primary (system and data, if they're on separate drives/partitions) drives, an external (USB/Firewire/whatever) drive that gets the primaries cloned (SuperDuper or CCC - I like SuperDuper a LOT!) to it daily, and taken off-site, and a connected internal or external drive that gets the primaries cloned once a week. An additional bonus would be to have an additional drive for both backups so one offsite drive stays offsite, and so the weekly backup has both a one-week and two-week old version.

All this assumes you're doing something with the server where $300 in backup drives is cheaper than losing/restoring/etc. your data.

I DO agree that Apple doesn't do a good job of clarifying Time Machine's operation with OS X Server. Same with some of their other 'consumer' oriented tech that exists on the Server version of the software. Spotlight can be a problem with a server also.

My final thought is that you broke the #1 rule - TEST YOUR BACKUP/RESTORE!!!! having a backup is only the first half of preserving data. Successfully restoring it is the other half. You must test your backup/restore plan, and make sure it actually works both before you commit to using it for real data, and on an ongoing basis. It happens ALL the time that a backup solution doesn't quite perform as advertised, and you need to know that before doing a critical restore. This is sysadmin 101, IMHO.

 

[angelneo]

I'm stumped by why you chosen Time Machine as your backup software for your server. It has always been position as a consumer software for your everyday users with emphasis on ease of use. If you are just backing up raw data, it's still understandable, but why rely on it for all those intricate server settings?

 

[RedTomato]

It happens ALL the time that a backup solution doesn't quite perform as advertised, and you need to know that before doing a critical restore.

Agreed. If I had a quid for every time a server-grade backup solution has failed on me, I'd have enough for a nice posh sandwich from Marks and Sparks.

No need to be rude to the OP. Making mistakes like this is all part of the long hard path to becoming a server admin. I've made mistakes like this, we all have.

 

[Silas1066]

Let me tell you a little about Windows going down ...

Earlier this year, I put an Exchange Server 2007 into my AD environment at work. Everything seemed to install fine--then the problems started.

Turns out, the install did complete 100% (there were some errors that didn't show up). You would think, o.k., no big deal right? Wrong.

Active Directory was corrupt to the core, all over the network. I had to use ADSI edit and several other tools to repair dozens of user accounts. Countless registry changes, patches, etc. The Exchange Server was down for a week, and it took 3 weeks to fix the domain.

This problem went to 3rd level MS support who told me I was SOL (I ultimately fixed the problems on my own).

So count your blessings--I've never seen anything that catastrophic happen to an OSX system or network.

 

[codeus]

You performed a restore on a production server with no known good backup then call it a 'Scam' when it fails?

always always ALWAYS test your backup before making major changes or performing a restore, just because it says 'Backup Successful' doesn't mean TimeMachine will have to explain the problem to your customers when it goes titsup - you will.

And don't use TimeMachine for critical server backups, use Carbon Copy Cloner.

But whatever you use, make sure you know it is working before you start pushing buttons.

 

[RedTomato]

Servers are hard. ***** happens. Expensive commercial server-grade software regularly fails on servers.

Same as on your home computer.

The only difference is at home it just annoys you. At work it affects the working lives of tens to thousands of people.