question: 'Consistency' of 'time machine' backups

[tkermit]

I hope someone is able to answer this question:

How does Time machine guarantee a consistent backup of your files, when it comes to libraries (e.g. iPhoto, iTunes etc.) or index files in general ?


What I have been thinking about is that, especially during its first backup, the backup process can take quite a while. During this process it can very well be, that some of the files are changed again.

Does the very first 'time machine'-backup on a machine represent the state of the file system at the beginning or at the end of that backup - or (hopefully not) does it represent the state of each file at the exact moment that it is backed up ?


Wouldn't that mean that, if for example you're installing something big during a time machine backup, you may end up with a backup in which this program is "half-installed" ?

Also, what about libraries, where you have an actual index of the files, and also the files themselves (e.g. the iTunes library) - what happens if Time machine has already backed up the index file and is now backing up the individual files, but at this point you delete some of them => the index changes, but may have already been backed up - and is therefore outdated.

I do know about FSEvents and some of its inner workings, I just don't know whether time machine uses that information during its backups...


I want to believe Apple have thought this through - but I'd really love to get a deeper understanding of this. It also doesn't inspire confidence that Apple themselves warn against backing up Aperture libraries due to consistency issues...

Thanks in advance, guys !

[tkermit]

did some experiments....it seems to check for recorded FSEvents that happen during backup and continues backing up until it has arrived at a consistent state.... still don't feel all that safe about it - can only hope iPhoto & iTunes, mail etc. update their index files instantly...

edit: I spoke too soon .... Backups started during an MS Office installation all happily finish while files are still being copied - which in my mind means that I end up with a corrupt / inconsistent state of my files ?! Well - guess there's no other way to do this - it probably stops as soon as the FSEvents queue is empty even though there may still be copy operations going on...

[cmaier]

did some experiments....it seems to check for recorded FSEvents that happen during backup and continues backing up until it has arrived at a consistent state.... still don't feel all that safe about it - can only hope iPhoto & iTunes, mail etc. update their index files instantly...

edit: I spoke too soon .... Backups started during an MS Office installation all happily finish while files are still being copied - which in my mind means that I end up with a corrupt / inconsistent state of my files ?! Well - guess there's no other way to do this - it probably stops as soon as the FSEvents queue is empty even though there may still be copy operation going on...

Since it probably only keeps a single "last run" time/date, it probably gets a list of all directory changes from FSEvents since that last run, and copies those files, so that any change that occurs after querying FSEvents won't be caught. However, since FSEvents operates on the directory level, if you make more changes to a directory after FSEvents is queried and before TM analyzes the directory, it might catch them.

[THX1139]

I haven't used Time Machine (yet), but I can't help wonder why anyone would be doing major system modifications while TM is busy backing up. Isn't there a way to set TM to do backups when your not working on the computer? So you wouldn't have a absolutely current backup as it would only be valid for the last time TM updated, but I'd rather lose a day of backups than have a drive full of corrupt or incomplete files.

Does TM work that every time you make a change, it creates a backup? If so, seems like it would be a resource hog. I'd be more interested in having a solid backup at the end of the day. Maybe Superduper is better?

[tkermit]

...any change that occurs after querying FSEvents won't be caught...

:o sure hope you're wrong about that... (I take it you mean that FSEvents is only queried at the beginning of the backup)

As I see it, during backup, when 'Time machine' has copied everything according to the list of directories that it got from querying FSEvents, it queries FSEvents again and again until at some point an empty list is returned... ?!

[kbmb]

Does TM work that every time you make a change, it creates a backup? If so, seems like it would be a resource hog. I'd be more interested in having a solid backup at the end of the day. Maybe Superduper is better?

Time Machine itself runs every hour to do hourly backups. So basically after the initial full backup, it will run every hour.

Here's some details from the Ars Technica review:
http://arstechnica.com/reviews/os/mac-os-x-10-5.ars/14

So far I'm been happy with the performance. The hourly backups only do the changes detected, usually smaller amounts of data.

I'd imagine the only time you would start to see performance hits would be if TM had to copy many GBs of data over.

-Kevin

[cmaier]

:o sure hope you're wrong about that... (I take it you mean that FSEvents is only queried at the beginning of the backup)

As I see it, during backup, when 'Time machine' has copied everything according to the list of directories that it got from querying FSEvents, it queries FSEvents again and again until at some point an empty list is returned... ?!

Yes, I believe it is queried only at the beginning, and only once. If it repeatedly queried after handling all directories, it could end up thrashing to the point where it never completes.