Backing up large files or directories.

[Multipass]

I use Time Machine. In addition to the I backup stuff to external drives and store them offsite.

The problem is how long it takes to do that.

For example, I copy my iPhone library to the external drive, it's very large.

Then 2 weeks later I bring back the external drive and delete the old iPhoto library, which takes a while, and then copy the entire new library over.

Is there any other way to do it in which only the newer items would be added instead of having to delete the old version and copy over the entire new one?

 

[ChrisA]

....
Is there any other way to do it in which only the newer items would be added instead of having to delete the old version and copy over the entire new one?

Yes, "rsync". It's a 30 year old UNIX utility included on every Mac. It does exactly what you want. The trouble is that now days few people know how to type "rsync" at a terminal window. Good news for them is that there are a great many point and click apps that will allow you to run rsync without the need to remember how to type.

If you Google "mac rsync app" you get quite a few hits.

Actually you can do the same thing with Time Machine. TM will allow you to connect more than one drive. It two are connected it will alternate using them so each drive is used every other hour. Plug in a second TM drive. Let it make the first backup over night then unplug it and take it off site. Later bring it back and plug it in and TM will update it with changes. Then remove it This is easy to do and easy to restore files if yu have to.

The rsync based apps will use your existing back drive with no need to re-do the initial first time backup. Type "man rsync" in the terminal to display the very long manual page that explains how to use rsync or get one of the GIU front ends for it. Or just use TM.

 

[prisstratton]

Yep, have to agree with ChrisA, use RSYNC.

Watch this video on setting up a server, he explains how rsync works and provides practical working examples along with how to automate the process….very helpful.

https://www.youtube.com/watch?v=tG4Tge11GOo

 

[mojolicious]

I refuse to install iPhoto so can't check, but isn't the Library still a monolithic package file? If so, are you saying that rsync can update individual images contained within the Library, but Time Machine will only update the entire package?

 

[ApfelKuchen]

I refuse to install iPhoto so can't check, but isn't the Library still a monolithic package file? If so, are you saying that rsync can update individual images contained within the Library, but Time Machine will only update the entire package?

A package is not a file, it's a file system directory (i.e., a folder) with specific properties. Inside, it's full of regular old folders and files. Right-click/Control-click on a library icon, and select Show Package Contents. Voila!

A package presents absolutely no barrier to anything but the casual clumsiness of endusers. From Apple's Mac Developer Library File System Programming Guide:

Packages and bundles. Packages and bundles are directories that the Finder presents to the user as if they were files. Bundles hide the internal workings of executables such as apps and just present a single entity that can be moved around the file system easily. Similarly, packages allow apps to implement complex document formats consisting of multiple individual files while still presenting what appears to be a single document to the user

Time Machine does not backup an entire package when something changes inside the package. As with everything else Time Machine backs up, it saves only that which has changed within the library - individual files, and any directory lists (folders) that were modified. (And it's useful to remember that a folder is a metaphor - folders are directories/inventory lists, they are not physical containers.)

RSYNC and Time Machine "see" an iPhoto library in exactly the same way - as a file system directory. Both have a similar approach, "change only the things that have changed, leave the rest alone." The difference is purely in the end results. With RSYNC, if a file or directory has changed, the old version is deleted. Time Machine is designed to maintain a historical archive - it keeps old versions of directories and files as well as new versions.

 

[mojolicious]

RSYNC and Time Machine "see" an iPhoto library in exactly the same way - as a file system directory. Both have a similar approach, "change only the things that have changed, leave the rest alone."

I appreciate that the package contains discrete files, but was unsure how TM handled it. From what you say, in terms of the amount of data copied there's no difference between rsync and TM?

I think the OP needs to clarify what he's trying to achieve. The sentences "Then 2 weeks later I bring back the external drive and delete the old iPhoto library, which takes a while, and then copy the entire new library over. Is there any other way to do it in which only the newer items would be added instead of having to delete the old version and copy over the entire new one?" are rather confusing. Is he manually deleting his TM backup, and effectively using TM to produce a single 'snapshot' rather than to maintain multiple dated backups? Does he not want to take advantage of the 'time' bit of the Time Machine? Is he actually looking for SuperDuper or CarbonCopyCloner?

 

[Multipass]

I think the OP needs to clarify what he's trying to achieve. The sentences "Then 2 weeks later I bring back the external drive and delete the old iPhoto library, which takes a while, and then copy the entire new library over. Is there any other way to do it in which only the newer items would be added instead of having to delete the old version and copy over the entire new one?" are rather confusing. Is he manually deleting his TM backup,

No, this has nothing to do with Time Machine. I drag and drop important stuff to an eternal drive. Then when I want to update those backups I need to delete everything on that drive and drag and drop the new versions. I gave an example of iPhoto Library. When I put it on the external drive last week it was 30Gigs. Since then I added 2Gigs worth of pictures to it. Now this week I want to back it up again and put the 32Gig library on the external drive, which means I have to delete the 30Gig library and move the new 32Gig library to the drive. This is a waste of lots of time, I'd rather find a way to do it by only having to transfer the 2 Gigs. Some type of incremental backup, but Time Machine won't work because these drives are too small for the entire TM backup. I have the TM backup going to one large drive that is always connected.

 

[mojolicious]

Ah, gotcha. I think .

You're after something that allows incremental backup of specified files and folders, and aren't bothered about versioning because that's already being dealt with by Time Machine?

I know someone who uses Backuplist+ – basically a GUI front end for rsync – for achieving similar.

Personally I approach it the other way round, ie I use SuperDuper! for incremental backup of my entire disk, and Time Machine for versioned backup of selected files/folders. Horses for courses etc.

EDIT: because...
a) The SuperDuper! backup is bootable, so zero downtime in event of disaster
b) I've got more faith in SD! than TM as a restore source when I address above disaster
c) TMing my entire startup disk would involve a large Parallels image; haven't tried it for a while, but Parallels' 'optimise for Time Machine' setting didn't seem to do much, size-wise.

 

[Multipass]

Yeah, I really don't care about versioning. I always have the option with Time Machine if I find that I need it one day.

My main goal is to backup the important things to multiple inexpensive small drives for redundancy. Things like my iPhoto library as well as a directory filled with business stuff that is very important.

The problem is having to spend all the time deleting the old stuff and copying over the entire new version.

I am going to look into Backuplist and rsync, thanks.

 

[chown33]

Carbon Copy Cloner can also do incremental backups. It uses rsync to accomplish this.

 

[Multipass]

Carbon Copy Cloner can also do incremental backups. It uses rsync to accomplish this.

I thought that was more for cloning the drive than for doing individual file and folder backups?

 

[mojolicious]

You can include/exclude specific files and folders with CCC (SuperDuper only works with entire disks). CCC is feature-packed these days, as it should be given the price. It also allows you to retain rather than overwrite older versions, but from what I've read I'm not sure you can restore your entire disk to a point in time, à la Time Machine.

 

[ApfelKuchen]

I appreciate that the package contains discrete files, but was unsure how TM handled it. From what you say, in terms of the amount of data copied there's no difference between rsync and TM?

All things being equal, there's no substantive difference. In other words, if Time Machine backed up once a week and RSYNC was used once a week, then the amount of data copied would be, for all practical reasons, the same. Since Time Machine is backing up every hour, the amount of data handled during the course of a two-week period is going to be greater than the amount copied during a single, bi-weekly backup.

But this isn't really about bits and bytes, it's about time and space. Space-wise, if I had the OP's goals (on-site full backup using Time Machine, off-site backup of only the iPhoto library), I wouldn't let the size of my backup drives determine the solution - the cost difference between, say, a 32GB and 64GB flash drive (or 32 vs. 64GB of cloud storage) is too small to worry about. Time, on the other hand... I'd look to fully automate the off-site backup as well, and backup more frequently. Why wait around at all for a backup to finish? Also, I don't want to be screwed if I forget to run that bi-weekly backup, or inadvertently leave the backup drive at home. And once an automated solution is in place, there's no need to go two weeks between off-site backups.

According to their marketing materials (I don't use it, but lots of folks here do recommend it), Carbon Copy Cloner will do the job. It can do incremental backups of selected folders and files, and their current marketing materials show an example of exactly the use case we're talking about - backup of only the Pictures folder, and only of specific folders within Pictures. https://bombich.com/features)

 

[Multipass]

I'm thinking maybe it would be better to just use Crashplan or Backblaze to backup my important files and directories. Along with Time Machine for backing up everything and giving me versioning options.

 

[jdechko]

I'm thinking maybe it would be better to just use Crashplan or Backblaze to backup my important files and directories. Along with Time Machine for backing up everything and giving me versioning options.

If you do end up going with CrashPlan, you could replace TimeMachine as well. The same interface will allow you to back up locally to an external drive and to the CrashPlan cloud service. You can also back up to another computer that you (or a friend/ family member) own. The same software handles the backup scheduling, so you don't have to worry about both TM and CP trying to run backups at the same time.

However, CrashPlan does not offer a bootable backup the way that CCC does and you can't do a straight system restore like you can with TM. So in addition to CrashPlan local & cloud backups, about once every few months I do a clone with CCC, and I skip TM completely.

 

[Multipass]

To be honest, I don't really want a bootable backup or a full restore. I actually like fresh installations, I just want to make sure I have up-to-date backups of my important files. Crashplan seems like it would work well for this.

 

[jdechko]

To be honest, I don't really want a bootable backup or a full restore. I actually like fresh installations, I just want to make sure I have up-to-date backups of my important files. Crashplan seems like it would work well for this.

It does work pretty well. And even though I use CrashPlan, Backblaze is also a really good service. I think both of them have free trials, so you can try one or both and make a decision. Regardless of which service you choose, having a local backup is also nice because it's much faster to restore a lot of files. I look at online backup like an insurance policy. You pay a small fee every month and you hope you never have to use it. But it's always there in case of an emergency.

 

[Multipass]

It sounds like Crashplan would work better for local backups than Time Machine, just like you said earlier. I don't like the idea of having to restore -everything- after a catastrophic failure like Time Machine makes you. I'd rather just drag and drop the important files into a newly installed OS.

 

[Multipass]

I set up Crashplan to back up to an external drive. It seems to be the same as what Time Machine offers, only better because I could restore individual files instead of having to do everything like Time Machine makes you.

I'm curious why more people don't use Crashplan instead of Time Machine, since the local backup version is free.

I see that Crashplan is defaulted to backup every 15 minutes, I changed that to an hour. I figure that's good enough?

 

[chown33]

I set up Crashplan to back up to an external drive. It seems to be the same as what Time Machine offers, only better because I could restore individual files instead of having to do everything like Time Machine makes you.

When I've used Time Machine to restore files, I've always done it for specific files, or for specific folders of files. I've never used it for a "restore everything".

I don't know where you get the idea that TM requires you to restore everything. Maybe you just didn't know it could do that. See here for info, under the headings "Restoring specific files or folders" and "Restoring and reverting within apps":
https://support.apple.com/en-us/HT201250

You can find more info with the google search terms: restore files from time machine.

If you've tried the actions for restoring specific files or folders and it didn't work for some reason, that would be a bug or problem with TM, or maybe the backup, or maybe with how you tried it. But that doesn't mean TM is only capable of restoring everything.

 

[Multipass]

When I've used Time Machine to restore files, I've always done it for specific files, or for specific folders of files. I've never used it for a "restore everything".

I don't know where you get the idea that TM requires you to restore everything. Maybe you just didn't know it could do that. See here for info, under the headings "Restoring specific files or folders" and "Restoring and reverting within apps":
https://support.apple.com/en-us/HT201250

You can find more info with the google search terms: restore files from time machine.

If you've tried the actions for restoring specific files or folders and it didn't work for some reason, that would be a bug or problem with TM, or maybe the backup, or maybe with how you tried it. But that doesn't mean TM is only capable of restoring everything.

I was told in this thread (as well as in other places that I read about it after making that thread) that if I had a catastrophic failure (or bought a new computer) and had to do a new installation, Time Machine would give me the option to restore the apps, system files, and user data. But I wouldn't be able to pick and choose anything from those groups. It would be all the user data or none of it.

If I did have a failure or if I did get a new computer, I would be happy with the fresh installation. All I would want to do is copy over some files such as my Documents folder and my iPhoto Library. That's my goal. I'd rather not try to "migrate" over all kinds of stuff from the old installation.

 

[chown33]

I was told in this thread (as well as in other places that I read about it after making that thread) that if I had a catastrophic failure (or bought a new computer) and had to do a new installation, Time Machine would give me the option to restore the apps, system files, and user data. But I wouldn't be able to pick and choose anything from those groups. It would be all the user data or none of it.

If I did have a failure or if I did get a new computer, I would be happy with the fresh installation. All I would want to do is copy over some files such as my Documents folder and my iPhoto Library. That's my goal. I'd rather not try to "migrate" over all kinds of stuff from the old installation.

That's how it works in the "restore after catastrophe" case. However, that's not the only way TM can be used. For example, you could create a new user account, without restoring anything at that time. You could then go into Time Machine and selectively restore the specific things you want.

I'm not suggesting that you have to use TM for doing backups. You may well be happier with other processes. I'm only saying that it's inaccurate, or at best incomplete, to say that TM requires a complete restore. It doesn't. How you use the TM backup is up to you. The choices of what you can do will depend on the point where you access the TM backup. But selective restore is certainly possible with TM.

If you really want to find out how TM works, I suggest using it to make a backup, then try doing a restore from it. You can try different scenarios, such as restoring into a completely new account (System Preferences > Users & Groups), and see what the options are. You should also select what TM backs up in the first place, again so you can see how that works.

Regardless of what you use (TM or something else, I strongly recommend running restore procedures before they're actually needed, so you can see if there are problems. If there are, you want to discover those when the data isn't critical, rather than when it is. Think of it as a fire drill or a lifeboat drill on a ship. You want to know how to save yourself before you need to do it for real.

 

[Multipass]

That's how it works in the "restore after catastrophe" case. However, that's not the only way TM can be used. For example, you could create a new user account, without restoring anything at that time. You could then go into Time Machine and selectively restore the specific things you want.

I'm not suggesting that you have to use TM for doing backups. You may well be happier with other processes. I'm only saying that it's inaccurate, or at best incomplete, to say that TM requires a complete restore. It doesn't. How you use the TM backup is up to you. The choices of what you can do will depend on the point where you access the TM backup. But selective restore is certainly possible with TM.

Everything else I read seems to contradict what you just said.

Even if you are correct, it still seems like I would need to do some type of "work-around" with Time Machine in order to get what I want.

With the free version of Crashplan (saving to a local HD instead of on their cloud is free), I can do everything that Time Machine does as well as being able to just drag and drop the exact files/folders I need after a new installation. Seems like a win-win situation for me.

 

[chown33]

Everything else I read seems to contradict what you just said.

You'll have to point to specific things you read.

Even if you are correct, it still seems like I would need to do some type of "work-around" with Time Machine in order to get what I want.

In this article:
https://support.apple.com/en-us/HT204350

Under the heading "Migrate from a Time Machine backup or external drive", item 8 says:

8. You can customize the type of information that is migrated on the next screen. If you want to transfer only a portion of an account, deselect anything you don't want migrated. If there is more than one user on your original Mac, you can click the triangle next to a user's icon and deselect anything you don't want to migrate for specific users.
​

Note the underlined.

The screenshot for "Select the Information to Transfer", shows a "John Appleseed" account. It has an expansion triangle next to it, implying that you can expand it and then select/deselect portions thereunder.

This is one reason I suggested actually trying it. This will show you exactly what is selectable and what isn't. You can test Migration Assistant by using it when you make a new test account.

With the free version of Crashplan (saving to a local HD instead of on their cloud is free), I can do everything that Time Machine does as well as being able to just drag and drop the exact files/folders I need after a new installation. Seems like a win-win situation for me.

Then by all means, use it.


My only point in all this is that it's inaccurate to say that TM requires one to do a complete restore.

 

[Multipass]

You could pick apart everything I say all that you want. It's not going to change the fact that Time Machine won't let me pick only a few individual files when restoring to a new system.