Toggle sidebar Toggle sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

Trying to make a simple backup folders/files shell script

A

ahrenba

macrumors member
Original poster
Hey guys,

I just have about 10 folders to backup and have been trying to make a shell script that I can run and backup the folders to my external.

Right now I am just using the cp command to:

cp -r /folder1 /folder2 /folder3 /backuplocation

I realize this is a very crude way to backup files, and I am wondering if there is a better/more efficient way to do this? rsync? ditto?

Also, is there a way to have the script print it's progress so far? Kind of like a percentage or estimated time left in the terminal window?

You are probably wondering why I just don't use a backup program, but I am using this a learning experience, too.

Thanks!
 
Way back when I did something like this with AppleScript ... I used a TextEdit window to serve as a "progress" indicator, sort of like a console.

mt
 
angelwatt said:
I use rsync for some of my smaller backups. I use CCC for my large backup tasks.
Click to expand...

After I posted this I started playing with rsync and I like it a lot. However, I can't seem to figure out the --exclude attribute.

This works:
rsync -avE --progress --exclude='Podcasts/' /Users/MYUSERNAME/Music /Volumes/EXTERNAL_HARD_DRIVE/Backup/

But I want to narrow it down to only the Podcasts folder in the iTunes Music folder. So I tried everything:

rsync -avE --progress --exclude='/iTunes/iTunes\ Music/Podcasts/' /Users/MYUSERNAME/Music /Volumes/EXTERNAL_HARD_DRIVE/Backup/

That doesn't work. Any ideas here?
 
The exclude is relative to the folder you're backing up. I think the first / is messing it up. I don't know exactly how your folder structure is setup, but you may need it like,
Code: 
rsync -avE --progress --exclude='/Users/MYUSERNAME/Music/iTunes/iTunes Music/Podcasts/' /Users/MYUSERNAME/Music/Volumes/EXTERNAL_HARD_DRIVE/Backup/
Note: you only need a \ in front of a space when you're not using quotes around the path.
 
angelwatt said:
The exclude is relative to the folder you're backing up. I think the first / is messing it up. I don't know exactly how your folder structure is setup, but you may need it like,
Code: 
rsync -avE --progress --exclude='/Users/MYUSERNAME/Music/iTunes/iTunes Music/Podcasts/' /Users/MYUSERNAME/Music/Volumes/EXTERNAL_HARD_DRIVE/Backup/
Note: you only need a \ in front of a space when you're not using quotes around the path.
Click to expand...

Wow, I am so stupid. Here is the command that worked:

Code: 
rsync -avE --progress --exclude='iTunes/iTunes Music/Podcasts/' /Users/MYUSERNAME/Music /Volumes/EXTERNAL_HARD_DRIVE/Backup/
It was the damn escape tag that I was using inside the ' ' . I wasn't aware that I didn't need it there. Thanks for the help.
 
Ok, we have a new problem.

Code: 
rsync -avE --progress --exclude='Cool/Complete/' --exclude='Virtual Machines/' --exclude='test/' /Users/USERNAME/Documents /Volumes/HARDDRIVE/BACKUP/

In this statement, rsync recognizes that I want to exclude the folder called Complete and the folder called test . For some reason it is not picking up that I want to exclude the folder called Virtual Machines .

My only guess is because Virtual Machines is a folder in the base of /Users/USERNAME/Documents (what I am backing up) and not further down like /Users/USERNAME/Documents/Cool/Complete was.

Any ideas on this? I've tried everything.

EDIT: Ok, the only way I can get this to work is by doing the following. However, it's not what I want. If I use this:
--exclude='Virtual Machines'

It will exclude the folder called Virtual Machines. However, this ignores all Virtual Machines folders across the whole backup. So if I had another Virtual Machines folder in, say, /Users/USERNAME/Documents/test/ that would be ignored to. I want to specify the exact folder to exclude.

EDIT 2:

I found out how, but I don't understand the solution.

In my previous example, rsync knew it was in the Music folder already, so I just had to:
--exclude='iTunes/iTunes Music/Podcasts/'

To make this Documents backup work I had to:
--exclude='Documents/Virtual Machines/'

See what happened there? I had to specify a level higher this time (Documents). I don't understand why because before I didn't have to specify the folder that I was in (the one I was backing up [Music in the first case, Documents in the second case])

Any ideas on this? Explanations?
 
angelwatt said:
When you run the command are you in your home directory, or you Documents folder? You might see where I'm going with that question so you can see if that's the issue here.

Since you're excluding multiple items, you may want to keep your exclude list in a separate file and just note that file using an --exclude-from option.
Click to expand...

Thanks for your help. Unfortunately, I've tried running it in my home directory and in other locations, no luck. 🙁

Yeah, I was planning to make an exclude file once I figured out the basics, but I am just running some tests with doing multiple --exclude in the line.

What I can't seem to figure out is here is the command I used for my Music backup:

Code: 
rsync -avE --progress --exclude='[COLOR="Blue"]iTunes[/COLOR]/iTunes Music/Podcasts/' /Users/MYUSERNAME/[COLOR="DarkGreen"]Music[/COLOR] /Volumes/EXTERNAL_HARD_DRIVE/Backup/

Here is the command I used for my Documents backup:

Code: 
rsync -avE --progress --exclude='[COLOR="Blue"]Cool[/COLOR]/Complete/' --exclude='[COLOR="DarkGreen"]Documents[/COLOR]/Virtual Machines/' --exclude='test/' /Users/USERNAME/[COLOR="DarkGreen"]Documents[/COLOR] /Volumes/HARDDRIVE/BACKUP/

According to the man page and other articles I have read, the rsync command relates all --excludes to the directory that is being backed up. For instance, in the first example, using an exclude of:

Code: 
--exclude='[COLOR="Blue"]iTunes[/COLOR]/iTunes Music/Podcasts/'

will, in actuality exclude the directory:

Code: 
/Users/USERNAME/Documents/[COLOR="Blue"]iTunes[/COLOR]/iTunes Music/Podcasts/
It builds the file link based on the directory that is being backed up. This worked great for the first example.

However, in the second example, this "Virtual Machines" folder is in the actual backup folder, not levels deeper. For some reason, I had to include the directory being backed up (Documents) before I put Virtual Machines, making the whole exclude Documents/Virtual Machines .

According to everything I have read, I should have just had to do a command like --exclude='/Virtual Machines/' and it should have worked.

Here is a link to the man page. Let me know if anyone figures this out. Thanks.
 
Hmmmm........... Well I feel stupid. :/

It appears I forgot the / at the end of the source that tells rsync it's a folder.

Here's what I had:
Code: 
rsync -avE --progress --exclude='Cool/Complete/' --exclude='Documents/Virtual Machines/' --exclude='test/' /Users/USERNAME/Documents /Volumes/HARDDRIVE/BACKUP/

Here's the correct command that made it work:
Code: 
rsync -avE --progress --exclude='Cool/Complete/' --exclude='/Virtual Machines/' --exclude='test/' /Users/USERNAME/Documents/ /Volumes/HARDDRIVE/BACKUP/Documents

Notice the / after Documents in the specified source. Notice how the Documents/ before Virtual Machines is no longer needed now. Notice I also needed to specify the folder on the receiving side too.

All that hard work and it was simple....I guess that's the way it normally works. :/

However, one last minor problem. I can't figure out how to specify a single folder.

If I do:
--exclude='folder1/'
It will exclude all folder1's no matter how deep they are. Same goes for:
--exclude='/folder1/'

Anyone have any ideas on how to specify individual folders?
 
Ok let me clarify this as easily as I can.

If you use a trailing slash in your source folder, let's say you are backing up:

/Users/name/files

If you put /Users/name/files in the source of the rsync command, you HAVE to include the directory name in the --excludes.

--exclude='files/folder_to_exclude/'

If you put /Users/name/files/ (with that trailing slash) in the source of the rsync command, you ONLY need to use a slash in the --excludes.

--exclude='/folder_to_exclude/'

This command excludes ANY folder called folder_to_exclude anywhere in the backup:

--exclude='folder_to_exclude/'

See how that works. That took me TOOO long to figure out, but that's the way it works.
 
One more final explanation (from the man page):

a trailing slash on the source changes this behavior to transfer all files
from the directory src/bar on the machine foo into the /data/tmp/.
A trailing / on a source name means "copy the contents of this directory".
Without a trailing slash it means "copy the directory".
This difference becomes particularly important when using the --delete option.

Can someone explain to me the significance of the --delete attribute in reference to if there is a trailing slash or not?
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back