Trying to make a simple backup folders/files shell script

Discussion in 'macOS' started by ahrenba, Oct 13, 2009.

  1. ahrenba macrumors member

    Joined:
    Feb 17, 2008
    #1
    Hey guys,

    I just have about 10 folders to backup and have been trying to make a shell script that I can run and backup the folders to my external.

    Right now I am just using the cp command to:

    cp -r /folder1 /folder2 /folder3 /backuplocation

    I realize this is a very crude way to backup files, and I am wondering if there is a better/more efficient way to do this? rsync? ditto?

    Also, is there a way to have the script print it's progress so far? Kind of like a percentage or estimated time left in the terminal window?

    You are probably wondering why I just don't use a backup program, but I am using this a learning experience, too.

    Thanks!
     
  2. mysterytramp macrumors 65816

    mysterytramp

    Joined:
    Jul 17, 2008
    Location:
    Maryland
    #2
    Way back when I did something like this with AppleScript ... I used a TextEdit window to serve as a "progress" indicator, sort of like a console.

    mt
     
  3. ahrenba thread starter macrumors member

    Joined:
    Feb 17, 2008
    #3
    Ok, cool, was it very simple and did you recall how you did it?
     
  4. angelwatt Moderator emeritus

    angelwatt

    Joined:
    Aug 16, 2005
    Location:
    USA
    #4
    I use rsync for some of my smaller backups. I use CCC for my large backup tasks.
     
  5. ahrenba thread starter macrumors member

    Joined:
    Feb 17, 2008
    #5
    After I posted this I started playing with rsync and I like it a lot. However, I can't seem to figure out the --exclude attribute.

    This works:
    rsync -avE --progress --exclude='Podcasts/' /Users/MYUSERNAME/Music /Volumes/EXTERNAL_HARD_DRIVE/Backup/

    But I want to narrow it down to only the Podcasts folder in the iTunes Music folder. So I tried everything:

    rsync -avE --progress --exclude='/iTunes/iTunes\ Music/Podcasts/' /Users/MYUSERNAME/Music /Volumes/EXTERNAL_HARD_DRIVE/Backup/

    That doesn't work. Any ideas here?
     
  6. angelwatt Moderator emeritus

    angelwatt

    Joined:
    Aug 16, 2005
    Location:
    USA
    #6
    The exclude is relative to the folder you're backing up. I think the first / is messing it up. I don't know exactly how your folder structure is setup, but you may need it like,
    Code:
    rsync -avE --progress --exclude='/Users/MYUSERNAME/Music/iTunes/iTunes Music/Podcasts/' /Users/MYUSERNAME/Music/Volumes/EXTERNAL_HARD_DRIVE/Backup/
    Note: you only need a \ in front of a space when you're not using quotes around the path.
     
  7. ahrenba thread starter macrumors member

    Joined:
    Feb 17, 2008
    #7
    Wow, I am so stupid. Here is the command that worked:

    Code:
    rsync -avE --progress --exclude='iTunes/iTunes Music/Podcasts/' /Users/MYUSERNAME/Music /Volumes/EXTERNAL_HARD_DRIVE/Backup/
    It was the damn escape tag that I was using inside the ' ' . I wasn't aware that I didn't need it there. Thanks for the help.
     
  8. ahrenba thread starter macrumors member

    Joined:
    Feb 17, 2008
    #8
    Ok, we have a new problem.

    Code:
    rsync -avE --progress --exclude='Cool/Complete/' --exclude='Virtual Machines/' --exclude='test/' /Users/USERNAME/Documents /Volumes/HARDDRIVE/BACKUP/
    In this statement, rsync recognizes that I want to exclude the folder called Complete and the folder called test . For some reason it is not picking up that I want to exclude the folder called Virtual Machines .

    My only guess is because Virtual Machines is a folder in the base of /Users/USERNAME/Documents (what I am backing up) and not further down like /Users/USERNAME/Documents/Cool/Complete was.

    Any ideas on this? I've tried everything.

    EDIT: Ok, the only way I can get this to work is by doing the following. However, it's not what I want. If I use this:
    --exclude='Virtual Machines'

    It will exclude the folder called Virtual Machines. However, this ignores all Virtual Machines folders across the whole backup. So if I had another Virtual Machines folder in, say, /Users/USERNAME/Documents/test/ that would be ignored to. I want to specify the exact folder to exclude.

    EDIT 2:

    I found out how, but I don't understand the solution.

    In my previous example, rsync knew it was in the Music folder already, so I just had to:
    --exclude='iTunes/iTunes Music/Podcasts/'

    To make this Documents backup work I had to:
    --exclude='Documents/Virtual Machines/'

    See what happened there? I had to specify a level higher this time (Documents). I don't understand why because before I didn't have to specify the folder that I was in (the one I was backing up [Music in the first case, Documents in the second case])

    Any ideas on this? Explanations?
     
  9. angelwatt Moderator emeritus

    angelwatt

    Joined:
    Aug 16, 2005
    Location:
    USA
    #9
  10. ahrenba thread starter macrumors member

    Joined:
    Feb 17, 2008
    #10
    Thanks for your help. Unfortunately, I've tried running it in my home directory and in other locations, no luck. :(

    Yeah, I was planning to make an exclude file once I figured out the basics, but I am just running some tests with doing multiple --exclude in the line.

    What I can't seem to figure out is here is the command I used for my Music backup:

    Code:
    rsync -avE --progress --exclude='[COLOR="Blue"]iTunes[/COLOR]/iTunes Music/Podcasts/' /Users/MYUSERNAME/[COLOR="DarkGreen"]Music[/COLOR] /Volumes/EXTERNAL_HARD_DRIVE/Backup/
    Here is the command I used for my Documents backup:

    Code:
    rsync -avE --progress --exclude='[COLOR="Blue"]Cool[/COLOR]/Complete/' --exclude='[COLOR="DarkGreen"]Documents[/COLOR]/Virtual Machines/' --exclude='test/' /Users/USERNAME/[COLOR="DarkGreen"]Documents[/COLOR] /Volumes/HARDDRIVE/BACKUP/
    According to the man page and other articles I have read, the rsync command relates all --excludes to the directory that is being backed up. For instance, in the first example, using an exclude of:

    Code:
    --exclude='[COLOR="Blue"]iTunes[/COLOR]/iTunes Music/Podcasts/'
    will, in actuality exclude the directory:

    Code:
    /Users/USERNAME/Documents/[COLOR="Blue"]iTunes[/COLOR]/iTunes Music/Podcasts/
    It builds the file link based on the directory that is being backed up. This worked great for the first example.

    However, in the second example, this "Virtual Machines" folder is in the actual backup folder, not levels deeper. For some reason, I had to include the directory being backed up (Documents) before I put Virtual Machines, making the whole exclude Documents/Virtual Machines .

    According to everything I have read, I should have just had to do a command like --exclude='/Virtual Machines/' and it should have worked.

    Here is a link to the man page. Let me know if anyone figures this out. Thanks.
     
  11. ahrenba thread starter macrumors member

    Joined:
    Feb 17, 2008
    #11
    Hmmmm........... Well I feel stupid. :/

    It appears I forgot the / at the end of the source that tells rsync it's a folder.

    Here's what I had:
    Code:
    rsync -avE --progress --exclude='Cool/Complete/' --exclude='Documents/Virtual Machines/' --exclude='test/' /Users/USERNAME/Documents /Volumes/HARDDRIVE/BACKUP/
    Here's the correct command that made it work:
    Code:
    rsync -avE --progress --exclude='Cool/Complete/' --exclude='/Virtual Machines/' --exclude='test/' /Users/USERNAME/Documents/ /Volumes/HARDDRIVE/BACKUP/Documents
    Notice the / after Documents in the specified source. Notice how the Documents/ before Virtual Machines is no longer needed now. Notice I also needed to specify the folder on the receiving side too.

    All that hard work and it was simple....I guess that's the way it normally works. :/

    However, one last minor problem. I can't figure out how to specify a single folder.

    If I do:
    --exclude='folder1/'
    It will exclude all folder1's no matter how deep they are. Same goes for:
    --exclude='/folder1/'

    Anyone have any ideas on how to specify individual folders?
     
  12. ahrenba thread starter macrumors member

    Joined:
    Feb 17, 2008
    #12
    Ok let me clarify this as easily as I can.

    If you use a trailing slash in your source folder, let's say you are backing up:

    /Users/name/files

    If you put /Users/name/files in the source of the rsync command, you HAVE to include the directory name in the --excludes.

    --exclude='files/folder_to_exclude/'

    If you put /Users/name/files/ (with that trailing slash) in the source of the rsync command, you ONLY need to use a slash in the --excludes.

    --exclude='/folder_to_exclude/'

    This command excludes ANY folder called folder_to_exclude anywhere in the backup:

    --exclude='folder_to_exclude/'

    See how that works. That took me TOOO long to figure out, but that's the way it works.
     
  13. ahrenba thread starter macrumors member

    Joined:
    Feb 17, 2008
    #13
    One more final explanation (from the man page):

    a trailing slash on the source changes this behavior to transfer all files
    from the directory src/bar on the machine foo into the /data/tmp/.
    A trailing / on a source name means "copy the contents of this directory".
    Without a trailing slash it means "copy the directory".
    This difference becomes particularly important when using the --delete option.


    Can someone explain to me the significance of the --delete attribute in reference to if there is a trailing slash or not?
     
  14. VS007 macrumors member

    Joined:
    Apr 12, 2008
    #14

Share This Page