Need advice/feedback on my backup solution (rsync)?

[MaKin211]

Hello guys,

I am working on a backup solution for two MacBooks to seperate partitions on a windows server storage. Time Machine does not work for me due to SMB and Mavericks, so had to look further and found rSync to be „the one solution“ for me. Unfortunately I still have some questions I could not find answers to by myself. Please apologize when my questions have been answered here already but I could not find any answers that would help me.

I am planning to do incremental backups every evening (triggered by an calendar event) for which I am using rsync 3.1.0 with sudo in order to solve the „Permission Denied (13) issue“. To skip the password prompt when running it in a shell script I was planning to edit the sudoer file and add a line in which I am allowed to run this script (and just that one) without password prompt. Does that work?

In the following you can see the rsync command I am currently using:

sudo rsync -a --info=progress2 /* /Volumes/name/Backup --exclude ".DS_Store" --exclude ".fseventsd" --exclude ".Spotlight-V100" --exclude ".TemporaryItems" --exclude ".Trashes" --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/Volumes/*,/media/*,/lost+found}

I am not sure about -z though. I know it stands for compression and should be turned on when doing rsync through network but when trying it out on my external hd (usb 3) it cut down the transfer speed.

When doing a backup I got this line during the progress:

76,279,379,222  61%   17.76GB/s    1:12:02  xfr#68035, ir-chk=6219/904515

Is my understanding correct?

ir-chk: the incremental scan is running and the file list is not yet completed but the transfer has already begun?
6219/904515: 904515 files are on that file list at the moment and 6219 have to be checked (whether they have to be updated) yet?
76,279,379,222: about 76GB of data have been already transferred (or checked)?

I hope some of you will have mercy on me and help.

I know we are talking about millions of files here but although I had my macbook connected via ethernet (Gigabit) it took almost 6 hours for the backup to be completed. And when running the rsync command a second time to see how long it takes to run the incremental scan it took about 90 minutes. Is that alright? And will the incremental scan take longer via WiFi?​

Should I add more folders/files to the exclusion list?​

Is the partition format of my backup partition on the windows 8 server (NTFS, default cluster size) an issue?​

Will I be able to use the backups to restore my system in case of a total crash? I know I won’t be able to simply „restore on install“ like I would be when using Time Machine, but still…​

Do you have any advice on what I could improve? My last „clean installation“ of Mac OS X was after I bought my beloved MBPr (about 2 years ago) so is there any chance that a new installation would solve the „too many files to backup“-issue?

I am looking forward to hearing from you. Thanks in advance!

 

[MaKin211]

It took 4 hrs to run it a second time via WiFi. And I dont know why but it seems that there were 3GB transfered again although I did not change anything...

rsync: mknod "/Volumes/Martin_Storage/Backup/private/var/folders/b_/pl2cy7yj3qb4512cxpqdg8n40000gn/T/icssuis501" failed: Operation not supported (45)

...

3,410,016,470   1%  217.94kB/s    4:14:39 (xfr#1791, to-chk=0/1681366)    
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0]

Any idea? I cannot run daily backups for 4hrs... And what does the error above mean?

 

[mfram]

It means that your backups aren't really working. You may be "backing up" your data, but you aren't doing it in a way that will allow it to be restored in any reasonable way. You will be losing all filesystem permissions and any attributes associated with the files by trying to send them over raw SMB. Backing up data with no real way to restore it is pretty much worthless.

Your only hope is to create a sparse disk image file on the remote server and mount it on the Mac. You'd have to format it HFS+. Then backup to the mounted drive. This is all what Time Machine would be doing in a supported configuration.

The other alternative is to do what you're already doing, but only for your own home directory. It still won't be perfect, but might get you back some of your data later. You certainly won't be able to restore any data without re-installing the operating system first anyway. And you won't get any Apps you installed on the system.

 

[MaKin211]

Thank you very much for your answer mfram! I really appreciate your help!

It means that your backups aren't really working. You may be "backing up" your data, but you aren't doing it in a way that will allow it to be restored in any reasonable way.

You will be losing all filesystem permissions and any attributes associated with the files by trying to send them over raw SMB. Backing up data with no real way to restore it is pretty much worthless.

I see the point. I thought through "sudo" they will persist.​

Your only hope is to create a sparse disk image file on the remote server and mount it on the Mac. You'd have to format it HFS+. Then backup to the mounted drive. This is all what Time Machine would be doing in a supported configuration.

Can you point me in the right direction on where to find more information on how to do that? Are there any "disadvantages" on that side? With Time-Machine I often had the problem that sometimes the backup (including its history) went corrupt and therefore was deleted...​

The other alternative is to do what you're already doing, but only for your own home directory. It still won't be perfect, but might get you back some of your data later. You certainly won't be able to restore any data without re-installing the operating system first anyway. And you won't get any Apps you installed on the system.

Actually this was my only hope. To be able to restore my Apps including its data. Indeed, a backup that cannot be restored does not make any sense. Thank you very much for enlightening me! ​

Maybe you could help me out a little on how to properly backup my data (being able to restore it later) through my environment (windows nas server with dedicated partition for the backup)?

Another point that I really do not understand is: there are so many articles, threads, tutorials on "how to do a full backup of your Mac System with rsync" including commands similar to mine and nobody has ever mentioned that these solutions are rather useless. How come? Therefore again, thank you very much!

 

[MaKin211]

No ideas?

 

[mfram]

Depends on what your goal is. What do you want your restore scenario to look like? Make sure you test it. What it comes down to is this: the only supported (by Apple) way to backup your Mac over a network is with a Time Capsule. Anything else would have to be supported by some other software.

Doing backups straight over SMB to a Windows Server is probably the least attractive backup choice for a Mac. SMB doesn't provide native Mac functionality. It's not clear to me how you'd ever restore that data in any reasonable way. You'd have to experiment.

Like I said in the previous post, the only reasonable choice over SMB to a Windows server is creating a native sparsebundle image using the Mac Disk Utility. Then you'd format the sparsebundle to HFS+ and use rsync to that. But you'd likely still have to restore manually. You might be able to get Migration Assistant to pull data from it. Not sure.

Let's say your hard drive died on the Mac. You'd have to replace the hard drive and re-install Mac OS from scratch. Now you have a fresh install, but none of your data. You'd have to restore that manually. That's going to be tricky.

Compare to other choices:

1. Time Capsule. You re-install OS on new drive. Install runs Migration Assistant to copy all your settings, documents, and applications from the backup.

2. External hard drive with Time Machine. Same as #1 but faster.

3. Software like Carbon Copy Cleaner. You clone your Mac hard drive to an external hard drive for backups. The backups are in the native Mac filesystem. Restore is a matter of booting off the backup and restoring to the new internal hard drive.

4. Some other Mac backup software? I'm sure there are other choices. Don't know what they are.

I use a Time Capsule and have used it to migrate my data from old Macs to new Macs. I know it works.

Maybe someone else has an idea.

TL;DR version:

Backing your Mac up to a Windows server is unsupported and does not provide a satisfactory restore experience. I wouldn't bother unless you copy over your Documents every once in a while.

 

[MaKin211]

Depends on what your goal is. What do you want your restore scenario to look like? Make sure you test it. What it comes down to is this: the only supported (by Apple) way to backup your Mac over a network is with a Time Capsule. Anything else would have to be supported by some other software.

All I want is to be able to restore my private and app data, it does not have to be a simple "click here to restore from your backup" - solution. But I do not want have all my app settings reset.

Doing backups straight over SMB to a Windows Server is probably the least attractive backup choice for a Mac. SMB doesn't provide native Mac functionality. It's not clear to me how you'd ever restore that data in any reasonable way. You'd have to experiment.

Again, I cannot understand why everyone calls "rsync to network drive via smb" a valid time machine backup alternative then. What you say makes perfect sense!

Like I said in the previous post, the only reasonable choice over SMB to a Windows server is creating a native sparsebundle image using the Mac Disk Utility. Then you'd format the sparsebundle to HFS+ and use rsync to that. But you'd likely still have to restore manually. You might be able to get Migration Assistant to pull data from it. Not sure.

What do you mean by saying "you'd have to restore manually"? Automatically means choosing the "install from backup" function right?

Let's say your hard drive died on the Mac. You'd have to replace the hard drive and re-install Mac OS from scratch. Now you have a fresh install, but none of your data. You'd have to restore that manually. That's going to be tricky.

That is what I was planning actually. I did not know that permissions will get lost and therefore kind of corrupted. I thought the main advantage of rsync was the raw transfer of files without any restrictions of reading them (like with time machine).

Compare to other choices:

1. Time Capsule. You re-install OS on new drive. Install runs Migration Assistant to copy all your settings, documents, and applications from the backup.

That sounds great sure, but I have already invested in server hardware and I am not planning on buying "another server" as long as I do not need to.


2. External hard drive with Time Machine. Same as #1 but faster.

The easiest solution, sure. But I'd like to have the freedom of being able to backup in background wirelessly.

3. Software like Carbon Copy Cleaner. You clone your Mac hard drive to an external hard drive for backups. The backups are in the native Mac filesystem. Restore is a matter of booting off the backup and restoring to the new internal hard drive.

As far as I know CCC is also using rsync. So what is CCC doing differently?

4. Some other Mac backup software? I'm sure there are other choices. Don't know what they are.

I use a Time Capsule and have used it to migrate my data from old Macs to new Macs. I know it works.

Maybe someone else has an idea.

TL;DR version:

Backing your Mac up to a Windows server is unsupported and does not provide a satisfactory restore experience. I wouldn't bother unless you copy over your Documents every once in a while.

The TL;DR version was not necessary as I really appreciate your answers and read every bit (twice).

What about the following solutions:

-Installing a VirtualMachine with FreeNAS for TM Backups? (will this work?)
-Formatting the Windows Partition in HFS+ in order to transfer via rsync with persisted permissions?
-Sparse Image on the NAS partition (as you mentioned)?

Thank you very much!

 

[mfram]

CCC works because it's writing to a native Mac filesystem on a direct attached drive.

You can't put HFS+ on the drive on a Windows system. Then Windows couldn't read it. Besides, that's not relevant to the issue. The issue is that SMB doesn't support native Mac functionality.

Looks like someone wrote up a web page on how to do Time Machine over SMB. Looks reasonable to me. Again, restore will be tricky because you'll have to re-install if your hard drive ever dies and manually restore data because you won't be able to use Migration Assistant. But at least all your data and the proper filesystem metadata will be saved.

Check it out: Time Machine over SMB

 

[MaKin211]

Thanks, seems to work. Should I still use rsync for the most important files?

 

[r0k]

Thanks, seems to work. Should I still use rsync for the most important files?

I use Crashplan for my most important files. This way if an asteroid or tornado strikes my house, I can get "important" files back from Crashplan's cloud. Crashplan is free unless you use their cloud. I find it to be worth it to have an "off site backup" so I pay for the full version.

I would think rsync would work just fine. There are some graphical rsync front ends. Some are free while others are freeware. I would take a look at those as well. One of my favorite programs on Windows was "Syncback" which was a freeware program that allowed sync of two folders at a specified time. I picked 1am every night. This means when I walked in with my 2008 Macbook, ready to make the full and final switch to MacOS, I only had to power off my Dell, power on my Macbook and copy everything I wanted down from the NAS drive I had been backing it up to. At the time I used Buffalo and Iomega NAS drives. I tried a few others, including a Time Capsule but today I've moved on to using Synology NAS drives. I have 2 of them that more or less mirror one another so if one dies, I have time to pick up a replacement unit and re-mirror everything before the second one goes. I still allow Time Machine to back up to my Time Capsule but I never depend on it for restore. Lastly, I have once migrated by doing a brand new fresh Time Machine backup to a Firewire drive, then using Migration Assistant to restore from that drive. I've also used target disk mode twice. Sadly I never succeeded in migrating from any Time Machine backup done over the network. It is for this reason I have a multi tier backup solution: Time Machine/Time Capsule, Crashplan(cloud) and NAS drives.

 

[MaKin211]

Thank you very much guys! My current backup strategy will be the following:

-TM backup on sparse bundle image on NAS
-Most important files via rsync (documents, pictures, etc.) on NAS and partially Dropbox (off-site) - I did not know crash plan yet!

Another question: Let's say I want to save my private files AND certain apps (with data and settings) seperate to my TM backup. How can I do this? Example: TimeTracker App. I can redownload from the AppStore but the statistics will be missing (no cloud feature).

EDIT: How about doing a crash plan backup of the backed up data on the server? This way I would not have to bother with the crash plan app on my Mac.