4TB Network backups

[ecker00]

Hi,

I know this have been asked over and over, but I think my situation is a bit different from others.
With 3.5TB of data over WiFi to a NAS.


Situation
Data to Backup: Currently I have 3.5TB of work data (on a 6TB RAID0, 3x2TB), which needs backing up (increasing with about 500-1000 GB a year).

Backup target: I also have an NetGear 2-Bay ReadyNAS device (with 8TB RAID0, 2x4TB) I want to backup to. Unfortunately this is via WiFi, with transfer rates around 3MB/s.


Time Machine
I've tried Time Machine but it became corrupt after 14 days of trying to backup and during that time, it only backed up 800GB of 3.5TB.

But Time Machine seems to work inefficient, it takes a break between files before it continues to copy the next one. Technically speaking with 3MB/s the backup should only take 12 days.

Illustration of a break during Time Machine backup


Recovery
I'm not too worried about recovering the files from the NAS, as I can hook it up to the computer with cable for a day, and at 100-120MB/s (gigabit cable) the recovery should be done in about 8-10 hours.

(I could also do this with the initial backup, but in the long run it has to be over WiFi, and I don't want any more "Backup corrupt, start over again" messages)


Solution?
Which of the following software is suitable for this situation? (Big data over WiFi)

Carbon Copy Cloner
SuperDuper
ChronoSync
Any Other ones?

The most important thing is that it's more efficient in the way it transfers files over network, and doesn't have to restart completely if it become corrupted (like Time Machine would).


(I currently have a temporary backup of all the sensitive data to individual 500GB drives, while I'm getting my proper backup solution up and running, so I don't mind if the initial backup take 1 week or more.)

 

[Wando64]

I would attach the NAS with a cable for the initial backup.
Successive backups over WiFi should not be a problem for any backup software as they are likely to be comparatively small.

 

[ecker00]

I would attach the NAS with a cable for the initial backup.
Successive backups over WiFi should not be a problem for any backup software as they are likely to be comparatively small.

Okey, thank you. Then I guess it's not such a big deal which one of them I pick as long as they support network backup.

 

[960design]

Update your WiFi router?
At home I have a Netgear N750 that peaks near 300Mbps ( ~ 10 times faster than yours ) on 5Ghz; daisy chained behind are two 4TB Western Digital NAS.

Backups are no problem, even with multiple computers backing up at the same time.

There are even faster routers available, something like 750Mbps.

 

[ecker00]

Update your WiFi router?
At home I have a Netgear N750 that peaks near 300Mbps ( ~ 10 times faster than yours ) on 5Ghz; daisy chained behind are two 4TB Western Digital NAS.

Backups are no problem, even with multiple computers backing up at the same time.

There are even faster routers available, something like 750Mbps.

Yes, that would be something to consider. But best solution would be a gigabit cable, I might get that sorted. But I think the slow speeds are mainly because of the distance, with the router in the basement and my computer on the 2nd floor.

For now I'm looking for the most suitable backup software, the fastest preferably.

 

[zhenya]

I agree that in the long run a wired connection would be worth the effort, and if it can't be done, to upgrade to a modern wireless standard. Powerline networking may also be worth looking in to.

For software, you could use Crashplan's software for free if you are not backing up to their cloud service, and it handles large volumes of data very well. Even better is the fact that it handles changed files more efficiently than any other backup software I've ever seen, so it would minimize the amount of data that would be transferred after the initial seed.

 

[960design]

Yes, that would be something to consider. But best solution would be a gigabit cable...

Only if your router has a gigabit line out. I'm betting if you are running 3MB/s then you probably have the old 10/100Mbps wired lan.

Yes, wired it typically better.

 

[ecker00]

For software, you could use Crashplan's software for free if you are not backing up to their cloud service, and it handles large volumes of data very well. Even better is the fact that it handles changed files more efficiently than any other backup software I've ever seen, so it would minimize the amount of data that would be transferred after the initial seed.

I've been testing out crash plan now, and wow... that program is great!


It is perfect for my situation, as it compress the data before sending it to my NAS, reducing the network usage. As well as keeping multiple versions of files, which I love about Time Machine. And it's free for local setups!


Downside is that I can't get the files out of the backup manually. That's only possible using CrashPlan's own recovery, so I have to trust them to be able to decompress and recover it in the future and hopefully no corruption.


I did a test backup for 11GB, which took 52 minutes. That makes for an average network speed of 3.53 MB/sec, but I was monitoring how much data was actually being sent which was about 1.50~2.50 MB/sec and ~20% CPU. Proving that the compression is working, which is perfect for this situation. Freeing up some bandwidth for downloads and browsing.


This software seems perfect for me. I'll use Time Machine for my internal SSD with OSX and Crash Plan for my work data. Also love that little terminal window CrashPlan have where you can type different commands (click on the CrashPlan Logo top right).


Thanks for the tip on CrashPlan, I didn't know about that one!
Now just to hook it up with cable and transfer the remaining 3.5TB.

 

[zhenya]

Great!

I have extensive experience with Crashplan - been using it home for many years - and just this year moved our workstation backups to their service at work. It's been extremely reliable (as far as consistent backups with no corruption) and restores always work flawlessly.

One issue that a surprising number of backup systems have is that they don't handle changed files very intelligently. This is a big deal in your situation on a slow connection with a huge volume of data. Take the following potential situation; you are cleaning house on the main data store and move the file structure around. Say 2TB of your data is 'changed' in that the file path is now slightly different. The next backup comes around and the software now sees much of that as 'new' data and the software begins to duplicate your backups by sending the entire 2TB again! I've seen this happen numerous times (and paid for it with some cloud providers).

This doesn't happen with Crashplan as they do block-level file deduplication. I have over 1TB on their servers at home, and about the same from our office, and it just works.

 

[ecker00]

This doesn't happen with Crashplan as they do block-level file deduplication. I have over 1TB on their servers at home, and about the same from our office, and it just works.

I noticed the "de-duplication" option, that is genius. Relieving to hear you have good experience with it, and that it's been reliable for you.

Again, thanks for sharing!

 

[opinio]

Hi,

I know this have been asked over and over, but I think my situation is a bit different from others.
With 3.5TB of data over WiFi to a NAS.


Situation
Data to Backup: Currently I have 3.5TB of work data (on a 6TB RAID0, 3x2TB), which needs backing up (increasing with about 500-1000 GB a year).

Backup target: I also have an NetGear 2-Bay ReadyNAS device (with 8TB RAID0, 2x4TB) I want to backup to. Unfortunately this is via WiFi, with transfer rates around 3MB/s.


Time Machine
I've tried Time Machine but it became corrupt after 14 days of trying to backup and during that time, it only backed up 800GB of 3.5TB.

But Time Machine seems to work inefficient, it takes a break between files before it continues to copy the next one. Technically speaking with 3MB/s the backup should only take 12 days.

Image
Illustration of a break during Time Machine backup


Recovery
I'm not too worried about recovering the files from the NAS, as I can hook it up to the computer with cable for a day, and at 100-120MB/s (gigabit cable) the recovery should be done in about 8-10 hours.

(I could also do this with the initial backup, but in the long run it has to be over WiFi, and I don't want any more "Backup corrupt, start over again" messages)


Solution?
Which of the following software is suitable for this situation? (Big data over WiFi)

Carbon Copy Cloner
SuperDuper
ChronoSync
Any Other ones?

The most important thing is that it's more efficient in the way it transfers files over network, and doesn't have to restart completely if it become corrupted (like Time Machine would).


(I currently have a temporary backup of all the sensitive data to individual 500GB drives, while I'm getting my proper backup solution up and running, so I don't mind if the initial backup take 1 week or more.)

CCC is far better over network than Super Duper.

 

[flynz4]

Crashplan+ is great. It has deep versioning, good de-dup, monitors changes in the background, and if very resource efficient.

I personally use it with Crashplan Central on a family unlimited plan... so we typically have about 8 computers being backed up. At work, I also use Crashplan Pro Enterprise for my MBA.

I am a pretty strong advocate of using dual independent backup programs. I personally use the combination of TM for local backup and CP+ for cloud backup.

One thing that you might consider is adding remote capability of backing up to an off-site location using Crashplan's ability to back up to "friends". You can create a local backup using a direct attached drive... and then move it to an offsite computer for continuing backup. This allows you to effectively "seed" your offsite backup at no cost of internet bandwidth.

/Jim

 

[ecker00]

CCC is far better over network than Super Duper.

This is what I was hoping to hear about when making this forum post (Until I discovered CrashPlan). But good to know for future reference.

I personally use it with Crashplan Central on a family unlimited plan... so we typically have about 8 computers being backed up. At work, I also use Crashplan Pro Enterprise for my MBA.

I might go for CrashPlan Central in the future, but I want a bit more experience with using CrashPlan locally before I trow almost 4TB on it.

 

[MacTribe]

Crashplan is the top one on the list for me too, its like time machine on steroids and always works.. From the enterprise side (they do enterprise server), you can actually seed your backup directly so essentially you could host a server at a friends house or office (if they have a static IP) and only the changes / new stuff will be synced without the headache of having to upload to an online cloud or crashplan central.

Enterprise version is not expensive, you can buy it for the cost of one user licence - check it out on their website. Its really great!

The only downside to crashplan over time machine is time machine is a bare metal backup, meaning if your laptop or desktop died, you could 'migrate' all settings, apps, everything to a new machine in one click. Crashplan is just data, so you would only recover your personal files.

Its always a weigh up between data backups and bare metal backups, but I think you have to choose reliability over convenience.

We run a CP enterprise server at work for our clients, nothing but positive reviews and smiling faces. Cant go wrong!

----------

FYI some links to setup and other stuff if you're tempted to go down the enterprise 'DIY' server.. If you have any questions on how to set this up, hardware required etc, let me know. I've set this up plenty.

How to install PROe server

Single Licence - Pricing from their website:
PROe Pricing

Youtube - Server setup (their video)

 

[onekerato]

Also check out BitTorrent Sync, for mirroring folders between computers.

Can be configured to work purely on LAN, without seeking out any BitTorrent servers. Acts like a personal Dropbox.

 

[MacTribe]

Also check out BitTorrent Sync, for mirroring folders between computers.

Can be configured to work purely on LAN, without seeking out any BitTorrent servers. Acts like a personal Dropbox.

Nice find.. Thats a really useful idea for a change. Love the privacy! No servers, just your own machines syncing securely.

Just encase anyone needs a link to the website, here it is