Best Backup Solution For Large Data Configutations?

[Zwhaler]

I'm running 13TB on my Mac Pro with a standard 2TB boot drive, and 3+4+4 the fourth being a Time Machine Backup drive. It can hold some things, but isn't nearly large enough to hold all of my data (currently using 8TB between the first 3 storage drives).

What is the best way to back all of this data up preferably using Time Machine? I'm already excluding lots of stuff to get my key items to fit on the 4TB, but have been bad about making regular backups because of the storage rapidly running out.

Can 12TB external RAID configs be read as a single drive? Or can anybody point me how to back up data using Time Machine spread over more than one drive if it is possible. Just wondering how to approach this, feel free to recommend any equipment that comes to mind. THANK YOU!

 

[ssls6]

I'm running 13TB on my Mac Pro with a standard 2TB boot drive, and 3+4+4 the fourth being a Time Machine Backup drive. It can hold some things, but isn't nearly large enough to hold all of my data (currently using 8TB between the first 3 storage drives).

What is the best way to back all of this data up preferably using Time Machine? I'm already excluding lots of stuff to get my key items to fit on the 4TB, but have been bad about making regular backups because of the storage rapidly running out.

Can 12TB external RAID configs be read as a single drive? Or can anybody point me how to back up data using Time Machine spread over more than one drive if it is possible. Just wondering how to approach this, feel free to recommend any equipment that comes to mind. THANK YOU!

It's called JBOD mode or just a bunch of discs. Not fast like stripping or safe like mirroring but it can be done. The mac has it build into core_services or the raid controller may as well.

I would go with Raid 5 for what you want. If you have 8 x 2TB drives in it, you would get 14TB total storage with some redundancy. Raid 6 would be 12TB etc...JBOD is all 16TB but running naked.

 

[brand]

Best Backup Solution For Large Data Configutations?

At work I just purchased two EqualLogic 48TB SANs and am looking at purchasing another one in about a month or so. Just one of those SANs should work for your needs. The 48TB SANs were $25K each if you are curious about pricing. If that is more than what you were wanting to spend we also got a quote for the 24TB configuration at $15K each but decided to go for the higher capacity.

 

[Zwhaler]

It's called JBOD mode or just a bunch of discs. Not fast like stripping or safe like mirroring but it can be done. The mac has it build into core_services or the raid controller may as well.

I would go with Raid 5 for what you want. If you have 8 x 2TB drives in it, you would get 14TB total storage with some redundancy. Raid 6 would be 12TB etc...JBOD is all 16TB but running naked.

I was thinking the same time, can anyone confirm that this will work with Time Machine? It seems like it would but I'm not buying anything like that until I know lol. Like I said I just need this JBOD to be recognized as a single disk for all intents and purposes. I don't have a RAID card so would I need one if reading off an external RAID setup or not?

At work I just purchased two EqualLogic 48TB SANs and am looking at purchasing another one in about a month or so. Just one of those SANs should work for your needs. The 48TB SANs were $25K if you are curious about pricing. If that is more than what you were wanting to spend we also got a quote for the 24TB configuration at $15K each but decided to go for the higher capacity.

What would you recommend for the 8-12TB range?

 

[nigelbb]

A NAS is an alternative. I use a Synology Diskstation 1511+ with 5x3TB disks in RAID-5 for around 12TB of usable storage.

 

[phoenixsan]

Imho......

this info is worth a look:

https://discussions.apple.com/thread/2260487?start=0&tstart=0

https://forums.macrumors.com/threads/617679/

 

[brand]

What would you recommend for the 8-12TB range?

Synology makes some very nice products. Alternatively you could roll your own FreeNAS server if you were up to it.

 

[bsbeamer]

CalDigit HDPro2 is configurable up to 16TB. More than your needs probably require, but does mount as a single drive and is setup as RAID-5. Can work with Time Machine, if setup that way. It's a fantastic drive. Data read/write speeds in RAID-5 are faster than many SSDs.

 

[deconstruct60]

I'm running 13TB on my Mac Pro with a standard 2TB boot drive, and 3+4+4 the fourth being a Time Machine Backup drive. It can hold some things, but isn't nearly large enough to hold all of my data (currently using 8TB between the first 3 storage drives).

You should consider not trying to dump all of your back-ups into one giant pile.

The growth and change rate of your boot/user account/ apps is likely going to be substantially different than your 3+4+4 collection. If that is largely a bunch of compressed media files that are largely being "collected" (if not archived ) then that is a completely different rate of change and growth.

Static media files don't really need to be back-up as much as duplicated. RAID 6 or 5 is a waste. It is multiple copies you need without the associated write holes.

If at 8GB (presuming most of that is on the 3+4+4 ) and relatively quickly going to fill to 12GB then you really should be looking pretty hard at tape. If you need two (or more ) copies of 10GB then the "all HDD" solution is going to start to get expensive. If on track to a couple years from now looking to crack the 20-25GB level that train is coming.

What is the best way to back all of this data up preferably using Time Machine?

Time Machine isn't a good match for spiting up the different volumes for different policies. It is free and oriented to "one big pile for everything" approach but the archiving methodology is questionable at these (and larger ) sizes.

For example your boot drive may have several variant versions of older files , executables , etc. stored in its TM archive. Meanwhile the vagarious media files are consuming the vast majority of the targeted TM disk. TM machine can decide to chuck the old versions from the boot drive to make room for the newer media files. Once chucked they are gone.

TM is designed for users who assign a drive with lots of extra room as being the target volume. For example users who on average consume 200-300GB and TM target drive of 1TB. It is not designed for users who are using 850GB (of 1TB ) drive and 1TB TM target drive.

Can 12TB external RAID configs be read as a single drive?

You can construct a logical (virtual) drive that is increasingly larger but that drive increasingly becomes a single point of failure for your system.
If your single Time Machine drive fails you are screwed. You whole archive is gone.

That is the questionable premise with Time Machine. Users contemplate the failure of that drive being backed up and totally ignore the failures possible of the drive copying the data too.

When the TM volume is a single drive a simple clone of that drive can "back up the back up". When it a logical drive with multiple internal points of failure that gets more complicated.

 

[Tesselator]

Yup! (although, personally I would never consider tape... I tried that already. )

I think it's a good idea to go with multiple simultaneous connections!

The HighPoint products are good for this. Either the 4-port eSATA or the 4-port USB 3.0.

Then just connect one drive per port. With it like that you can either RAID0 them for a single super-fast backup volume or use mirroring software to backup each drive to it's external counterpart as described above (if the mirroring occurs simultaneously it's just as fast as it would be setup as RAID0 too. And you can easily turn them off individually when backing up isn't needed. Keep in mind that a lot of mirroring software is cheap or free, direction specifiable, can be instanced via duplication, and typically only needs a few seconds to do it's job once the initial volumes are backed up the first time.

http://www.hptmac.com/product.php?_index=102
http://www.hptmac.com/product.php?_index=87

http://www.hptmac.com/product.php?_index=5

Also assuming that you keep the usual rule of not filling drives you use for I/O, beyond about 65% you only need 3TB drive to backup your 4TB drives and 2TB drives to backup your 3TB drives. In both cases Seagate Barracudas are super fast if you select their 1TB platter models. (ie 2TB - 2 platters, 3TB - 3 platters.)

I would time machine your system drive onto one of similar or larger size tho. TM is really useful to backup boot and project partitions!

 

[dwmc]

For what it's worth, I would not recommend TimeMachine as your primary means of backup. deconstruct60 is correct.

One of the big issues with TimeMachine is that you don't have a lot of control of the automation. You can tweak the frequency, turn it on and off, and of course exclude some things, but that's it (though you can do quite a bit more if you write some scripts to utilize tmutil). One of the problems I run into with TimeMachine on some machines is that it will kick off at an inopportune moment where someone has dropped a decent amount of data (say 100 gigs) for temporary use in a location that isn't in the exclusion list. TimeMachine backs it up. Then the user changes that data. TimeMachine backs it up again. The oblivious user doesn't notice until TimeMachine starts failing due to not having enough space on the target disk. And it's not very smart about it... TimeMachine can easily put itself in a position where it can't perform a backup because there's not enough space to be cleaned by wiping old backups because much of the space that's used is for recent backups, not old backups. And it's not always easy to fix it without wiping the whole TimeMachine target and doing a full backup. Mess up or delete the wrong thing on the target and TimeMachine will resort to wiping and doing a full backup. That can be a real drag if you've got a lot of data to back up and/or your target disk is not particularly speedy or your source disk is your boot disk and isn't SSD.

If using TimeMachine for backups, be very careful about what you allow to be backed up and make sure you aren't putting gobs of temporary data in locations that are backed up. Even my usual recommendation of TimeMachine target disk space of 4X the space being backed up can run into trouble. It's happened to two machines I admin this year... users don't always follow the rules of where to put their temporary data. "You mean it wasn't OK for me to drop 20 new blu-ray rips in my home directory a few times a day? But I deleted them before I put the next 20 on there! Why was that a problem?"

I use TimeMachine but it's for the convenience of being able to step back in time, mostly for files in a user's home directory. I have it alternate between two targets, one local and one over the network. I don't consider these backups. My backups are done with Carbon Copy Cloner to both local and network disk (a server using ZFS). One of the big reasons I prefer this to TimeMachine is that I can cycle the local disks off site each week and everything is stored as normal files... I don't need to restore to a disk to use them. This is a huge win on many fronts, one of which is the ability to boot from the clone of the boot disk in an instant. So you can keep working in a matter of minutes instead of hours. I can also cherry pick files off of the clones very quickly, say onto my Macbook Pro or a machine in the office.

 

[TwoBytes]

Drobo. Grow with new drives and you ca mix & match

 

[derbothaus]

Drobo. Grow with new drives and you ca mix & match

Slowly you can, yes. And loose tons of money to parity.

 

[Tesselator]

Slowly you can, yes. And loose tons of money to parity.

And never have very fast backups or data retrieval either.

Sloooow solution... (overpriced too... $850 for a 4-bay box... no drives... no cache module...) )

 

[brand]

Drobo. Grow with new drives and you ca mix & match

I would not trust any of my data to a Drobo, they are a joke.

 

[ibgb]

What is the best way to back all of this data up preferably using Time Machine?
I'm already excluding lots of stuff to get my key items to fit on the 4TB, but have been bad about making regular backups because of the storage rapidly running out.

Can 12TB external RAID configs be read as a single drive?

... Just wondering how to approach this, feel free to recommend any equipment that comes to mind. THANK YOU!

Forget time machine!

Read up on ZFS, snapshots....

Modern disk have a read error rate of 1 bit of every 10 ^ 14 bits. A raid 5 solution requires all the bits be correctly read to create the parity, so if a disk fails, the raid set can be rebuilt by reading all the bits and reconstructing the parity. So what is the biggest raid 5 set you can make and be 99.9 % sure you can have a disk fail and get back the raid set? Do the math.

You have a mac pro so you can get zfs from zevo for free. You might consider a Nas appliance as well, from Nexenta, Freenas, etc.

Think of how to chunk your data, so you don't have a 12TB chunk. I use disk images, sparsebundles. I chunk media in .5 TB chunks. I can fit 2 on a 1TB disk, 4 on a 2TB disk, etc. So I have MediaArchiveVolume1, MediaArchiveVolume2, MediaArchiveVolume3, etc. Sparsebundles can be stored on just about any filesystem, and when you mount a disk image it is always under /Volumes/, irrespective of where the actual disk image is stored.

You have alot of data. If you don't want to google this stuff, talk to a consultant. Maybe Nexenta would help you if the sell you a system.

 

[Tesselator]

I can vouch for ZFS slash RAID-Z being worth looking into! Pretty nice!

 

[rcook55]

Your running a Mac which at it's core is Unix, rsync and cron are your friends.

Build a freeNAS with 7 drives, put 5 of them in a RAID Z1, get a small SSD for a cache if you have spare money, throw a bucket of RAM at it. Designate one extra drive as a hot spare, keep the other drive in it's original packaging as a cold spare. No this is not overkill, RAID is not backup etc. If you pickup 3Tb drives you'll have ~15Tb of online storage, a hot spare to fail over to when one of your live drives fail (they will) and then when you can power down and swap it out you'll have that cold spare to put in the box while you send the failed drive off for (hopefully) warranty replacement.

Once you have the freeNAS up and running configure rsync to backup what you consider important and then schedule it with cron. All done.

I currently have 48Tb of online production data under my control. This is what I would consider a minimum for non-redundant home storage. Even better would be to have a second large disk that you sync to monthly at minimum and is kept offsite.