Future Proof Data Storage Solution?

[murphychris]

What I meant by saying Macs cannot backup the network storage using time machine is that apparently a mac cannot backup the data held on the network storage to a time machine in any way.

a.) Time Machine is a Mac OS process. So yes, you need a Mac to run Time Machine.

b.) The NAS storage will be larger than a single disk. Time Machine only backs up to single volumes. To use Time Machine with a MacPro or Mini based NAS, you'd have to create either a concatenated or striped array of multiple disks in order to create a large enough logical volume for Time Machine to backup the NAS. So you're inherently creating risky backup storage for your NAS. Not best practices. There is no built-in software RAID 5 or 6 on Mac OS X to provide a margin of safety for that backup, as well as aggregate storage so its big enough to receive the backup.

My concern was how do you backup the network data if its all connected to the network and not directly to the mac?

There are replication features in the various ZFS based products, including the ability to export the entire file system as a file, and snapshots as files. Those can go to any kind of storage: disks, arrays, NAS, or remote off-site storage like Crashplan for one computer.

Other products have slightly less sophisticated replication features, but nevertheless will let you build a 2nd RAID 5/6 array and rsync to it as offline backup (i.e. the backup is non-accessible over the network, only to the NAS).

I am still unsure what the difference really is here. Since I am saving for a mac pro I might as well use that with directly connected external drives, to share them over a network to a macbook air, TVs and xbox.

Thus far you're talking about 6 to 12 external disks. That's a minimum of 6-12 TB of storage. It sounds like you could be considering quite a bit more than that, possibly 18TB to 48TB based on your diagram. You haven't told us what your present day storage requirements are, or what the growth rate is. So no one can really answer your questions.

The fact of the matter is that Mac OS X is not a good operating system for what you're talking about. It does not have an enterprise class file system, or logical volume manager, or software RAID 5 or 6. Other operating systems do. You're confusing why you want a Mac desktop, with how you want your storage to function. And storage functionality on Mac OS X in the capacity you're talking about simply sucks.

If you build two RAID 0 arrays on Mac OS X, you can rsync (or CCC) from one to the other. But the instant one of them dies because one disk fails, you are at extremely high risk. As in, emergency, if you care at all about the data. It will take hours to days to rebuild a large array like what you're talking about, and that rebuild puts stress on the source. This is one reason why RAID 5 isn't used in enterprise much anymore (write hole, and long rebuild times means it's possible, even somewhat likely, a 2nd disk will fail and render a RAID 5 toast).

And not least, DAS is high risk for file system corruption, if there is a panic or unclean unmounting of the file system, it will be damaged. The journal is there only to make fsck faster, it does not make the file system more reliable.

You could make a smaller RAID 0 for your Mac Pro, with a few RAID 0 15K drives or SSDs if you want. It would be screaming fast. And then you can have the bulk of the data on the NAS.

Also the way I see it the mac pro or mini has unlimited expandability and access to a lot of software to manage the drives.

This does not compute. You need to be more clear why you think this. NAS hardware comes in more varieties than the total number of Mac models from the first 128K Mac to today. The software for managing this kind of storage preeminently exists on FreeBSD, OpenSolaris/Indiana, and Linux. Not Mac OS X.

Airport express is fairly limited and probably too slow. NAS is very limited in long term because although you can get multiple bay enclosures and some you can add another expansion onto it, theres a limit.

Obviously. What's unclear is precisely how you're coming to the conclusion that Apple hardware is less limited.

I just like the freedom of normal drives shared over network via mac pro. Theres also a smaller pay out for each expansion rather than buying an expensive multi bay enclosure. I must admit I had a NAS once and it was awful and such a waste and i am reluctant to get another hence the want to get something more normal but just network share it.

This is approaching a huge waste of time. The thread is in its fifth day and you just now tell us about your NAS bias, clearly because you got the wrong product once, while simultaneously not saying what product, what size, why it was awful, etc.

What you want is not normal. It's extremely abnormal from a storage expert's opinion because you are merely creating something with which YOU are familiar, not something that is actually best practices for the size of storage and expandability you keep talking about.

Aggregated storage is cheaper. Individual drives translating into a dozen icons on the desktop, and a dozen file shares inherently means inefficient storage and inefficient workflow and requires more and more drives because you're not using what you have efficiently.

I haven't checked lately, but I don't even know what personal file sharing on Mac OS X client's share limits are. Will it even let you share 12 disks?

It might even be better to keep the drives separate as then if one fails I can just restore that drive to a new one from a TM backup without complications on raid or jbod arrays, unless some raid arrays can do this well enough and efficient enough to the point where it doesn't cost twice as much per tb!

What you are describing is not best practices for a business. It's a huge inefficient waste of time. You're basically asking a plumber to do interior design, so it's any wonder why your storage is familiar to the plumber rather than someone who understands good storage practices.

p.s. whats wrong with HFS+? I have Mac OS X Extended file system on all my drives and haven't seen any issues. I thought it was the best file system?

Sorry to disappoint you. It's well past due for being replaced. No checksummed journal, no checksummed metadata, no checksummed data. It offers nothing beyond what the drive ECC does.

 

[jcpb]

Can't backup using TM
What I meant by saying Macs cannot backup the network storage using time machine is that apparently a mac cannot backup the data held on the network storage to a time machine in any way. NOT whether the mac can backup to a network storage using time machine, obviously this is just like time capsule. My concern was how do you backup the network data if its all connected to the network and not directly to the mac?

Use another network storage device to backup data from your existing NAS, or use an offsite archiving / mirroring service.

Mac mini/pro Server vs Airport express + HDD's vs NAS
Performance and expandability

Put it simply, can you remotely access the data on any of the storage devices directly attached to your Mac Pro when the system is powered off?

The answer is no.

Replacing the storage in the Mac mini Server - if those drives should fail - isn't a walk in the park. There's a reason NAS and real servers have HDD bays: if a drive should fail, take it out and swap a new one in its place, but you don't necessarily have to power off the NAS/server to do this.

One of the HDDs on the Mac mini dies? If it's an external drive, that's fine. Internal? You have to take the entire server offline to replace the dead HDD.

Using HDDs connected to AirPort Express: no, it doesn't work - that USB port only works with printers. You must use AirPort Extreme Base Station for connecting USB2 HDDs over the network. Either way, not really recommended as you'll get maybe 30-40MB/s off the connection.

NAS is limited by your network and then by the total amount of storage and power supplies available. Get past those three items, and sky's the limit in terms of expansion.

p.s. whats wrong with HFS+? I have Mac OS X Extended file system on all my drives and haven't seen any issues. I thought it was the best file system?

Network storage doesn't use HFS+.

Macs cannot natively write to NTFS - that's 99%+ of Windows systems and external HDDs (often cheaper than Mac versions while doing the same things) - without a third-party driver.

Windows cannot natively write to HFS+ without a third-party driver.

 

[murphychris]

Better than individual disk icons floating around, and still cheap is nested RAID 10. Disk Utility can combine one set of disks as RAID 0, another set of disks as RAID 0, then combine the two RAID 0's using RAID 1. That's then RAID 10.

It is not growable. To make it bigger, you have to build a new array. But it's significantly more reliable and easier to manage than JBOD which is a total PITA. You'd need two of them anyway, to have an on-site asynchronous "replicant". That is a backup that is identical to the primary, and could replace the primary if it goes down (i.e. you'd use it live rather than being totally offline until you rebuild the primary). It is not exactly identical, it's asynchronous meaning it's synced on a schedule. The mirrors are synchronous, obviously. But if each RAID 0 set loses 1 drive, the whole array is toast, which is why you'd need an onsite, for a business, 2nd RAID 10 array.

When it's time to expand, you either: build a 3rd one and migrate; then reconfigure array's 1 & 2 to make a new bigger asynchronous secondary, with spare drives available. Or you get serious with something at or approaching enterprise quality if you're talking about 8+TB, which will have the ability to be expanded.

 

[chiefsilverback]

I'm just in the process of overhauling my data storage/backup solution.

Up until now I've been using a Drobo formatted as a single huge volume and using Time Machine to backup my system and external drives in one go.

Time Machine and Drobo don't play all that nice together. The Drobo is presented as a 16TB volume, but I've only got 3TB of physical storage in it so Time Machine will just keep trying to add to the backup until it hits 16TB and gets confused when it runs out of physical storage at 3TB!

There are some workaround for this but some remove the "Enter Time Machine" functionality and others mean you can't benefit from growing the storage with time because you set a fixed size partition that can't be expanded...

The model I'm moving to is to re-format the Drobo and create a number of partitions to match the size of my physical drives. I'll use a tool like Carbon Copy Cloner to manage multiple backup jobs from the external drives to their corresponding Drobo partition and then have 'stand alone' Time Machine drive for my system disk.

I replaced the OEM HDD in my 2007 iMac with a 128GB SSD so all of my data is on externals. This model As long as I ensure that my total volume of data is less than the physical space available on the Drobo this model will provide fast access to my primary storage and a full backup. As my total storage requirements start to reach the physical space in my Drobo I can switch in larger drives and the additional space will be available to which backup partition requires it...

Obviously there's a single point of failure in that if the Drobo hardware dies I will lose all my backups. Safest option at this point would be to power down my externals for a couple of days until I can get a new Drobo ordered, delivered and configured then power up an external at a time create new backups.

I'm also going to get the OWC dual drive mini unit configured with 2 x 1TB drives in a RAID 1 mirror that will be used for a weekly backup of really critical data and I'll keep it in my fireproof safe.