Future Proof Data Storage Solution?

Personally, I would not recommend that your backup drive be part of your array. You are opening yourself up to a situation where an error impacts both your data, and its backup at the same time.

/Jim

 

I've also gathered quite some external HDDs during the last years to accomodate data and backups. Bunch of wires, different USB cables and noise.

I started thinking that ethernet and even WIFI connections have become faster than USB 2.0 (most drives) but somewhat still slower than USB 3.0 or FW.

So - why not completely seperate data and my computer(s). I started looking into NAS devices. Especially larger ones - 5+ bays to accomodate all my drives and be at least partially future proof.

I finally bought a Synology 1812 - an 8 bay enclosure. Expensive - yes, but less if you think of data as one of more important aspects of your setup. If a new laptop costs 1500 euro, than 900 for a NAS seemed reasonable. I dissasembled all my external drives and put the disks into NAS.

No need for for partitions anymore. You have volumes, which span accross multiple disks, with data redundancy. The filesystem can also be encrypted for more security. No more problems with incompatible filesystem, when switching between Windows and Mac systems... SMB works everywhere.

I have one local backup to an external USB drive and one I do manually over the internet to another location twice a month. This is only for most important data, I trust others with data redundancy and UPS protection against surges.

Oh, and Hybrid RAID makes it possible to switch older disks with new ones, expand the array and retain all data in the process.

Currently I have a bunch of older 500 and 1000 GB drives and therfor only about 5GB space. If I put 8x4TB, that would be 28 GB of data... maybe even more if Synology updates the firmware for larger HDDs, when they arrive.

But then I also expect that in a couple of years you will be able to rent a couple of TB of Dropbox space for much less than today.

 

No, and no. Basically what you want doesn't exist on Mac OS X. An OS X software RAID 0 or 1 array can't be expanded.

So you're looking for a network based solution for large primary storage. And then for fast local storage (I assume video editing) you can continue to use a RAID 0 array.

LVM2 and mdadm on Linux can do what you describe, and have been doing it for quite a while. The storage can be hosed either over AFP or NFS shares over a network.

The closest thing there is for direct attached storage option based on your description is a Drobo. That's proprietary and I'm fairly dead set against proprietary storage for important data especially if migration is expected. And I've just heard of (directly from) and read (indirectly from) too many users with data loss. So it's not a solution I can recommend.

 

WiFi performance sucks. Even n speeds are hideous compared to gigabit. And if you're doing HD video editing, gigabit likely will not have the bandwidth you require. HD video professionals using SAN/NAS are now using 10GigE.

What you want are physical drives placed into a storage pool, and then the pool is divvied up into logical volumes. This has been on Linux as LVM for something like 15 years. And it's possible to resize while data is still available. You can even move extents from one physical disk to others, and remove that physical disk (e.g. prior to failure, or because it's the smallest disk in the pool), while actively using any of the logical volumes.

Mac OS X 10.7 kinda sorta has nacent logical volume management in Core Storage, but the command line tools are extremely basic. So you can't really do what you want. I have no idea if the tools will have more features in 10.8, but it's something to keep an eye on.

NAS can be put on a network with internet access, and you can access files on it with ssh (secure shell) with a GUI application like CyberDuck or Filezilla. Obviously this will be slow.

If you're talking about playback, BD 1x is 36Mb/s, which ought to be possible to stream even over WiFi, if the signal is decent. But I don't know what you mean by full untouched.

 

A NAS is what you need.
With a Synology NAS you have Synology Hybrid RAID, which is like RAID5 but with the ability to use different size disks.

It's not ideal to have the back on the same unit or array though, but you can always buy an expansion DX510 unit for backups, which is what I have done.

It is very easy to add another drive and expand volumes across to it, so expansion is not an issue. If the unit you have fills up with drives, again the Dx510 unit can be used to attach another 5 days to your original volume.

Check my videos on YouTube - link in sig - which will teach you how to expand and create volumes, and how to set up backup etc when it comes to the NAS. Its really, really easy and completely expandable.

I have since got rid of my external HDDs and only have one small drive for backup images of my MacBook, and another for my scratch disk with iMovie etc. I now have no concerns with expand ability etc at all, as I know all I have to do is add an additional HDD and simply expand the current volume!

 

Concatenated disk arrays cannot survive even one failed disk, all data will be lost. This is completely incompatible with an "Archive" which contains important information. Only if you are OK expecting complete array failure and all data lost in that array should you use either RAID 0 or Concatenate arrays.

Unfortunately at this time Apple offers nothing for a solution to your problem, so I can't recommend a Mini. It simply brings nothing to the table compared to other offerings.

You cannot live edit files over the internet of any size. You'd have to download the file, edit the file locally, and then push the copy back up to your NAS. My idea of a large Photoshop file is 2GB or greater. That will take a while in either direction because most internet connections are upload bandwidth limited, and you have an upload occurring on one end no matter what. I get roughly 4 hours if either end has a 1Mb/s upload. That's in each direction.

 

Synology appears to be using Linux and mdadm for RAID. Meaning if any components in the NAS hardware die outside of warranty, you conceivably can regain access to your data by moving the disks to alternative hardware. Here they describe Hybrid RAID as being not proprietary as well.

Based on the illustrations, it appears they're partitioning all disks with consistent volume sizes that match the lowest common denominator drive size. And then applying mdadm RAID among those partitions across disks, such that it actually ends up with multiple arrays. Then those multiple arrays are likely added to an LVM volume group, and then you specify through an interface what logical volumes you want. It could be one, or hundreds.

My suggestion is that you never trust anything to a sole copy of your data. RAID is not a backup, nor does it make things more reliable. All it is about is data availability in the face of an inevitable hard disk failure. Before you expand the array or replace a disk, you need a copy. Either exported to external disks, or another NAS.

 

Sound advice

 

Synology, and some other products out there, are leveraging enterprise level technology. But the catch is that they can't compel consumers to leverage enterprise level best practices as well. Consumers will just expect a product they pay for with a particular feature to "just work". But that's not how enterprise avoids data loss and corruption - they avoid it through using the best technology they can, and best practices. In this case, that's replicating the data, backing it up, and archiving it. Three separate things. You would never find any enterprise, spending millions of dollars on very high end hardware and software development, do a volume group or array expansion without multiple backups. Ever.

Storage is a bitch, pretty much.

 

As for local network speed, GigE has been used for live editing of non-HD video. So it seems like you should be able to do the same thing with .psd/.psb files. But my understanding is Adobe doesn't support this workflow, at least for Photoshop. My best guess is that the results are mixed because networks for video editing are very rigorously put together and tested. As or more rigorous are data center networks. Whereas home, home office, and even most businesses with design departments don't have such rigorously configured or tested networks.

If you attempt live editing, I personally would disqualify WiFi, and I'd also suggest you use derivative files (make a copy work on the copy). The potential for corrupting the original is just too high.

Best practices though is still to copy the file off the NAS, edit the file locally, and then push the renamed derivative file back to the NAS. If you don't rename, and overwrite, you risk corrupting the lone copy on the NAS if the network copy fails for some reason. And not all failures come with warnings (see silent data corruption, which happens on networks as well as drives). So you're better off making derivatives, and cleaning them up later when you determine you don't need them.

 

Possibly a useful discussion on this, not quite 1 year old.

http://indesignsecrets.com/editing-indesign-files-directly-off-a-server-crazy.php

Again, it *should* work if the (wired) network is reliable, as video people do this as their standard workflow and they are pushing/pulling way way more data. But because of this, any glitches cause MAJOR problems for them, so they are assured of very high quality components: all runs meet the IEEE requirements for max length, max bends, minimum permissible radius for those bends, minimum safe distance from power sources and transformers like fluorescent ballasts, and properly punched panels. Inordinate numbers of networks just have cables on the floor. Not OK. Pinched, bent, even a stapled cable is not OK.

For my craptastic home cabling setup (ratty CAT 5e cables, but they are short), I can routinely push "big" 10MB+ files over 100+MB/s over NFS, to/from Mac OS X and a Linux box (pretty much out of the box Fedora setup, very little tuning). So I know it can be done...

 

NAS is designed to hold drives internally, or in an expansion enclosure, as it's specifically designed for the task of serving files. NAS also usually offers a wide assortment of protocols, with straigtforward enabling and configuration, like http, ftp, scp, NFS, AFP, SMB, and so on. Whereas these things may be in very disparate form on other kinds of general purpose servers.

 

Huh?

I can backup via Time Machine to a network share on my NAS just fine.

 

A desktop has the exact same expandability limitation. A NAS can certainly accept external disks via eSATA if you don't mind the ensuing clutter.

A flexible headache perhaps, as you have an icon on the desktop for each drive you plug in. And they will not be all that fast unless you use (e)SATA or Thunderbolt. Further you're stuck with the limitations of HFS+ which is not a file system I have much trust in - it offers nothing in the way of checksumming metadata or data on the disk, it defers entirely to the disks's ECC. Bit rot, and silent data corruption are real problems, in particular in the amount of storage you appear to be desiring in your diagram.

Untrue. Time Machine support over a network is a function of AFP, and BSD and Linux NAS's almost all have recent versions of Netatalk to provide AFP and network Time Machine support.

Well it's a little bizarre that you'd complain about the lack of expandability and performance of a NAS compared to direct attach storage, and then ask about by far the slowest and least expandable option imaginable.

 

I've been follow this thread till now and found it quite interesting (though your problem doesn't apply to me)

The cheapest thing you could probably do is to do a software JBOD, you can find information about that there: http://systemsboy.com/2006/04/apple-software-raid-tests-or-what-to-do-if-you-dont-have-an-intel-mac.html
That way if your array becomes too small, you simply add another drive and you are good to go.
It seems to be quite dangerous though, because one failed drive and the whole array is worth ****.

So two NAS or SAN units (one for backup) seem to be the best thing you can do. That is IF you can just replace one or two of the drives with bigger drives when you need more space. Is that possible on a 5bay Synology, without rebuilding the whole array?

 

I have not expanded a concatenated array on Mac OS X, but this is usually a two part process. One is adding the disk to the array, making the array larger. But the file system also needs to be resized to use that additional space. And my experience with HFS+ has been hit or miss on successful resizing. More than 1/2 the time, using command line fsck_hfs -r to force rebuild the catalog will then allow such a volume to be resized successfully. This rebuild option is not available in Disk Utility.

For sure NAS meaning file-based access, rather than SAN or block-level access. There are open source alternatives to Apple's XSan which is now free in Lion and Lion Server, but I can say how well supported they are on Mac OS X. XSan though still requires fibre channel. Not cheap or easy to lay that down, and it requires at least one, preferably two, metdata servers. I'd so much more quickly use a 10GigE NAS running a Linux/XFS or OpenIndiana/ZFS solution if I needed something like this.

It can't be arbitrary. You'd need to look at what mdadm can do since it appears that's what Synology are using. Some kinds of resizes are possible without rebuilding, and others aren't. I've done RAID 1, 5 and 6 resizes without rebuilding but always with bigger disks than what was already in the array. Some resizes don't actually get you bigger arrays until all disks are replaced.

No doubt it's possible to download a user manual for Synology and find out what it can and can't do. There's also practical user interface limitations, even if the command line can do it, some of the logic gets quite complex and defies even well done UI. Safer to just make it a non-option.