Future Proof Data Storage Solution?

[LERsince1991]

Hi,

So I've been buying hard drives quite often, selling old ones and upgrading to larger capacities and more bay enclosures, but to what end?

I'd like to know what solution people suggest which has the ability to expand when needed to larger capacities at a good price?

The major problem for me is that every time I reach my capacity with my current hard drive system I look at buying more hard drives and another enclosure or replacing the whole lot with a more elegant solution.

I would like pretty much 2 or 3 partitions on my whole drive array however that may be setup. So I need "Time Machine", "Media", "Archive". Time machine backing up the media and archive partitions/drives as well as my macbook (soon to be replace by a Mac pro and macbook air which would connect to the drives through the mac pros network sharing I think).

If these 3 partitions were created from multiple drives using OS X Disk Utility Raid 0 maybe. Is there a way to expand that raid without moving and replacing all the data on them? So for example if I had a 2 drives making a raid 0 for say "Time Machine" and I want to expand the time machine with another drive or upgrade one of the existing drives could I combine them all together without deleting all the data and needing to copy it all back over?

I currently own 2x2tb western digital green hard drives (£65 couple of year ago) connected in an icebox 2 bay enclosure (£50 couple of years ago). I'm not very happy with the enclosure as I would prefer it to power on with the mac instead of flicking a switch every time, I also have to manually control the fan from off to 2 fan speeds. Also have a 1tb WD my book from 3 years ago. I was then thinking about replacing them with a 4 or 5 bay enclosure like a Lacie 4big or drobo but then thought, what happens when I fill that up in a couple of years! I suppose one solution (not sure how it couple work in terms of connections and multiple drives) could be buying multiple single enclosure hard drives and buying another hard drive when needed to stick into another single enclosure. So you start with say 4 enclosures (maybe all daisy chained with FW, estata or thunderbolt?) then in a couple of years time I need another 2 drives I could connect another couple of the same enclosures/drives into the daisy chain. If all the enclosures are the same the setup should look quite elegant as well as being easy and manageable to expand simply by adding to the daisy chain. Not sure about the many plug sockets though, or how this could work when wanting to maintain 3 visible drives/partitions/raid's.

Summary
I want to create an elegant solution which can expand in capacity through time possibly whilst maintaining the same appearance (3 partitions/drives/raid0's)

Should be an interesting topic to discuss!
I look forward to hearing your thoughts on the topic.
Thanks, Luke.

 

[LERsince1991]

I guess I am asking 2 questions;

1 - What is the most elegant and good value for money method to create an expandable storage solution? I realised that effectively how ever many bays the enclosure would have are just how ever many increments of hard drives I would want to expand to each time. So if I bought single drives I would expand maybe yearly with a new drive each year, or a 2 bay maybe every 2 years.

2 - Is there a way to expand data storage hardware (i.e. more drives and more capacity) without reconfiguring the entire setup each time if I want to keep 2 or 3 visible partitions on the mac.


And to add some context into the discussion I am using the storage for personal use as well as an expanding workload for architectural university projects, product/furniture design ventures and lots of personal photography.

 

[flynz4]

I would like pretty much 2 or 3 partitions on my whole drive array however that may be setup. So I need "Time Machine", "Media", "Archive". Time machine backing up the media and archive partitions/drives as well as my macbook (soon to be replace by a Mac pro and macbook air which would connect to the drives through the mac pros network sharing I think).

Personally, I would not recommend that your backup drive be part of your array. You are opening yourself up to a situation where an error impacts both your data, and its backup at the same time.

/Jim

 

[LERsince1991]

Ok,

Sorry I meant to really say have 2 or 3 arrays to expand into something like so;

Time Machine
HDD 1
HDD 2
HDD 3
HDD 4

Media
HDD 5
HDD 6

Archive
HDD 7
HDD 8


I support the media and archive could be in the same array but this is how I intended to do it. Does your comment/warning still stand or does this solve that issue?

Thanks for the input!

 

[537635]

I've also gathered quite some external HDDs during the last years to accomodate data and backups. Bunch of wires, different USB cables and noise.

I started thinking that ethernet and even WIFI connections have become faster than USB 2.0 (most drives) but somewhat still slower than USB 3.0 or FW.

So - why not completely seperate data and my computer(s). I started looking into NAS devices. Especially larger ones - 5+ bays to accomodate all my drives and be at least partially future proof.

I finally bought a Synology 1812 - an 8 bay enclosure. Expensive - yes, but less if you think of data as one of more important aspects of your setup. If a new laptop costs 1500 euro, than 900 for a NAS seemed reasonable. I dissasembled all my external drives and put the disks into NAS.

No need for for partitions anymore. You have volumes, which span accross multiple disks, with data redundancy. The filesystem can also be encrypted for more security. No more problems with incompatible filesystem, when switching between Windows and Mac systems... SMB works everywhere.

I have one local backup to an external USB drive and one I do manually over the internet to another location twice a month. This is only for most important data, I trust others with data redundancy and UPS protection against surges.

Oh, and Hybrid RAID makes it possible to switch older disks with new ones, expand the array and retain all data in the process.

Currently I have a bunch of older 500 and 1000 GB drives and therfor only about 5GB space. If I put 8x4TB, that would be 28 GB of data... maybe even more if Synology updates the firmware for larger HDDs, when they arrive.

But then I also expect that in a couple of years you will be able to rent a couple of TB of Dropbox space for much less than today.

 

[LERsince1991]

Ok sounds like a step in the right direction.

I need to find out more about hybrid raid?
And yeh possibly If gigabit Ethernet or wifi can provide a good speed then a network hard drive seems like a good way to go actually. Mainly because of the added benefit to be able to access the data and backup over a network with multiple computers. And since I intend to replace my MacBook with an air and a mac pro this seems like a good match.

I see that a 5 or even 8 bay nas is going to last a long time. However I don't really want to spend a huge amount on an 8 bay as it still has a limit, I'd be inclined to see if there is some sort of separates system here. So sort of a network USB/esata/FireWire hub which I can daisy chain a lot of single hard drives from and which will be able to use these hard drives as a nas would as bays. This will give unlimited expand ability as the single drives can be daisy chained over some connection maybe even thunderbolt. To add more storage just add to the daisy chain. But when the "hub" that te drives are connected to fails or becomes obsolete I won't have to upgrade to another 8 bay enclosure. Instead te hub could be replaced and just connect a daisy chain onto a new hub/sharing device.

Is what I am referring to as a networking hub here a "server"? I have no experience with this sort of advanced networking stuff. However it seems that having some sort of hub/server separate to the bays and daisy chaining a external hard drives and treating them as bays would create a truest expandable and future proof system as the server can be easily replaced when the time comes with something else if needed but the "external bays" could just sit there an then be daisy chained to a new server/hub.

As an added thought. Is there a certain thing I would be looking for to enable the networked drives to be accessible over the Internet from anywhere in the world? This would be extremely useful.

And another thought I could also look into, I have a lot of media on the drives, is the network/gigabit Ethernet fast enough to stream a ful untouched blue ray video file over? And is there a certain feature to look for that will enable sharing of this media to a Samsung smart tv or Xbox 360?

 

[murphychris]

If these 3 partitions were created from multiple drives using OS X Disk Utility Raid 0 maybe. Is there a way to expand that raid without moving and replacing all the data on them? So for example if I had a 2 drives making a raid 0 for say "Time Machine" and I want to expand the time machine with another drive or upgrade one of the existing drives could I combine them all together without deleting all the data and needing to copy it all back over?

No, and no. Basically what you want doesn't exist on Mac OS X. An OS X software RAID 0 or 1 array can't be expanded.

So you're looking for a network based solution for large primary storage. And then for fast local storage (I assume video editing) you can continue to use a RAID 0 array.

LVM2 and mdadm on Linux can do what you describe, and have been doing it for quite a while. The storage can be hosed either over AFP or NFS shares over a network.

The closest thing there is for direct attached storage option based on your description is a Drobo. That's proprietary and I'm fairly dead set against proprietary storage for important data especially if migration is expected. And I've just heard of (directly from) and read (indirectly from) too many users with data loss. So it's not a solution I can recommend.

 

[murphychris]

If gigabit Ethernet or wifi can provide a good speed then a network hard drive seems like a good way to go actually.

WiFi performance sucks. Even n speeds are hideous compared to gigabit. And if you're doing HD video editing, gigabit likely will not have the bandwidth you require. HD video professionals using SAN/NAS are now using 10GigE.

So sort of a network USB/esata/FireWire hub which I can daisy chain a lot of single hard drives from and which will be able to use these hard drives as a nas would as bays.

What you want are physical drives placed into a storage pool, and then the pool is divvied up into logical volumes. This has been on Linux as LVM for something like 15 years. And it's possible to resize while data is still available. You can even move extents from one physical disk to others, and remove that physical disk (e.g. prior to failure, or because it's the smallest disk in the pool), while actively using any of the logical volumes.

Mac OS X 10.7 kinda sorta has nacent logical volume management in Core Storage, but the command line tools are extremely basic. So you can't really do what you want. I have no idea if the tools will have more features in 10.8, but it's something to keep an eye on.

As an added thought. Is there a certain thing I would be looking for to enable the networked drives to be accessible over the Internet from anywhere in the world? This would be extremely useful.

NAS can be put on a network with internet access, and you can access files on it with ssh (secure shell) with a GUI application like CyberDuck or Filezilla. Obviously this will be slow.

And another thought I could also look into, I have a lot of media on the drives, is the network/gigabit Ethernet fast enough to stream a ful untouched blue ray video file over? And is there a certain feature to look for that will enable sharing of this media to a Samsung smart tv or Xbox 360?

If you're talking about playback, BD 1x is 36Mb/s, which ought to be possible to stream even over WiFi, if the signal is decent. But I don't know what you mean by full untouched.

 

[LERsince1991]

Sounds ok.

Just to clarify I don't do any HD video editing so not worried about having a directly attached storage. A networked storage seems like te best way. Having some sort of basic server manage clients and disk spanning, say a mac mini with a daisy chain of thunderbolt drives connected which can fairly easily expand as a new drive is added by adding the new drive to span.

Over the Internet speeds aren't too much of an issue, just a thought which would be useful. Over the network speeds will probably be ok by the sounds of it as I wouldn't be regularly copying huge files over, just time machine backups and large blue ray copies every now and again, or accessing documents over.

Does the network speed matter if I was to be workin on large photoshop files over the network?

 

[Ifti]

A NAS is what you need.
With a Synology NAS you have Synology Hybrid RAID, which is like RAID5 but with the ability to use different size disks.

It's not ideal to have the back on the same unit or array though, but you can always buy an expansion DX510 unit for backups, which is what I have done.

It is very easy to add another drive and expand volumes across to it, so expansion is not an issue. If the unit you have fills up with drives, again the Dx510 unit can be used to attach another 5 days to your original volume.

Check my videos on YouTube - link in sig - which will teach you how to expand and create volumes, and how to set up backup etc when it comes to the NAS. Its really, really easy and completely expandable.

I have since got rid of my external HDDs and only have one small drive for backup images of my MacBook, and another for my scratch disk with iMovie etc. I now have no concerns with expand ability etc at all, as I know all I have to do is add an additional HDD and simply expand the current volume!

 

[murphychris]

Having some sort of basic server manage clients and disk spanning, say a mac mini with a daisy chain of thunderbolt drives connected which can fairly easily expand as a new drive is added by adding the new drive to span.

Concatenated disk arrays cannot survive even one failed disk, all data will be lost. This is completely incompatible with an "Archive" which contains important information. Only if you are OK expecting complete array failure and all data lost in that array should you use either RAID 0 or Concatenate arrays.

Unfortunately at this time Apple offers nothing for a solution to your problem, so I can't recommend a Mini. It simply brings nothing to the table compared to other offerings.

Does the network speed matter if I was to be workin on large photoshop files over the network?

You cannot live edit files over the internet of any size. You'd have to download the file, edit the file locally, and then push the copy back up to your NAS. My idea of a large Photoshop file is 2GB or greater. That will take a while in either direction because most internet connections are upload bandwidth limited, and you have an upload occurring on one end no matter what. I get roughly 4 hours if either end has a 1Mb/s upload. That's in each direction.

 

[murphychris]

Synology appears to be using Linux and mdadm for RAID. Meaning if any components in the NAS hardware die outside of warranty, you conceivably can regain access to your data by moving the disks to alternative hardware. Here they describe Hybrid RAID as being not proprietary as well.

Based on the illustrations, it appears they're partitioning all disks with consistent volume sizes that match the lowest common denominator drive size. And then applying mdadm RAID among those partitions across disks, such that it actually ends up with multiple arrays. Then those multiple arrays are likely added to an LVM volume group, and then you specify through an interface what logical volumes you want. It could be one, or hundreds.

My suggestion is that you never trust anything to a sole copy of your data. RAID is not a backup, nor does it make things more reliable. All it is about is data availability in the face of an inevitable hard disk failure. Before you expand the array or replace a disk, you need a copy. Either exported to external disks, or another NAS.

 

[Ifti]

My suggestion is that you never trust anything to a sole copy of your data. RAID is not a backup, nor does it make things more reliable. All it is about is data availability in the face of an inevitable hard disk failure. Before you expand the array or replace a disk, you need a copy. Either exported to external disks, or another NAS.

Sound advice

 

[murphychris]

Synology, and some other products out there, are leveraging enterprise level technology. But the catch is that they can't compel consumers to leverage enterprise level best practices as well. Consumers will just expect a product they pay for with a particular feature to "just work". But that's not how enterprise avoids data loss and corruption - they avoid it through using the best technology they can, and best practices. In this case, that's replicating the data, backing it up, and archiving it. Three separate things. You would never find any enterprise, spending millions of dollars on very high end hardware and software development, do a volume group or array expansion without multiple backups. Ever.

Storage is a bitch, pretty much.

 

[murphychris]

Does the network speed matter if I was to be workin on large photoshop files over the network?

As for local network speed, GigE has been used for live editing of non-HD video. So it seems like you should be able to do the same thing with .psd/.psb files. But my understanding is Adobe doesn't support this workflow, at least for Photoshop. My best guess is that the results are mixed because networks for video editing are very rigorously put together and tested. As or more rigorous are data center networks. Whereas home, home office, and even most businesses with design departments don't have such rigorously configured or tested networks.

If you attempt live editing, I personally would disqualify WiFi, and I'd also suggest you use derivative files (make a copy work on the copy). The potential for corrupting the original is just too high.

Best practices though is still to copy the file off the NAS, edit the file locally, and then push the renamed derivative file back to the NAS. If you don't rename, and overwrite, you risk corrupting the lone copy on the NAS if the network copy fails for some reason. And not all failures come with warnings (see silent data corruption, which happens on networks as well as drives). So you're better off making derivatives, and cleaning them up later when you determine you don't need them.

 

[murphychris]

Possibly a useful discussion on this, not quite 1 year old.

http://indesignsecrets.com/editing-indesign-files-directly-off-a-server-crazy.php

Again, it *should* work if the (wired) network is reliable, as video people do this as their standard workflow and they are pushing/pulling way way more data. But because of this, any glitches cause MAJOR problems for them, so they are assured of very high quality components: all runs meet the IEEE requirements for max length, max bends, minimum permissible radius for those bends, minimum safe distance from power sources and transformers like fluorescent ballasts, and properly punched panels. Inordinate numbers of networks just have cables on the floor. Not OK. Pinched, bent, even a stapled cable is not OK.

For my craptastic home cabling setup (ratty CAT 5e cables, but they are short), I can routinely push "big" 10MB+ files over 100+MB/s over NFS, to/from Mac OS X and a Linux box (pretty much out of the box Fedora setup, very little tuning). So I know it can be done...

 

[LERsince1991]

Some very interesting discussions and many thanks for the posts!

Q/ Could I ask what is the difference between a data server with external HD's connected to it and a NAS?

Scrap the idea of live editing of files, was just a thought! - I will continue to keep live projects on my local storage until the job is done then I compress the folder and copy it to my archive external drive.

 

[murphychris]

Q/ Could I ask what is the difference between a data server with external HD's connected to it and a NAS?

NAS is designed to hold drives internally, or in an expansion enclosure, as it's specifically designed for the task of serving files. NAS also usually offers a wide assortment of protocols, with straigtforward enabling and configuration, like http, ftp, scp, NFS, AFP, SMB, and so on. Whereas these things may be in very disparate form on other kinds of general purpose servers.

 

[LERsince1991]

My girlfriend is an interior designer at a high end interior design office which specialise in offices. They provide her with a laptop so she can work on a number of live projects away from the office. She seems to sync her laptop with the server at the office every hour or so over the internet in her house or anywhere else of course. - same with the other 3 designers. They edit autocad, photoshop, 3D Sketchup files etc...

Then in their office they have about 4 PCs set up on the network in their office which connect to the server to access the files as well but I am not sure if these work on the files directly to the server or do the same sync thing the laptops do.

Just some info there that might be useful for the discussion.

Thinking about a few ways of approaching it in theory it would be good to have a sort of separates system like in a HiFi setup. You can then change the speakers, amplifier, processor etc... when you want/need providing a good way to expand and upgrade. Lending this method to the data storage solution could be something like separating the box the controls, serves, manages and formats the hard drives, and separating the hard drives from this. This could allow the hard drives to be stored in separate enclosures for easy upgrading/expanding at a fairly low cost each time rather than a large payout for another multi bay enclosure/nas which would normally be the case. However I am not sure how this would work in terms of software or connections. I am also not sure what the best way to manage the drives in terms of volumes/raids and backups would be.

A simple example could be an airport extreme base station with drives connected via a USB hub and shared over the network. single enclosures can be added individually to provide expandability of say 2tb a year at a manageable price instead of larger lump sums. I will probably need 1tb extra usable storage a year, then account for an extra backup of that data would be looking at 2tb a yr on average.

 

[LERsince1991]

So I'm thinking since I am saving/waiting for a Mac Pro, I could directly connect single drives directly to it and then add more if needed later on. I could share the drives via the mac pro over the network to a macbook air. Also maybe think about sharing the media around the house, maybe to 2 TV's via xbox or samsung smart ability if possible.

I've added an example below.
I don't like the idea of a NAS because they seem limited in expandability and usability to me. Directly connected drives shared would be much for flexible for future use, be extra fast for serious work and also allow the mac pro to back up the storage array through time machine, something that apparently macs can't do with networked drives.

How does this sound? Also how would it compare to the idea before of using an airport extreme instead of the Mac Pro or mini or NAS?

 

[jcpb]

So I'm thinking since I am saving/waiting for a Mac Pro, I could directly connect single drives directly to it and then add more if needed later on. I could share the drives via the mac pro over the network to a macbook air. Also maybe think about sharing the media around the house, maybe to 2 TV's via xbox or samsung smart ability if possible.

I've added an example below.
I don't like the idea of a NAS because they seem limited in expandability and usability to me. Directly connected drives shared would be much for flexible for future use, be extra fast for serious work and also allow the mac pro to back up the storage array through time machine, something that apparently macs can't do with networked drives.

How does this sound? Also how would it compare to the idea before of using an airport extreme instead of the Mac Pro or mini or NAS?

Huh?

I can backup via Time Machine to a network share on my NAS just fine.

 

[murphychris]

I don't like the idea of a NAS because they seem limited in expandability and usability to me.

A desktop has the exact same expandability limitation. A NAS can certainly accept external disks via eSATA if you don't mind the ensuing clutter.

Directly connected drives shared would be much for flexible for future use, be extra fast for serious work.

A flexible headache perhaps, as you have an icon on the desktop for each drive you plug in. And they will not be all that fast unless you use (e)SATA or Thunderbolt. Further you're stuck with the limitations of HFS+ which is not a file system I have much trust in - it offers nothing in the way of checksumming metadata or data on the disk, it defers entirely to the disks's ECC. Bit rot, and silent data corruption are real problems, in particular in the amount of storage you appear to be desiring in your diagram.

and also allow the mac pro to back up the storage array through time machine, something that apparently macs can't do with networked drives.

Untrue. Time Machine support over a network is a function of AFP, and BSD and Linux NAS's almost all have recent versions of Netatalk to provide AFP and network Time Machine support.

Also how would it compare to the idea before of using an airport extreme instead of the Mac Pro or mini or NAS?

Well it's a little bizarre that you'd complain about the lack of expandability and performance of a NAS compared to direct attach storage, and then ask about by far the slowest and least expandable option imaginable.

 

[smellalot]

I've been follow this thread till now and found it quite interesting (though your problem doesn't apply to me)

The cheapest thing you could probably do is to do a software JBOD, you can find information about that there: http://systemsboy.com/2006/04/apple-software-raid-tests-or-what-to-do-if-you-dont-have-an-intel-mac.html
That way if your array becomes too small, you simply add another drive and you are good to go.
It seems to be quite dangerous though, because one failed drive and the whole array is worth ****.

So two NAS or SAN units (one for backup) seem to be the best thing you can do. That is IF you can just replace one or two of the drives with bigger drives when you need more space. Is that possible on a 5bay Synology, without rebuilding the whole array?

 

[murphychris]

It seems to be quite dangerous though, because one failed drive and the whole array is worth ****.

I have not expanded a concatenated array on Mac OS X, but this is usually a two part process. One is adding the disk to the array, making the array larger. But the file system also needs to be resized to use that additional space. And my experience with HFS+ has been hit or miss on successful resizing. More than 1/2 the time, using command line fsck_hfs -r to force rebuild the catalog will then allow such a volume to be resized successfully. This rebuild option is not available in Disk Utility.

So two NAS or SAN units (one for backup) seem to be the best thing you can do.

For sure NAS meaning file-based access, rather than SAN or block-level access. There are open source alternatives to Apple's XSan which is now free in Lion and Lion Server, but I can say how well supported they are on Mac OS X. XSan though still requires fibre channel. Not cheap or easy to lay that down, and it requires at least one, preferably two, metdata servers. I'd so much more quickly use a 10GigE NAS running a Linux/XFS or OpenIndiana/ZFS solution if I needed something like this.

That is IF you can just replace one or two of the drives with bigger drives when you need more space. Is that possible on a 5bay Synology, without rebuilding the whole array?

It can't be arbitrary. You'd need to look at what mdadm can do since it appears that's what Synology are using. Some kinds of resizes are possible without rebuilding, and others aren't. I've done RAID 1, 5 and 6 resizes without rebuilding but always with bigger disks than what was already in the array. Some resizes don't actually get you bigger arrays until all disks are replaced.

No doubt it's possible to download a user manual for Synology and find out what it can and can't do. There's also practical user interface limitations, even if the command line can do it, some of the logic gets quite complex and defies even well done UI. Safer to just make it a non-option.

 

[LERsince1991]

Ok Thanks for the discussion.

Can't backup using TM
What I meant by saying Macs cannot backup the network storage using time machine is that apparently a mac cannot backup the data held on the network storage to a time machine in any way. NOT whether the mac can backup to a network storage using time machine, obviously this is just like time capsule. My concern was how do you backup the network data if its all connected to the network and not directly to the mac?

Mac mini/pro Server vs Airport express + HDD's vs NAS
Performance and expandability
I am still unsure what the difference really is here. Since I am saving for a mac pro I might as well use that with directly connected external drives, to share them over a network to a macbook air, TVs and xbox.
Also the way I see it the mac pro or mini has unlimited expandability and access to a lot of software to manage the drives. Airport express is fairly limited and probably too slow. NAS is very limited in long term because although you can get multiple bay enclosures and some you can add another expansion onto it, theres a limit. I just like the freedom of normal drives shared over network via mac pro. Theres also a smaller pay out for each expansion rather than buying an expensive multi bay enclosure. I must admit I had a NAS once and it was awful and such a waste and i am reluctant to get another hence the want to get something more normal but just network share it.

Software expandability
It seems like this is a bit of a grey area, although its easy enough to make a raid or jbod array, no one is sure of a good way to expand it once its been set. This must be possible though. In my architectural practice we all work directly off a network and its easily fast enough even for big photoshop files. But the disks seem to expand and always have the same free space. I'll have to go snoop around the server room. It might even be better to keep the drives separate as then if one fails I can just restore that drive to a new one from a TM backup without complications on raid or jbod arrays, unless some raid arrays can do this well enough and efficient enough to the point where it doesn't cost twice as much per tb!

p.s. whats wrong with HFS+? I have Mac OS X Extended file system on all my drives and haven't seen any issues. I thought it was the best file system?