Lots of media, 4 hard drives need a solution

I understand your concerns about using RAID. I can't quite wrap my head around it. I fully understand what RAID 5 is, and how it protects my data. But it sure seems like if something goes wrong when rebuilding the RAID in case of drive failure, that everything is lost. I have so much anxiety about losing data that I backup my backup drive. I almost would rather a 4 bay enclosure (like the MediaSonic), with 4 independent drives in it that I can manage and do what I want on them.

I just feel like if I have a 4 bay Synology, with 2 TB drives giving me about 5 TB in RAID5, that I stand to lose a lot of data if something goes wrong. How can I backup the data on the Synology? I understand that it has a USB port for external drives, but is there any easy way to hookup multiple USB drives to the Synology and have it automatically backup to those drives too?

AppleDApp said:

Folders seem like a good option. Definitely a lot easier to setup.

Click to expand...

Great.

How do I link my media folder to itunes and plex? For itunes do I change the iTunes media folder location?

Click to expand...

In iTunes change the default folder location. Preferences-->Advanced. Point to the new folder of your media.

This is confusing could you elaborate?

Click to expand...

Basically iTunes does not allow you to move files to your iDevice from a server configuration. So if you have an iPhone hooked up to your client computer you have to the music files loaded up in the client computer. Not stored on the computer but loaded from the NAS device. This is one thing that drives me nuts and I have not been able to find a work around for it.

How do I point iPhoto to the photo folder on my nas?

Click to expand...

You can do this a couple of different ways. You can move the default iPhoto library to the nas device. Then when start up iPhoto hold down the "option" key and a menu will pop up asking you to create or choose other library. See screen shot, choose other and point it towards your library you moved to the nas. Before you do anything with this, make sure to back it up to another harddrive either internal or external. Always back up before you do any moving etc.

is there anyway that files download here and once the download is complete they go to another folder (video or music) on the NAS?

Click to expand...

You could probably do a automater script for this but I do not have any experience with this sorry.

I have an iMac (mine) and a macbook pro (dad's) it's probably best to setup a Time Machine folder for each user. How do I set this up? Are there any downfalls?

Click to expand...

Once you have setup the folders and shared them so you can see them under the harddrives, go to each computer and open up the system preference--> time machine. You will see "select disk" navigate to the folder that is for that computer and that is it. (See screen shot)

I don't know much about RAID, which is best or my case. I want the most capacity but also want some failure protection.

Click to expand...

I would do a little more research into this if I was you. You can't really have both. There are pro's and cons to each. I have mine in a raid 5 setup and I have a usb drive attached that backs up only those important files that I want more than just the NAS redundancy. i.e. photos, documents and media.

Hot swap is definitely great. Do you have experience with the synology web based UI? Thank you very much All this is very helpful.

Click to expand...

No I do not sorry. I use QNAP.

radiogoober said:

I understand your concerns about using RAID. I can't quite wrap my head around it. I fully understand what RAID 5 is, and how it protects my data. But it sure seems like if something goes wrong when rebuilding the RAID in case of drive failure, that everything is lost. I have so much anxiety about losing data that I backup my backup drive. I almost would rather a 4 bay enclosure (like the MediaSonic), with 4 independent drives in it that I can manage and do what I want on them.

I just feel like if I have a 4 bay Synology, with 2 TB drives giving me about 5 TB in RAID5, that I stand to lose a lot of data if something goes wrong. How can I backup the data on the Synology? I understand that it has a USB port for external drives, but is there any easy way to hookup multiple USB drives to the Synology and have it automatically backup to those drives too?

Click to expand...

Backing up a backup drive is not wise. Any data corruption, or systematic program error would just propagate through to the second backup. You would be much better off running a separate independent backup on your own primary data.

I assume that you are using some type of cloud backup. That is probably the single most important thing you can do to protect your data. I personally would recommend using cloud in conjunction with local backup so that you can do a quick recovery if necessary.

/Jim

AppleDApp said:

Is there any downfall to moving my iTunes and iPhoto library to the NAS?

Click to expand...

Personally, I prefer DAS for application libraries. In general, such libraries are not meant to be shared... and if another user on your LAN mucks with them, they could make your own database unstable.

Does your NAS allow DAS modes by connecting via Firewire or eSATA? Personally, I am switching to Thunderbolt.

/Jim

I found this product this AM. I have been strongly considering a MediaSonic 4 bay USB enclosure bc it's about $130 and you add whatever discs you want.

But.... Check this out:

http://store.westerndigital.com/sto...tegory.58727400/parid.13092300/catid.55286600

WesternDigital dual thunderbolt enclosure. 6TB for $700, 4TB for $600. Raid or JBOD. User serviceable discs, etc. You can daisy chain multiple units. This looks like a luxury item for those of us who do not wish to use a NAS.

Just buying the Mediasonic and two nice 3TB discs is almost $500. And USB is slow as hell.

flynz4 said:

Backing up a backup drive is not wise. Any data corruption, or systematic program error would just propagate through to the second backup. You would be much better off running a separate independent backup on your own primary data.

Click to expand...

While true, you're talking about a very different backup strategy to mitigate this problem than most people can deal with. Even if you have primary data, and three independent backups, most people recycle those backups rather than archive. Therefore any data corruption, (bit rot, silent data corrupt) on the primary disk is eventually propagated onto backups. Most people migrate to new computers and bigger hard drives by updating their backups, removing their primary disk, and restoring from one of the backups ensuring any backedup corruption has just been restored to the new disk. And as new corruption develops, that's backed up to all three independent backups.

This is why I like a ZFS solution for NAS, like FreeNAS/TrueNAS, Nexenta, OpenIndiana, etc. All data is checksummed, corruption is independently detectable and immediately corrected. And there is a replication feature. So you can backup a backup and both will contain identical data both of which are checksummed with a means of ensuring data integrity (including from bit rot and silent data corruption).

So you could have a 2nd NAS either on-site, or at a completely different remote location, and use the replication feature to produce a replica backup.

Another thing I like is RAIDZ single parity does not have the write hole problem that exists in RAID 5 (which is also single parity). You can have a case with RAID 5 where parity becomes corrupt, but there is no mechanism for this to be detected or corrected. Upon rebuilding a failed drive is the first indication of the problem and then there's data loss even though a 2nd drive hasn't failed. This is pretty much why RAID 6 is used for truly important data as well as high(er) availability. RAID 5 marginally increases data availability but does nothing for data integrity and definitely isn't a backup. I'd sooner use RAID 1. Single disks can now saturate GigE so the advantage of RAID for performance is slim to none, unless you have a 10 GigE network.

----------

The problem with DAS, is I simply don't trust JHFS+/X as a file system. It defers 100% of error detection and correction to the individual disk firmware, it does nothing to ensure the file system metadata or data reported by the disk is actually correct. Same situation with NTFS on Windows.

So the DAS, with JHFS+/X I consider strictly for fast data access, not a large volume of data and definitely not as a backup. I'd do Thunderbolt or eSATA for the fast connection. I see no advantage of Firewire 800 because, again a single disk can saturate it now, meaning the connection is the bottleneck not the hard drive.

AppleDApp said:

I do not currently have a NAS. I've seen many instances where iTunes libraries are shared across a network. It shouldn't be an issue.

Click to expand...

Sharing iTunes data across a network is NOT the same as having an application's database on a Network. I am not saying that it cannot be done... but the path is littered with failures of people who have tried but ended up with severe database problems.

It's your data... do what you want.

AppleDApp said:

I already have four drives. While thunderbolt is fast, it is not a network drive and cost is high.

Click to expand...

Choosing a storage strategy based on the fact that you already have 4 drives is a bit short-sided. The physical drives are a dime a dozen and are the least critical part of your storage strategy.

IMHO... DAS is superior to NAS for many, if not most applications. You can always use a DAS as a NAS by sharing data. You cannot take a NAS and make it into a DAS.

In fact, IMHO, even if you have a legitimate application that requires networked shared data (the prime reason for a NAS)... I think that computer based sharing is often a superior NAS because you have the complete set of tools of your client to manage and backup your data.

TB is indeed still somewhat expensive... but that is already changing fast, and the results are quite stunning. I acquired a Promise Pegasus R4 as an experiment, and I am delighted with the system. I am seriously considering to use it as the primary library storage for my new iMac once it is released.

For NAS... I am considering buying a MacMini Server... adding DAS to it, and then using that for sharing media over my network. One of the advantages is that I can add the MacMini into my current offsite backup strategy, since it is essentially "just another Mac". That is the biggest problem that I have with any NAS that I have ever purchased. Backing them up was always either impossible, or unstable. I currently have 4 perfectly good NAS boxes that are all powered off. I doubt that I will ever use them again.

/Jim

flynz4 said:

Sharing iTunes data across a network is NOT the same as having an application's database on a Network. I am not saying that it cannot be done... but the path is littered with failures of people who have tried but ended up with severe database problems.

Click to expand...

Many apps now have local file system dependencies like ~/Library/Application Support. And I don't see a way around that. But since the very beginning Mac OS X has supported the idea of a Network domain for applications, fonts, etc. Just as you have a user domain for applications:
~/Applications

There is an "all users" domain:
/Applications

And there is a Network domain:
/Network/Applications

It may be that the network location has a dependency on NFS rather than AFP, as NFS works considerably different than AFP.

IMHO... DAS is superior to NAS for many, if not most applications. You can always use a DAS as a NAS by sharing data. You cannot take a NAS and make it into a DAS.

Click to expand...

Not entirely correct. Most all NAS products support iSCSI and NFS, and in either case they behave as locally attached storage as far as the OS is concerned.

The only real advantage of DAS is speed. A DAS on a user computer requires that computer to be on to share anything on that DAS to other users. Major disadvantage.

In fact, IMHO, even if you have a legitimate application that requires networked shared data (the prime reason for a NAS)... I think that computer based sharing is often a superior NAS because you have the complete set of tools of your client to manage and backup your data.

Click to expand...

I don't know what this means. Every NAS I've worked with has replication features for backup to another NAS, or DAS attached to the NAS (it's local as far as the NAS box itself is concerned, and usually is not sharable on the network) using rsync or file system specific replication features.

TB is indeed still somewhat expensive... but that is already changing fast, and the results are quite stunning. I acquired a Promise Pegasus R4 as an experiment, and I am delighted with the system. I am seriously considering to use it as the primary library storage for my new iMac once it is released.

Click to expand...

See, and I just don't trust JHFS+/X as a file system, anymore than I do NTFS. I consider DAS storage for fast access to data, scratch space, temp or working space, and for the OS and apps (things that are easily replaced).

That is the biggest problem that I have with any NAS that I have ever purchased. Backing them up was always either impossible, or unstable.

Click to expand...

That's a little vague. Again, every NAS I've worked with at a minimum has rsync to sync backups, or even snapshots if the filesystem supports them. How do you propose a Mac Mini improves backup reliability to either locally attached storage, or to a remote location? I don't see how AFP improves the reliability at all. The most likely tool to use for remote backup of a Mini would actually be rsync over ssh, even with a Mini, but then you don't get the benefit of a modern file system (yet).

AppleDApp said:

Is there any downfall to moving my iTunes and iPhoto library to the NAS?

Click to expand...

I have yet to find one. Expandability and backup in one little box.

Others will chime in and let you know how they feel about it. But there pro's and con's to anything you setup. Anything.

rhett7660 said:

I have yet to find one. Expandability and backup in one little box.

Others will chime in and let you know how they feel about it. But there pro's and con's to anything you setup. Anything.

Click to expand...

The negatives of NAS in a nutshell: If you like being mobile, your stuff isn't available the same way inside and outside of the network. Most people work differently also, and most of the time this means you probably won't need direct access to iTunes/iPhoto content away from the network. And also, NAS performance is very contingent on the local network quality. And a lot of people definitely do not have best practices when it comes to network cable quality, lengths, bends, and location relative to power and ballasts. If you pinch a network cable under a sofa? It's normal to get 1/2 the performance. I'd even say it'd be unsurprising if the network connection were flakey.

So yeah it's a fair point that there's a pro and con to anything. DAS is simpler because it lacks the network layer. A NAS in a sense has DAS which it then makes available over a network, so you have complications of both.

For some people a NAS is about reliability. Others it's about bulk inexpensive storage. Others it's a way to hide that storage in a closet rather than on a desk. And still others it's about sharing that bulk storage, whether segregated by folder privileges, or share/mount points, or logical volumes.

murphychris said:

The negatives of NAS in a nutshell: If you like being mobile, your stuff isn't available the same way inside and outside of the network. Most people work differently also, and most of the time this means you probably won't need direct access to iTunes/iPhoto content away from the network. And also, NAS performance is very contingent on the local network quality. And a lot of people definitely do not have best practices when it comes to network cable quality, lengths, bends, and location relative to power and ballasts. If you pinch a network cable under a sofa? It's normal to get 1/2 the performance. I'd even say it'd be unsurprising if the network connection were flakey.

So yeah it's a fair point that there's a pro and con to anything. DAS is simpler because it lacks the network layer. A NAS in a sense has DAS which it then makes available over a network, so you have complications of both.

For some people a NAS is about reliability. Others it's about bulk inexpensive storage. Others it's a way to hide that storage in a closet rather than on a desk. And still others it's about sharing that bulk storage, whether segregated by folder privileges, or share/mount points, or logical volumes.

Click to expand...

Heck people practicing back up tend to be a bit flacky. I do the three locations. Two local, NAS box and attached USB box to the NAS. And off site for important can't loose documents.

NAS have improved somewhat as I am able to now access my files outside of my network which is nice.

I am coming from a four disk box that was setup as JBOD that had a script run that copied the same files to each drive or set of drives. But I always had a back up to the back up. I am to paranoid about loosing data as I have had that happen. Which cost me a lot of money at the time to get it back as I was unable to recover anything with the tool set I had. Drive disassembled and plates read for the info and the data was recovered that way. Pain in the butt and I learned my leason.

rhett7660 said:

Heck people practicing back up tend to be a bit flacky. I do the three locations.

. . .

But I always had a back up to the back up. I am to paranoid about loosing data as I have had that happen.

Click to expand...

Yeah but have you had data loss through corruption? And have you ever found all of your backups were corrupt because, by logical extension, they were all current backups and hence backups of corrupted data? One way of dealing with this is the archive, which is a non-current version of the data. If I had a rolling archive, at some point back in time, the file would not be corrupt. As this particular file was read, but not written to, it would have been adequate.

And that's the complaint I have about file systems not really helping us with this problem. They defer error correction and detection entirely to the hard drive firmware's ECC. Everyone knows that unrecoverable errors occur, and that bit rot occurs, and so does silent data corruption.

Therefore I'm fond of Nexenta and also FreeNAS, both supporting ZFS as the native file system for data. And for your backups of the backup, both support replication via periodic ZFS snapshots which are then exported via rsync to another NAS which then institutes that snapshot. And all of it's checksummed. So even if there's corruption, it's detected and corrected for with known good copies.

And while it's not a plug and play thing to learn about snapshots and rsync and zpools and ssh if you're doing remote rsyncs, if it's really important data, then it's worth it. Because it's not more difficult or time consuming to get what I'm talking about automated, compared to the typical user backup which is very manually driven. In particular the backups of backups.

murphychris said:

Yeah but have you had data loss through corruption? And have you ever found all of your backups were corrupt because, by logical extension, they were all current backups and hence backups of corrupted data? One way of dealing with this is the archive, which is a non-current version of the data. If I had a rolling archive, at some point back in time, the file would not be corrupt. As this particular file was read, but not written to, it would have been adequate.

And that's the complaint I have about file systems not really helping us with this problem. They defer error correction and detection entirely to the hard drive firmware's ECC. Everyone knows that unrecoverable errors occur, and that bit rot occurs, and so does silent data corruption.

Therefore I'm fond of Nexenta and also FreeNAS, both supporting ZFS as the native file system for data. And for your backups of the backup, both support replication via periodic ZFS snapshots which are then exported via rsync to another NAS which then institutes that snapshot. And all of it's checksummed. So even if there's corruption, it's detected and corrected for with known good copies.

And while it's not a plug and play thing to learn about snapshots and rsync and zpools and ssh if you're doing remote rsyncs, if it's really important data, then it's worth it. Because it's not more difficult or time consuming to get what I'm talking about automated, compared to the typical user backup which is very manually driven. In particular the backups of backups.

Click to expand...

Yup this is exactly what happened. As it was backing up to another drive the corruption happened. As a result my backup was corrupted and my backup of my backup was corrupted. This was before I did the three with some of the provisions you have mentioned here. i.e. rolling archive.

rhett7660 said:

Yup this is exactly what happened. As it was backing up to another drive the corruption happened. As a result my backup was corrupted and my backup of my backup was corrupted. This was before I did the three with some of the provisions you have mentioned here. i.e. rolling archive.

Click to expand...

Right. And actually, unless and until we have a more resilient file system on Mac OS, we don't actually (automatically) know the source data going to the backup isn't corrupt. So we can ZFS ourselves blue in the face, but a source file that's already corrupt will be assured of being precisely corrupt in the same fashion, on a ZFS based backup.

murphychris said:

Not entirely correct. Most all NAS products support iSCSI and NFS, and in either case they behave as locally attached storage as far as the OS is concerned.

Click to expand...

I agree completely. I think few consumers use these either in iSCSI or NFS configurations.

murphychris said:

I don't know what this means. Every NAS I've worked with has replication features for backup to another NAS, or DAS attached to the NAS (it's local as far as the NAS box itself is concerned, and usually is not sharable on the network) using rsync or file system specific replication features.

Click to expand...

Both examples that you list are local backups. The most fundamental principle of my backup strategy is to have at least two totally independent backup solutions... one locally, and one offsite. Both must be fully automatic. If I understand your proposal correctly, you would only be making local backups.

I also know that some NAS boxes do support offsite cloud based storage as well... but whenever I examined them, they seemed outrageously expensive. By sharing data from a Mac Mini (as an example)... the mini is capable of performing any multitude of automated backups. It is not limited to the backup provided by the NAS manufacturer.

/Jim

flynz4 said:

I agree completely. I think few consumers use these either in iSCSI or NFS configurations.

Both examples that you list are local backups. The most fundamental principle of my backup strategy is to have at least two totally independent backup solutions... one locally, and one offsite. Both must be fully automatic. If I understand your proposal correctly, you would only be making local backups.

I also know that some NAS boxes do support offsite cloud based storage as well... but whenever I examined them, they seemed outrageously expensive. By sharing data from a Mac Mini (as an example)... the mini is capable of performing any multitude of automated backups. It is not limited to the backup provided by the NAS manufacturer.

/Jim

Click to expand...

This price has come down a lot. $400.00 for a four disk nas box with cloud storage is the price range currently. My QNAP that I just purchased already has tools in place to do just this with automation.

I think it is safe to say. One solution is not the best especially for back up. However having a back up strategy is still better than having none at all.

flynz4 said:

I agree completely. I think few consumers use these either in iSCSI or NFS configurations.

Click to expand...

That's a different argument though. You said a NAS as DAS cannot be done, in the context of shared data and apps (being treated locally). There are a couple of tricks getting NFS setup, but it's at least built into the OS.

Both examples that you list are local backups. The most fundamental principle of my backup strategy is to have at least two totally independent backup solutions... one locally, and one offsite. Both must be fully automatic. If I understand your proposal correctly, you would only be making local backups.

Click to expand...

You understand incorrectly. Replication features in NAS support replication either to storage directly attached to the NAS, or to network storage on the local network, or to off-site remote storage, or any combination there of. All automatically. The file systems are completely independent in any case. In two of three examples, the hardware is also independent.

Doing this with DAS is vastly more expensive in comparison to NAS, because the hardware is more expensive. A significantly capable NAS box is a few hundred bucks compared to 2-4x that cost for a Mac Mini that does the equivalent job with far fewer features: no RAID 5, 6, or Zx; no SMART monitoring built in; no eSATA to inexpensively connect multiple drives; no UPS monitoring; no built-in replication features; no modern file system that ensures data integrity once it's on the NAS. Etc.

I also know that some NAS boxes do support offsite cloud based storage as well... but whenever I examined them, they seemed outrageously expensive.

Click to expand...

FreeNAS is free. Nexenta below 17TB of storage is free. The features I'm referring to are included as both use ZFS as a file system. Both support automatic (periodic) snapshots which can then be rsync'd to a local (LAN) or remote (WAN) replica of the same system where the snapshot is imported. This is all exposed in the web based GUI.

By sharing data from a Mac Mini (as an example)... the mini is capable of performing any multitude of automated backups. It is not limited to the backup provided by the NAS manufacturer.

Click to expand...

That's really vague. You've offered nothing a Mac Mini can do that a NAS cannot. I've offered quite a few features available in free NAS products (not free hardware but free software at least) that beat the pants off a Mac Mini when it comes to backup solutions.

Every single one I've looked at supports the latest version of AFP, supporting Time Machine as documented by Apple, via Netatalk. So if you really wanted to do Time Machine backups to a NAS, you can do that.

----------

rhett7660 said:

I think it is safe to say. One solution is not the best especially for back up. However having a back up strategy is still better than having none at all.

Click to expand...

As we (including me) get carried aware on myriad backup features, it's important to remember this. And I think on par with the importance of having a backup strategy is having a rehearsed restore strategy.

I'm in the exact situation as this.

I have 4 or 5 hdd's ranging from 500gb to 2tb that are sitting around with different data in each of them and would love to unite them all into one so that all my devices can read and write onto them (if not write, at least read).

My fiancee and I are an apple family so between us we have my MBP, her MBA, 2 iphones, 2 ipads and an apple tv.

Not that I'm trying to list what toys we have, but what I'm trying to achieve is that all these can have access to the data such as our back-up movies (I ripped all our dvd's and b/rays into digital format) as well as our ripped songs and photos etc.

eg I would love to be able to watch top gear on the tv and allow her to watch gossip girl on her laptop, and also allow both of us to access our photos for editing and listen to music etc.

I've done a bit of research into NAS and am willing to invest in a good 4 or 5 bay enclosure. I'm also open to the idea of DAS using either a Mac Mini (have always wanted to buy one as a Media center attached to my tv) or using my MBP (I'm planning on buying a new 15" when they get refreshed) as a server. I have no problem with leaving a computer on somewhere to download files and/or synchronize my data.

However I'm thinking long-term, I want expandability (I go through a heck of a lot of data - we both love taking and editing photos, I love watching my movies, tv-shows etc) and I want safety (I really need to backup important things such as photos, work files etc (I've gone through the pain of losing thousands of photos so don't want to do that again). Movies etc I can either re-rip or re-download so don't really care about them.

What's the best solution out there? Thanks for all the ideas so far - keep them coming! They're all v.useful but I'm kinda just getting more confused lol