Lots of media, 4 hard drives need a solution

[flynz4]

Originally Posted by murphychris

It increases complexity, increases the chances you'll encounter a problem you have to work through: determine if it's really a problem, and if so, what the work around is. It probably reduces the likelihood of total data loss, however. It's a catch-22, so the strategy should emphasize secondary storage that's low in complexity, even if low in features, so long as it doesn't compromise base requirements.

Do you really think it increases complexity? CP+ is one of the simplest things I have ever set up. Setting up Time Machine is even 1% more simple.

Personally, I think the combination of CP+ and TM is a great combination. Simple, secure, cheap... and you end up with two totally independent backup programs that are totally hands off... with one backed up automatically offsite.

/Jim

[radiogoober]

Originally Posted by flynz4

Do you really think it increases complexity? CP+ is one of the simplest things I have ever set up. Setting up Time Machine is even 1% more simple.

Personally, I think the combination of CP+ and TM is a great combination. Simple, secure, cheap... and you end up with two totally independent backup programs that are totally hands off... with one backed up automatically offsite.

/Jim

Hey Flynz,

I was thinking about something you wrote a few days ago. You mentioned possibly getting a Mac mini as a NAS/backup server type thing. It's a pretty interesting idea, and it's actually relatively cost effective compared to some other options. The mini (as we all know) as FW800, 4xUSB2, and a thunderbolt port. I was considering getting the cheapest mini, and another WD MyBook Thunderbolt Duo (2x3TB) and leaving it as JBOD. Each disk would be a network share, so it'd be two lightning fast 3 TB shares. Since it's a Mac, it'd be completely painless to setup CrashPlan on it, and let it automatically keep a copy of everything far away on their servers (or just let crashplan run the main computer.) One share would be a backup for all my media, and the other share a timemachine destination. Everything could be controlled, as needed, with Screen Sharing.

Cost wise, it's almost the exact same as the other solution I was looking at:
Mac mini - $600
WD MyBook Thunderbolt Duo - $700
= $1300

vs.

Synology RS812 - $700
4 x 2TB drives + 1 spare = ~$600 or so
= ~$1300

For the Mac mini setup, money could be saved by using USB2.0 external drives, and since it's for backup and not for streaming, the speed wouldn't be terribly important.

The thing that's pointing me towards the Mac is that, well, it's a Mac, so it's a full featured computer and is super powerful, and I'm sure that I would realize other neat things it could do as a full-featured, network attached computer. It really could do everything the synology does, plus more. It easily could by an iTunes server (one that *could* stream to AppleTV, whereas Synology can't), etc.

The thing that points me towards the Synology is that its easily expandable with the RX410 unit, and that it all can be easily rackmounted and look nice and cute (I know the Mac mini and HD's could be rackmounted easily... but I don't see any solution to rackmount multiple HDs for the mini, so it'd just be on a shelf, which I guess really doesn't matter when it's all in a closet in a room far, far away..)

I just guess, for the price, the Mac mini could be a fantastic server.

[radiogoober]

Holy crap, I almost considered spending $1300 just to get a "cool looking" rack mount storage setup that would only give me 3-4 TB of storage (being redundant). Screw all that. I just ordered an external FW800 3TB drive for $269, (which should be wayyyy faster than USB2 - in all my years I've never owned a FW device.)

So I'll now have two complete copies of my important media, along with APE/TM for revisions of other data/documents. The only thing left is to bite the bullet with CrashPlan+, get the seed drive, and start copying.

Maybe one day in the future I'll do the rackmounted backup solution, but it just seems extraordinarily expensive, and I just don't have any use for most of the stuff it can do. My wife only uses her MacBook to print recipes and grocery lists, and my Mac mini is already backed up in triplicate, I just can't justify the utility of a NAS or any other type of thing at this point.

[murphychris]

Originally Posted by flynz4

Do you really think it increases complexity?

That particular comment wasn't directed at Crashplan, but a general statement. I think anytime you add another layer, complexity is increased. But as you pointed out you're also distributing risk. So the goal when deciding you need more risk distribution, is to go with the least amount of complexity to mitigate that risk.

CP+ is one of the simplest things I have ever set up. Setting up Time Machine is even 1% more simple.

It's true, but limited. You can only backup an amount of data equal or less than the size of the target Time Machine disk. Works for probably 80%. But the other 20%, it's a non-option. It does work for my laptop, and I've extensively tested its restores, compared to other choices. So I'm OK with it, but I don't entirely trust it for the only local backup because I've had corrupt itself beyond repair, several times.

Personally, I think the combination of CP+ and TM is a great combination. Simple, secure, cheap... and you end up with two totally independent backup programs that are totally hands off... with one backed up automatically offsite.

In my opinion Time Machine lacks sufficient reliability to be the only local backup. You can mostly mitigate this by having multiple Time Machine backups. The chances of two of them going bad at the same time you have a hard drive failure, is pretty remote.

But TM backups are on the same media (HDD), and on the same file system (JHFS+). And being directly attached storage, it's at equally high risk of file system corruption compared to the primary storage being backed up. It's not really that independent, is my point. Power loss of either computer or TM disk, or disconnecting them, or a panic and suddenly the local disk and backup are simultaneously at significant risk of irreparable file system corruption.

And there's still nothing in this equation that helps avoid silent data corruption. The more stuff people have, the bigger this concern becomes. Crashplan, if it weren't tied to the bandwidth of the Internet connection, is actually a better solution than Time Machine. It's a completely different medium, file system, not direct attached, and it's off-site.

----------

Originally Posted by radiogoober

I just guess, for the price, the Mac mini could be a fantastic server.

I'm totally over it with Apple servers. They bring nothing to the table. In fact they bring headaches, not least of which is the lack of server grade stable releases, and at least 10 years of software and security updates. And given the 10.6 to 10.7 changes in server, I have negative interest in dealing with anymore Apple servers.

[ericinboston]

Has the OP solved the problem? If not, I think it would be valuable for him to upload a detailed diagram of what's where and exactly what the problem is right now. Are these all Macs...or PCs too?

Someone commented that USB 2.0 is too slow...well that depends on exactly how much data you are trying to move and what your time goals are. I certainly wouldn't expect to copy 1TB of data over USB 2.0 every day...that transfer alone would take 5+ hours...bogging down the computer, too.

USB 3.0 would help and if you did a bit of research on which USB 3.0 drives are fast/faster/fastest depending, again, that might solve the OP's needs.


Part of the answer may be to simply reduce the number of drives. 4 2TB drives...are you full at 8TB? That's 4 devices hanging off systems which requires system management. Do you need 8TB or more in the upcoming year/s? What about selling your 4 existing drives and investing in some 3TB drives...or 4TB or 8TB drives? http://www.newegg.com/Product/Produc...20and%20higher the investment wouldn't be too bad if you sold 4 drives at $80 a pop and thus had $320 to put towards say a $900 purchase.

If you can give us more info such as why you want/do not want NAS/SAN/External drives, that would help.

[murphychris]

Originally Posted by ericinboston

What about selling your 4 existing drives and investing in some 3TB drives...or 4TB or 8TB drives?

Bigger drives makes data loss more likely.

HDDs are not getting faster at the same rate they're getting bigger. So it's taking longer to rebuild their data when they die. Long enough that a drive containing duplicated data can die in the time it takes to rebuild the first. Hence why there's RAID 6. And why there are distributed file systems that have built-in replication and self-healing features, because managing large storage with just a bunch of disks as individual file systems is inherently risky.

[AppleDApp]

Originally Posted by ericinboston

Has the OP solved the problem? If not, I think it would be valuable for him to upload a detailed diagram of what's where and exactly what the problem is right now. Are these all Macs...or PCs too?

Someone commented that USB 2.0 is too slow...well that depends on exactly how much data you are trying to move and what your time goals are. I certainly wouldn't expect to copy 1TB of data over USB 2.0 every day...that transfer alone would take 5+ hours...bogging down the computer, too.

USB 3.0 would help and if you did a bit of research on which USB 3.0 drives are fast/faster/fastest depending, again, that might solve the OP's needs.


Part of the answer may be to simply reduce the number of drives. 4 2TB drives...are you full at 8TB? That's 4 devices hanging off systems which requires system management. Do you need 8TB or more in the upcoming year/s? What about selling your 4 existing drives and investing in some 3TB drives...or 4TB or 8TB drives? http://www.newegg.com/Product/Produc...20and%20higher the investment wouldn't be too bad if you sold 4 drives at $80 a pop and thus had $320 to put towards say a $900 purchase.

If you can give us more info such as why you want/do not want NAS/SAN/External drives, that would help.

I only use macs for the time being I also have a windows VM. I have four 2tb drives that were previously in my PC. One 1tb drive to backup my laptop and a portable 1tb to carry files around.

USB 2.0 is too slow often times when I move data around I move several gigs at once. Sometimes hundreds of gigs at once. USB 3.0 or something faster is a must.

I am not full at 8tb. All of my data accounts for about 2tb. I have two backups of my media. Currently making a 2nd backup of my laptop. I will not sell my drives because it is all in all a huge pain.

I was considering getting a NAS to backup my computers and store my media but I should also consider a backup solution for the NAS. I am also not sure how to move everything from itunes to the NAS and whatnot.

Originally Posted by murphychris

Bigger drives makes data loss more likely.

HDDs are not getting faster at the same rate they're getting bigger. So it's taking longer to rebuild their data when they die. Long enough that a drive containing duplicated data can die in the time it takes to rebuild the first. Hence why there's RAID 6. And why there are distributed file systems that have built-in replication and self-healing features, because managing large storage with just a bunch of disks as individual file systems is inherently risky.

Recovering a dead hard drive is indeed a pain one of my drive did earlier this week. I've been backing up my computer since. it's probably taken 36 hours or so.

[ericinboston]

Originally Posted by AppleDApp

I only use macs for the time being I also have a windows VM. I have four 2tb drives that were previously in my PC. One 1tb drive to backup my laptop and a portable 1tb to carry files around.

USB 2.0 is too slow often times when I move data around I move several gigs at once. Sometimes hundreds of gigs at once. USB 3.0 or something faster is a must.

I am not full at 8tb. All of my data accounts for about 2tb. I have two backups of my media. Currently making a 2nd backup of my laptop. I will not sell my drives because it is all in all a huge pain.

I was considering getting a NAS to backup my computers and store my media but I should also consider a backup solution for the NAS. I am also not sure how to move everything from itunes to the NAS and whatnot.

Well I still don't know exactly what you are backing up, how large that backup is, and how often. Let me give you my example of my backup strategy for a few TBs worth of data/media/operating system (I'm on Windows in this example):

I have Windows, pix, Word docs, and similar text docs on my 500GB C drive. It usually takes up about 90GB total. On my 2TB internal D drive, I am eating about 1.75TB with gazillions of MP3/audio files as well as movies/video files. Now, I do NOT back up my D drive often. Why? Because a)there's no sense in always backing up the same 1.75TB every week/month and b)my media on it doesn't change daily and c)I'm not in a paranoid state (such as my computer is dying) that I need to back up 1.75TB every night and d)I have a baseline backup and every few weeks I backup about 1GB of new stuff at most. Now let's address each point:

A)Why would I want to back up, say, the same 100 movies every night/week? Once I have a backup it's backed up. Done. Sure, maybe have 2 1.75GB backups if you are really worried. If there are items that you need to back up every night/week, just back up those while keeping your giant movie archive already on the older backup.

B)Does your media chance daily? What exactly are you backing up? Are you using software or some simple scripts to copy the files from place to place?

C)Nothing is forever. You can have the best backup solution and then your house catches on fire...or your 4 year old kid smashes a drive. If you have 1 or 2 backups of your absolutely cannot-lose data, that's fine.

D)Again, I have no idea how much data you are trying to back up and how often.



Solutions:
---------

1)Maybe you should segment your media on different drives...keep audio on 1 drive while video on another drive. This way you are not senselessly backing up EVERYTHING just because you added 5 mp3 files.

2)Maybe you should not be using backup software. For my MP3 backup, I simply use a 1-line command line statement that copies only newer MP3s from drive A to drive B. So this way if I add 44 songs to my drive A, I can run the 1-line command and only the 44 new ones get added to by drive B (which is my mp3 backup). But for my my OS I "image" it every few weeks which compresses the 90GB down to about 50GB.

3)Have you tried Bluray archiving? I bought a fantastic drive for $84 and a spindle of name-brand 30 25GB discs for $30. Now, this of course is really for the permanent backup and to get all your 2TB backed up as a starting point. Let's do the math: 30 25GB discs = 750GB of space for $30. You have 2TB so you would need about 4 packs of 30 discs which would be about $120 for the discs + $84 for the drive. So about $200 for a "permanent" backup solution where the medium cannot accidentally be destroyed. Again, nothing is forever but I would bet my cdrs/dvdrs/blurays over an external hard drive (or internal) any day...especially for things that can happen to drives like electrical spikes, water, magnets, dropping, virus, etc.

4)You state USB 2.0 is too slow...why not use eSATA? Or, forget externals altogether and just use internal SATA drives? You could buy some very nice 7200RPM drives that are extremely fast.

5)I think part of your solution is to consolidate these 4 physical drives into a few 3TB (or larger) drives so you have less stuff to manage and less stuff fighting for USB connections, network traffic, etc.

6)Get yourself a baseline backup (like I have)...say of 1 or 2TB. Then worry about what you want to back up above and beyond that.

7)If you really want to back up 2TB+ a day/week, I think you're going to need some serious drives that cost serious money. 2TB+ is still a LOT of info to back up every night/week for even pro-sumers in 2012. 10 years from it likely will a snap.


I have about 35 blurarys sitting in my $50 fire-proof box...those discs contain over 20,000 320k mp3s that I ripped from my extremely large cd collection...I've never purchased an mp3. If my D drive died tomorrow, I'd simply have to pop in those blurays and/or go back to 1 of my external drives for a restore. Yes, it's not fun the first time creating a 30+ bluray archive...but nobody ever thinks to start backing up Day 1 of your first computer experience.


I think you really need to think about exactly what you are backing up, how large it is, and how often. Let us/me know, too...

[ericinboston]

Originally Posted by murphychris

Bigger drives makes data loss more likely.

Not entirely true. It's all in 1)quality of the drive, 2)safety of the drive (is it on a UPS or just plugged into the wall?) 3)how long/often the drive is actually reading/writing (as well as sitting idle).

Just because a drive is "bigger" or "large" today (say a 2TB drive) compared to a 10GB drive in the mid 1990's (which was large then), the drives are not any worse for reliability. Also, data loss is data loss. But I'd rather lose 1 bad sector on a 5MB mp3 file rather than 1 sector on a giant 6GB video file. Each file would likely be rendered useless but I bet I would lose a larger chunk of "information" with the 6GB file.

Originally Posted by murphychris

HDDs are not getting faster at the same rate they're getting bigger. So it's taking longer to rebuild their data when they die. Long enough that a drive containing duplicated data can die in the time it takes to rebuild the first. Hence why there's RAID 6. And why there are distributed file systems that have built-in replication and self-healing features, because managing large storage with just a bunch of disks as individual file systems is inherently risky.

Right...but it all depends on what the OP has for paranoia, how hard he's really pushing the drives, and how much he is willing to spend for a solution. We can offer him solutions ranging from $300 to $5000. What's his budget? What's his need?

Nobody likes to lose data...and I've lost some over the past 30 years...but as a consumer who is a techie I also rarely back up. 1)I back up truly mission critical stuff like the OS so I can boot, 2)I back up data/information that truly feel I really, really need a backup of and 3)For large sets of similar data (like my 20,000 mp3s), I start with 1 master backup and then I add to it every few weeks.

I also do not believe in backing up and overriting the old backups every night. 1)The process of backing up every night wears on the backup device (unless you spend more money and buy a device that promises a higher MTBF) and 2)There are plenty experiences I've seen where the backup you just overwrote was good and the backup you replaced it with is bad (either due to physically bad backup device or the backup is missing info due to user error).

This is a great topic...but hard to convey all points on a forum. In person is much better.

[murphychris]

I think this is rather complex and is a huge reason why so many people just opt for attached hard drives to copy files over, despite the negatives of ancient file systems like JHFS+/X and NTFS; despite the limitations of USB/Firewire external hard drives for regular SMART tests to try and predict problem harddrives before they fail; and to have access to resilient file systems that, in effect, completely obviate silent data corruption, bit rot, and the RAID 5 write hole. There's simply no ambiguity with the data.

FreeNAS or NAS-4-free are pretty straightforward to setup, all configuration is through a web browser like with other NAS options (although you do have to install the software through a rudimentary appearing text faux-graphical user interface) and offers a resilient file system. For free. Includes reasonably up to date AFP for Time Machine support as well. So at least your backups can be made resilient, even if your local file system isn't. And it's storage that's Windows and Mac OS friendly.

I'm just very leery of locally attached storage. Many people do not understand that journalling does not make a file system safer, or more resilient to power failures or crashes. It simply reduces the time to repair a damaged file system. There's a point at which such a file system is damaged beyond repair. And even when the file system is healthy, does not mean every single sector on the drive is healthy.

But direct attached storage is easier for people because they're familiar with it. However less than a decade ago 2TB of data was a **** ton of data that medium sized businesses were managing with more sophisticated knowledge and software than we presently have on the Mac OS or Windows. So I don't think it's unreasonable for people who say they have valuable data, to take it upon themselves to learn more than what's offered with their operating system - which sadly is to say, not much. Time Machine is great conceptually because it actually does a very good job on both backups and restores, but it's still subject to all of the questions and concerns of JHFS+/X as a file system, which brings nothing to the table in terms of resiliency for lots of data.

Further a lot of people get a little too neurotic about performance differences between locally attached storage and NAS. It's true network storage is slower than fast locally attached storage arrays, but for people to hype performance over resilience is why we see so much data loss. People just don't give a crap about resilience until data is lost and then they have the experience, and it's just tough to be that sympathetic.

Most people need resilience, and want performance. But are emotionally attached to want, and discard what they need. And in my view persistently make the wrong storage choices. The choice is resilience vs performance. If you want both, prepare to spend a lot of money.

[md63]

I use a Sans Digital Towerstor 2.5 inch five bay raid enclosure. Pricy but very well built. I use a Synology DS411slim to back up my entire system. This is also using 2.5 inch drives.




http://www.sansdigital.com/towerstor/ts25ct.html

[radiogoober]

This might be a good place to ask. What is iSCSI? Is it a way for a NAS to appear as a device on your computer?

[AppleDApp]

Originally Posted by ericinboston

Well I still don't know exactly what you are backing up, how large that backup is, and how often. Let me give you my example of my backup strategy for a few TBs worth of data/media/operating system (I'm on Windows in this example):

I have Windows, pix, Word docs, and similar text docs on my 500GB C drive. It usually takes up about 90GB total. On my 2TB internal D drive, I am eating about 1.75TB with gazillions of MP3/audio files as well as movies/video files. Now, I do NOT back up my D drive often. Why? Because a)there's no sense in always backing up the same 1.75TB every week/month and b)my media on it doesn't change daily and c)I'm not in a paranoid state (such as my computer is dying) that I need to back up 1.75TB every night and d)I have a baseline backup and every few weeks I backup about 1GB of new stuff at most. Now let's address each point:

A)Why would I want to back up, say, the same 100 movies every night/week? Once I have a backup it's backed up. Done. Sure, maybe have 2 1.75GB backups if you are really worried. If there are items that you need to back up every night/week, just back up those while keeping your giant movie archive already on the older backup.

B)Does your media chance daily? What exactly are you backing up? Are you using software or some simple scripts to copy the files from place to place?

C)Nothing is forever. You can have the best backup solution and then your house catches on fire...or your 4 year old kid smashes a drive. If you have 1 or 2 backups of your absolutely cannot-lose data, that's fine.

D)Again, I have no idea how much data you are trying to back up and how often.

A) I only have music on my laptop at this time. That is backed up to time machine every time it backs up. I have a another drive that is a backup of my internal drive that I will edit when I have large changes.

B) My media changes weekly I use time machine to backup my music. I have a 2TB drive that contains all my music movies and shows that is backed up to another drive.

C) true

D) It depends sometimes it is several gigs other times it's 100+ gigs.

Originally Posted by ericinboston

Solutions:
---------

1)Maybe you should segment your media on different drives...keep audio on 1 drive while video on another drive. This way you are not senselessly backing up EVERYTHING just because you added 5 mp3 files.

all my media is on 1 drive documents and songs are on the macintosh HD.

Originally Posted by ericinboston

2)Maybe you should not be using backup software. For my MP3 backup, I simply use a 1-line command line statement that copies only newer MP3s from drive A to drive B. So this way if I add 44 songs to my drive A, I can run the 1-line command and only the 44 new ones get added to by drive B (which is my mp3 backup). But for my my OS I "image" it every few weeks which compresses the 90GB down to about 50GB.

I don't know much about command line.

Originally Posted by ericinboston

3)Have you tried Bluray archiving? I bought a fantastic drive for $84 and a spindle of name-brand 30 25GB discs for $30. Now, this of course is really for the permanent backup and to get all your 2TB backed up as a starting point. Let's do the math: 30 25GB discs = 750GB of space for $30. You have 2TB so you would need about 4 packs of 30 discs which would be about $120 for the discs + $84 for the drive. So about $200 for a "permanent" backup solution where the medium cannot accidentally be destroyed. Again, nothing is forever but I would bet my cdrs/dvdrs/blurays over an external hard drive (or internal) any day...especially for things that can happen to drives like electrical spikes, water, magnets, dropping, virus, etc.

I haven't tried bluray archiving.

Originally Posted by ericinboston

4)You state USB 2.0 is too slow...why not use eSATA? Or, forget externals altogether and just use internal SATA drives? You could buy some very nice 7200RPM drives that are extremely fast.

There are no eSATA ports on the mac unless you get the mac pro. laptops only have 1 internal drive. I have already bought several hard drives I will stay with the onces I have.

Originally Posted by ericinboston

5)I think part of your solution is to consolidate these 4 physical drives into a few 3TB (or larger) drives so you have less stuff to manage and less stuff fighting for USB connections, network traffic, etc.

So again you want to get higher capacity drives? Nothing is fighting for a USB port at this time.

Originally Posted by ericinboston

6)Get yourself a baseline backup (like I have)...say of 1 or 2TB. Then worry about what you want to back up above and beyond that.

I've done a baseline.

Originally Posted by ericinboston

7)If you really want to back up 2TB+ a day/week, I think you're going to need some serious drives that cost serious money. 2TB+ is still a LOT of info to back up every night/week for even pro-sumers in 2012. 10 years from it likely will a snap

I rarely have to move more then 500gb at once.

All i want is NAS for to put my drives in to know how to use it and a backup solution for the NAS.

Originally Posted by radiogoober

This might be a good place to ask. What is iSCSI? Is it a way for a NAS to appear as a device on your computer?

I don't know much about iSCSI but I found this. What is iSCSI?

[murphychris]

I haven't investigated BluRay as archive media. CD and DVD absolutely do not pass that test. If you hate your data, put it on CD or DVD media only. It's definitely not a backup or archiving option. So it would take considerable 3rd party derived empirical data to demonstrate that BluRay media available to consumers can archive data - and primary for me in that realm would be very good ECC. And I don't know that it has that or not, so in the meantime I wouldn't trust it any more than any other writable media we've had.

[ericinboston]

Originally Posted by murphychris

I haven't investigated BluRay as archive media. CD and DVD absolutely do not pass that test. If you hate your data, put it on CD or DVD media only. It's definitely not a backup or archiving option. So it would take considerable 3rd party derived empirical data to demonstrate that BluRay media available to consumers can archive data - and primary for me in that realm would be very good ECC. And I don't know that it has that or not, so in the meantime I wouldn't trust it any more than any other writable media we've had.

Backup doesn't have 1 definition or 1 test. That's the problem. You stating that cdr/dvdr doesn't pass "that test" has no merit. Consumers, Prosumers, and Businesses have been using cd/dvd/MOST technology for decades and it does fit quite a few scenarios.

I have cdrs from the mid 90s that I just checked a few weeks ago when I came across them in the closet...worked fine.

Again, the backup discussion can be held for hours in a conversation. Backup is defined differently, and done differently, by everyone.

What I like about cdr/dvdr/bluray is that they cannot be killed accidentally such as running a Delete command or a virus kills it or any other software problem...they are READ ONLY once the discs are created. Also, they cannot be killed by other common problems such as extreme heat or cold that would normally kill a hard drive...the discs have a LOT of benefits over a normal, standard, hard drive.

Nothing's perfect...and every technology has it's pros and cons. What I like about Bluray is that I can stuff 25GB of data on a disc (I haven't used the 50GB discs yet), a 30-pack spindle of QUALITY discs is $30, and my goal is to hold onto these discs for 5+ years. YES, I will certainly make other backups over time onto other media including hard drives, but it's nice to know these discs will fit easily in my fireproof box, are extremely like to last 5+ years, are inexpensive, and easy to use.

[murphychris]

Originally Posted by ericinboston

Backup doesn't have 1 definition or 1 test.

In information technology it's a rather specific thing and included in the definition and test is the ability to restore data. That's frequently missing in overly complex backup strategies where 99.9% of the time you're backing up fine but it goes to crap when restoration is required. Without the ability to cleanly, reliably restore data, you don't have a backup.

You stating that cdr/dvdr doesn't pass "that test" has no merit. Consumers, Prosumers, and Businesses have been using cd/dvd/MOST technology for decades and it does fit quite a few scenarios.

No merit? Have you even done basic research on this? Produce your sources if you have. There's a litany of research in this area and it's very consistently considered high risk.

Risks Associated with the Use of Recordable CDs and DVDs as Reliable Storage Media in Archival Collections

Excerpt 1:

representatives of manufacturers expressed sympathy with the concerns of the archivists, and while several products have been marketed since which attempt to optimise their longevity, the principle problem remained unsolved. As this publication explains, the problem can only be resolved by the adoption of standards for blank media and writers which aim to optimise the reliability and longevity of the recorded media. It is possible, however, to use recordable optical discs as reliable storage media even before this agreement is reached. It requires the application of a stringent procedure of testing and selecting of both the blank discs and the recording devices. Such testing is time consuming and requires expensive test equipment.

Excerpt 2:
As they were never intended for use as reliable media for long term preservation and have been developed primarily as a consumer product for the mass market their use in critical archival and preservation contexts constitutes a significant risk

And even in 2004 it was understood by audiophiles. CD and DVD Longevity: How Long Will They Last?

I have cdrs from the mid 90s that I just checked a few weeks ago when I came across them in the closet...worked fine.

Lucky, in my view. You have a hash for every single file so you can confirm they are identical to the original? Just because the directory is read and you get some icons displayed, and files read off the disc, doesn't mean every single file is identical to the original.

Again, the backup discussion can be held for hours in a conversation. Backup is defined differently, and done differently, by everyone.

Backup discussion is potentially very complex. The definition is not complex, isn't defined differently except by those who do not know what they're talking about.


[quote]What I like about cdr/dvdr/bluray is that they cannot be killed accidentally such as running a Delete command or a virus kills it or any other software problem...they are READ ONLY once the discs are created.[quote]

Next to meaningless. HDD/SDD can be mounted read only. As for SDC and bit rot, there are equivalents for RO media, in fact optical media is substantially more sensitive to light corrupting data than HDDs.

Also, they cannot be killed by other common problems such as extreme heat or cold that would normally kill a hard drive...the discs have a LOT of benefits over a normal, standard, hard drive.

You have it backwards. Optical discs have lower tolerance for heat than HDDs.

Did you know that ISO 9660 specifies the storage temperature for optical media in an archival context? And that it's 64 to 73 F, with humidity between 30% and 50%. Good luck getting that kind of consistency. HDD's are substantially more tolerant when they are spinning, let alone when they aren't. Did you know that there is a huge range in life expectancy depending on exactly what mechanism wrote the disc, and what specific make/brand of disc is used, as well as the storage requirements? It's non-trivial for people to figure this out. I hear and read of way more data loss stories, as a percentage, with optical media than HDDs.

YES, I will certainly make other backups over time onto other media including hard drives, but it's nice to know these discs will fit easily in my fireproof box, are extremely like to last 5+ years, are inexpensive, and easy to use.

Depends on the fire safe.

https://support.imation.com/app/answ...-storage-media

And BlueRay is not that cheap. $1 per 25GB disk is 4 cents per gigabyte, compared to 4-7 cents per gigabyte for a hard drive. And those 4 pennies don't include the cost of the Blu Ray drive.

I think optical media as backup is a waste of time. And especially because it's read only it's also rather non-ecological.

[AppleDApp]

Could we stay on topic. I won't be using optical media to backup.

[ericinboston]

@murphychris:

I understand everything you wrote...and have read it all before. I stand by my "no merit" comment because you were not specific on exactly what "doesn't pass the test" means. There are millions of users every year that "back up" their data on cdr or dvdr...and retrieve it days/weeks/months/years later. Will it last 50 years? Who knows. Will it last 25 years? Who knows...but that's the point of DEFINING your backup/restore solution. You also cannot compare a cd/dvd/bluray that IS read-only after being written to a drive that "can be" mounted as read-only and that read-only mechanism is controlled by software/firmware and will 100% be read-only and not hack-able or possibly corrupted/affected by a virus. It would be like telling me that cassette tapes are read-only because the little tab at the top of the shell is missing...well...I can put a piece of Scotch tape on it and POOF! now I can write to it. Not so with cd/dvd/bluray. Once it's written, that's the end of writing to it again. Forever. No hack.

A Backup solution should be defined as to 1)how much data you need to back up, 2)how often, 3)what style/method (full, incremental, etc), 4)the length of time you will need to hold onto the backup, 5)what the penalty is if your restore fails.

#5 is really what drives cost. Does the OP lose millions of dollars a day if he has no backup/failed restore and hence has to re-create all the data again? That's how businesses think. I don't believe the OP is a business so all my answers are based on the common consumer. He has a mix of music and movies...but I don't know what the percentages are. For example, if I had 100 dvd movies and 50 cds and my backup of them was dead, it wouldn't be that bad to re-rip 100 movies and 50 cds depending on the quality of the rip. Sure, I wouldn't want to do it...but it would be a lot easier/faster than say ripping 4000 cds and 300 dvd movies and 100 bluray movies (like my scenario). I likely simply would only re-rip the stuff I really really wanted...I could not spend literally years of time re-ripping all that.

For a common consumer, all your points really don't affect him (but are completely valid for many businesses). The OP has to think about (among other things, this list is not complete): 1)what is the likelihood his master drive(s) will fail and hence has to go to a backup...2)how old is that backup? 5 years? 1 year? 10 years?...3)what if part of the backup is bad? is that ok? Or is there a demand that it be 100% recoverable...4)does he want to create duplicate backups so he is extra insured? I have 3 copies (2 on different externals + 1 on bluray discs) of my 20k MP3 files because I don't want to re-rip thousands of cds again.

Again, I appreciate your lengthy reply but that's really geared towards businesses who are concerned about their data, lose a boatload of money every hour that it takes to re-create data or spending time recovering data from backups, and who are going to be spending tens of thousands of dollars a year on a backup solution plus employees' payroll to do backups/restores. I used to work in IT and had to manage plenty of backups. It's a never ending battle. At the end of the day the business/owner needs to evaluate the risks and the dozens of What If scenarios and make a decision.

Now all that being said, the OP says he does not want to use bluray for backup. I would suggest he DOES plunk down $200 and back up his stuff as an ARCHIVE. There's a difference between an archive and a backup. He should consider the low cost in doing a bluray archive, a dozen hours or so to create the archive, and be done with it. I would also recommend archiving to an external drive...take both the external and the blurays and put them in a firebox or safety deposit box or wherever he thinks is a good place.

Let me quote the OP again:

I only use macs for the time being I also have a windows VM. I have four 2tb drives that were previously in my PC. One 1tb drive to backup my laptop and a portable 1tb to carry files around.

Please explain exactly how/where each of your 4 (or more) 2TB drives are being used, how much free space is on each, and expected free space in 1 year.

USB 2.0 is too slow often times when I move data around I move several gigs at once. Sometimes hundreds of gigs at once. USB 3.0 or something faster is a must.

Well if you only use a Mac(s), you're stuck at USB 2.0. You should consider a)buying some kind of eSATA or USB 3.0 adapter or b)buying new computers with the better ports or c)somehow offloading all your data OFF the Macs and onto something that is far more blazing...you could go to NAS but what about buying an inexpensive, yet powerful, desktop computer (PC is gonna be cheaper than Mac) to act as a file server...something with numerous USB 3.0 ports, room for 2-5 internal SATA drives. Said box would likely cost $800-$1000 with 1-2 2TB SATA drives included...then spend another $300-$500 on extra drives if you need it. Let that 1 box be your file server and let it do all the backups locally...no network traffic, no bogged down Macs trying to play a movie while backing up at the same time, etc.

I am not full at 8tb. All of my data accounts for about 2tb. I have two backups of my media. Currently making a 2nd backup of my laptop. I will not sell my drives because it is all in all a huge pain.

If you only have 2TB of data (you said that's everything...ALL your data), then why are you mucking around with 4 2TB drives all over the place? If you've got all 2TB on 1 drive, for example, then use a 2nd drive (locally) to back it up every now and then.

I was considering getting a NAS to backup my computers and store my media but I should also consider a backup solution for the NAS. I am also not sure how to move everything from itunes to the NAS and whatnot.

Why are you so paranoid about losing data (I'm not being mean)...you are backing up backups. Realistically you should always be backing up the SOURCE...not backups...because if you back up a bad backup, you have 2 copies of a bad backup. And when I say "bad backup" it could mean it's either corrupt or simply you forgot to back up certain data. I think you're getting way too complex...backing up over a network to NAS...then somehow backing up the NAS again. And since you're on a Mac laptop, you likely have those wonderfully slow 5400RPM drives...that doesn't help any of your timelines for fast(er) backups. And I'm not sure what your network speed is, if it's wireless or wired, and if you've actually done performance tests. My house is wired for 1Gbit but none of my 3-5 year old machines can push that speed (with their so-so on-board NICs) over cable even if it was just a dedicated Switch. Even transferring over G wireless is extremely slow and I refuse to send more than 10GB because of the wireless slowness.

As far as other suggestions for the OP, I really still do not understand the entire scope of his problem and his attempted solution. 1)I don't use Time Machine and 2)If I could see his scenarios as a diagram, that would really help...I'm a picture person unless every single detail is written in bullet-form (which you started to do on your 8:38 reply but I need a lot more info). And what is your budget?

[AppleDApp]

Originally Posted by ericinboston

Let me quote the OP again:

Please explain exactly how/where each of your 4 (or more) 2TB drives are being used, how much free space is on each, and expected free space in 1 year.

I use one as a bootable HDD for my mac which probably has 1.5TB free or so.
I have a backup copy of my media. there is about 1tb free on that drive.

Originally Posted by ericinboston

Well if you only use a Mac(s), you're stuck at USB 2.0. You should consider a)buying some kind of eSATA or USB 3.0 adapter or b)buying new computers with the better ports or c)somehow offloading all your data OFF the Macs and onto something that is far more blazing...you could go to NAS but what about buying an inexpensive, yet powerful, desktop computer (PC is gonna be cheaper than Mac) to act as a file server...something with numerous USB 3.0 ports, room for 2-5 internal SATA drives. Said box would likely cost $800-$1000 with 1-2 2TB SATA drives included...then spend another $300-$500 on extra drives if you need it. Let that 1 box be your file server and let it do all the backups locally...no network traffic, no bogged down Macs trying to play a movie while backing up at the same time, etc.

I will be buying an iMac once they are released. I'm not sure how I would get that extra PC to act as a file server. NAS is definitely cheaper then building a PC.

Originally Posted by ericinboston

If you only have 2TB of data (you said that's everything...ALL your data), then why are you mucking around with 4 2TB drives all over the place? If you've got all 2TB on 1 drive, for example, then use a 2nd drive (locally) to back it up every now and then.

I only use two of my 2TB drive as mentioned above.

Originally Posted by ericinboston

Why are you so paranoid about losing data (I'm not being mean)...you are backing up backups. Realistically you should always be backing up the SOURCE...not backups...because if you back up a bad backup, you have 2 copies of a bad backup. And when I say "bad backup" it could mean it's either corrupt or simply you forgot to back up certain data. I think you're getting way too complex...backing up over a network to NAS...then somehow backing up the NAS again.

Yes I realized it might seem paranoid but losing files suck. The backups of backups I have done are good. I have yet to purchase a NAS that was just a thought.

Originally Posted by ericinboston

As far as other suggestions for the OP, I really still do not understand the entire scope of his problem and his attempted solution. 1)I don't use Time Machine and 2)If I could see his scenarios as a diagram, that would really help...I'm a picture person unless every single detail is written in bullet-form.

• The Mac I have has a 500gb internal drive.
• I have 6 external hard drives not all connected to the mac at once. They are either 1TB or 2Tb in size.
• One 1tb drive is used to move things on the go. Often times it's to watch movies at a friends house or something so this drive has all of my video content.
• The other 1Tb drive serves as a Time Machine Backup for my Mac.
• Now on to the four 2tb drives that I have, I use one of to as a bootable drive for my macintosh HD as per mentioned above.
• I use another 2tb drive to backup the video content that is on the 1tb drive.
• I don't currently use the other two hard drives.

I wanted to use a NAS to consolidate all my media and backup 2 computers. That seems like an ok option but seeing as no internal drive on an iMac will hold all my media (considering long term needs) I would probably also have to look at a way to backup the media that is on that NAS.

[ericinboston]

Originally Posted by AppleDApp

I will be buying an iMac once they are released. I'm not sure how I would get that extra PC to act as a file server. NAS is definitely cheaper then building a PC.

I will reply more later (and I edited my reply earlier (likely while you were replying to me) so can you go back and re-read it? I added many more questions)...but 1)you are not building a PC...you order it, power it on, and use it. 2)The PC's main purpose in your case could be the file server AND to back up its content (all your media)...but it could also be used for other things if you needed.


Think about it this way: you have a Mac or 2 on a network...you have a PC on the network that is essentially just the file server. The Macs/Airplay play all your media and life is good. When it comes to backups (of the media), the PC just backs itself up to an internal SATA drive or an external USB 3.0 drive. No network activity and should be lightning fast. If you want the PC to back up your Mac, that should be easy but I've never done it...likely it might be easier to just plug in an external to your Mac every month and back it up overnight.

I'm not trying to shove the PC option down your throat but in the grand scheme of things, it can solve your problem and do a lot more...a NAS is just a hard drive. It may be cheaper than a PC but the NAS will always be a NAS. A PC is 100x more than just a hard drive.

[ericinboston]

Originally Posted by AppleDApp

I use one as a bootable HDD for my mac which probably has 1.5TB free or so.
I have a backup copy of my media. there is about 1tb free on that drive.

I will be buying an iMac once they are released. I'm not sure how I would get that extra PC to act as a file server. NAS is definitely cheaper then building a PC.

I only use two of my 2TB drive as mentioned above.

Yes I realized it might seem paranoid but losing files suck. The backups of backups I have done are good. I have yet to purchase a NAS that was just a thought.

• The Mac I have has a 500gb internal drive.
• I have 6 external hard drives not all connected to the mac at once. They are either 1TB or 2Tb in size.
• One 1tb drive is used to move things on the go. Often times it's to watch movies at a friends house or something so this drive has all of my video content.
• The other 1Tb drive serves as a Time Machine Backup for my Mac.
• Now on to the four 2tb drives that I have, I use one of to as a bootable drive for my macintosh HD as per mentioned above.
• I use another 2tb drive to backup the video content that is on the 1tb drive.
• I don't currently use the other two hard drives.

I wanted to use a NAS to consolidate all my media and backup 2 computers. That seems like an ok option but seeing as no internal drive on an iMac will hold all my media (considering long term needs) I would probably also have to look at a way to backup the media that is on that NAS.

Thanks for more info...but how often are you backing up?...how much data per backup? are you backing up the same 100 movies (plus a few new ones) every day/week/month or just the new ones? etc. Walk us through a day in the life of a backup...or multiple backups...what is Time Machine actually doing/backing up? Are you using other software/tools to "back up"? Backing up over wifi or cable or direct USB etc...? How do you WANT TO CHANGE your current backup solution?

[murphychris]

Originally Posted by ericinboston

He should consider the low cost in doing a bluray archive, a dozen hours or so to create the archive, and be done with it.

It is not low cost. A $200 Blu Ray drive costs more than a single 2TB -2.5TB HDD that would do the same thing, and vastly faster.

The OP said his total data is 2TB. At 25GB per optical disc, that requires 80 25GB Blu Ray discs to produce an archive. Even 40 50GB discs is absurd. It exceeds the mass and volume of a single HDD, by a lot. And then the data is separated on 40 to 80 discs.

There is no meaningful advantage to RO in the case of a shelved drive that has no chance of becoming infected with a virus. If the OP is remotely concerned about others having unauthorized access, and deleting or modifying data in the archive, he can encrypt the drive losing no space or time in the process.

The Blu Ray suggestion is low efficacy. One 2.5TB drive solves this inside of 2 hours from start to finish.

Realistically you should always be backing up the SOURCE...not backups...because if you back up a bad backup, you have 2 copies of a bad backup.

I agree that derivatives of backups isn't a good idea. This requires qualified data replication and checksumming to be assured there's parity between backup and derivative; some NAS solutions have this. I think given the amount of data it's easier to just:

1. Time Machine backup daily/hourly/weekly. That's the regular on-site backup and archive.
2. CrashPlan is the regular off-site backup and archive.
3. Carbon Copy Cloner one time to one disk. That's the locally shelved archive in case Time Machine fails. He can use the archive to do primary restoration and CrashPlan to restore the more recent stuff.

Alternatively, two Time Machine backups can be employed and alternated between.

Advantage of CCC is it's possible to checksum all the data, so when reading it you can optionally confirm/deny that files match their checksums. If they don't, you can selectively restore those files from another source.

[murphychris]

Originally Posted by AppleDApp

USB 2.0 is too slow often times when I move data around I move several gigs at once. Sometimes hundreds of gigs at once. USB 3.0 or something faster is a must.

I don't understand why you're moving data around of such quantities.

Your options are limited to:

a.) NAS [1], wireless. Slower than USB 2.

b.) NAS, Gigabit ethernet. I can push 110MB/s over gigabit ethernet and NFS.

b.) NAS, 10GigE. If 100MB/s isn't fast enough, upgrade to 10GigE. Expensive.

c.) USB 3 or Thunderbolt drives. If 100MB/s isn't fast enough, and you don't want to go to 10GigE, this is your last option, which means physically moving one or more drives around, no consolidating of data.

I personally think people who think gigabit ethernet is slow are either really impatient, or they're working on HD video in which case 10GigE is the way to go.


[1] NAS will end the pushing files around as all data is consolidated. You can use the NAS for Time Machine backups for multiple computers, if the NAS supports Time Machine backups (via recent versions of NetaTalk). Depending on the NAS, it can optionally replicate its data reliably (FreeNAS, NAS4|free, Nexentastor) to other disks or another NAS.

[AppleDApp]

Originally Posted by ericinboston

I will reply more later (and I edited my reply earlier (likely while you were replying to me) so can you go back and re-read it? I added many more questions)...but 1)you are not building a PC...you order it, power it on, and use it. 2)The PC's main purpose in your case could be the file server AND to back up its content (all your media)...but it could also be used for other things if you needed.

Budget is limited for building/buying a PC. No more then 500$

Originally Posted by ericinboston

Think about it this way: you have a Mac or 2 on a network...you have a PC on the network that is essentially just the file server. The Macs/Airplay play all your media and life is good. When it comes to backups (of the media), the PC just backs itself up to an internal SATA drive or an external USB 3.0 drive. No network activity and should be lightning fast. If you want the PC to back up your Mac, that should be easy but I've never done it...likely it might be easier to just plug in an external to your Mac every month and back it up overnight.

Not much help here

Originally Posted by ericinboston

I'm not trying to shove the PC option down your throat but in the grand scheme of things, it can solve your problem and do a lot more...a NAS is just a hard drive. It may be cheaper than a PC but the NAS will always be a NAS. A PC is 100x more than just a hard drive.

True but I'd have to figure out how I will access the data how it will be backed up and what not.

Originally Posted by ericinboston

Thanks for more info...but how often are you backing up?...how much data per backup? are you backing up the same 100 movies (plus a few new ones) every day/week/month or just the new ones? etc. Walk us through a day in the life of a backup...or multiple backups...what is Time Machine actually doing/backing up? Are you using other software/tools to "back up"? Backing up over wifi or cable or direct USB etc...? How do you WANT TO CHANGE your current backup solution?

If I know which is the new content I will only back that up. if I don't I copy the whole content over. I do a backup every time there is a significant change.

Time machine only backups the internal drive of my macbook. I then have a CCC version of my internal drive if time machine fails.
MY media is stored on a 1tb drive which is copied on another 2tb drive. All my backups are currently done via USB. I want to use my four 2tb drive in a centralized system be it a NAS or tower but I want it all to be centralized and in a small form factor.

Originally Posted by murphychris

I don't understand why you're moving data around of such quantities.

Your options are limited to:

a.) NAS [1], wireless. Slower than USB 2.

b.) NAS, Gigabit ethernet. I can push 110MB/s over gigabit ethernet and NFS.

b.) NAS, 10GigE. If 100MB/s isn't fast enough, upgrade to 10GigE. Expensive.

c.) USB 3 or Thunderbolt drives. If 100MB/s isn't fast enough, and you don't want to go to 10GigE, this is your last option, which means physically moving one or more drives around, no consolidating of data.

I personally think people who think gigabit ethernet is slow are either really impatient, or they're working on HD video in which case 10GigE is the way to go.


[1] NAS will end the pushing files around as all data is consolidated. You can use the NAS for Time Machine backups for multiple computers, if the NAS supports Time Machine backups (via recent versions of NetaTalk). Depending on the NAS, it can optionally replicate its data reliably (FreeNAS, NAS4|free, Nexentastor) to other disks or another NAS.

I would like to use NAS and gigabit ethernet. I'd like an option that can replicate data to another device.

[ericinboston]

Originally Posted by murphychris

It is not low cost. A $200 Blu Ray drive costs more than a single 2TB -2.5TB HDD that would do the same thing, and vastly faster.

The OP said his total data is 2TB. At 25GB per optical disc, that requires 80 25GB Blu Ray discs to produce an archive. Even 40 50GB discs is absurd. It exceeds the mass and volume of a single HDD, by a lot. And then the data is separated on 40 to 80 discs.

I thought he only had 1TB of actual data yet it was sitting on a 2TB drive...and a subset of that 1TB data is actual media files (video and audio)...but I do agree with you, I wouldn't want to create a 40 disc set. I must have been thinking of a smaller amount of an archive and just made a mistake in my writing...sorry about that. However, bluray drives are not $200...they're $80 at any store. Discs cost about $1 each 25GB disc.

But anyway, I agree with everything else you mentioned...I just didn't quote it here.