Is a hard drive the best way to archive video for 20-40 years

[Dingo41]

I am trying to determine the best way to archive 8mm and Hi8 video. Since these tapes degrade over time, I have been running them through an imac and putting them on 500 gig Lacie external hard drive. It seems that hard drives dont last forever, so after period of time I need to back up the external hard drives to new external hard drives. I have 6 questions:

1) Do hard drives degrade over time similar to the was that Hi8 tapes do?

2) Each time I copy one hard drive to another, is some quality of the video/data lost?

3) with what frequency should I be backing up the hard drives-- every 5 years?

4) Is an enterprise class hard drive recommended for this. My cursory research suggests that these are muh higher quality drives than for example a Lacie external hard drive.

5) which external hard drive(and manufacturer) would you recommend. I need have about 2TB of data

6) are SSD drives optimal or are these really not a proven technology yet

[simsaladimbamba]

Originally Posted by Dingo41

1) Do hard drives degrade over time similar to the was that Hi8 tapes do?

Not in the same way, but they can fail. I have had HDDs fail on me after using them only for months, but I also have many HDDs still having an IDE interface, thus they are quite old (ten years or so) which still work.

2) Each time I copy one hard drive to another, is some quality of the video/data lost?

Normally not, as it is a digital copy, thus a 1 stays a 1 and a 0 stays a zero.
Such copying is best done with CarbonCopyCloner, as it does a checksum (verifying everything copied okay) of the copied stuff.

3) with what frequency should I be backing up the hard drives-- every 5 years?

There is no rule there.
I have one 500 GB HDD for my photographs (digital and analog) libraries and editing documents, one 500 GB HDD with my personal video footage in an editing friendly format.
Both 500 GB HDDs get backed up to one 1 TB HDD via CarbonCopyCloner.
And that 1 TB HDD gets backed up to another 1 TB HDD via CarbonCopyCloner.
Therefore I have three copies of my important data.
Once I register one HDD as failing, I will replace that HDD.

4) Is an enterprise class hard drive recommended for this. My cursory research suggests that these are muh higher quality drives than for example a Lacie external hard drive.

Enterprise HDDs are made for servers or workstations and rated for 24/7 use. No need for them in your case.

5) which external hard drive(and manufacturer) would you recommend. I need have about 2TB of data

Western Digital, Toshiba and Seagate, as they manufacture their own HDDs.

6) are SSD drives optimal or are these really not a proven technology yet

2 TB of SSD storage would cost you at least 1000 USD nowadays, and recovering data from a failed SSD is more cumbersome than from a failed HDD.
Best to have at least two copies of the important data.

[Fishrrman]

Have you considered M-disc technology:
http://www.milleniata.com/

Looks like an ordinary DVD, but uses a different kind of data-recording technology that is claimed to last a century...

[siraltus]

Originally Posted by simsaladimbamba

2 TB of SSD storage would cost you at least 1000 USD nowadays, and recovering data from a failed SSD is more cumbersome than from a failed HDD.
Best to have at least two copies of the important data.

EDIT: Misread post. My bad.

[mobilehaathi]

Originally Posted by siraltus

I think you put an extra zero there by accident.

A 2TB external hard drive can be had for $120-150.

Example: http://www.newegg.com/Product/Produc...82E16822178116

No, he said SSD. He was correct.

[Paulywauly]

Hard drives aren't designed to work for decades, like all mechanical devices they will eventually fail.

You would be better off keeping backup copies on a few drives, checking them once a year and if one doesnt work just replace it with a new one.

[siraltus]

Originally Posted by mobilehaathi

No, he said SSD. He was correct.

Ah, missed the SSD part. You are right.

[gnasher729]

Originally Posted by Dingo41

I am trying to determine the best way to archive 8mm and Hi8 video. Since these tapes degrade over time, I have been running them through an imac and putting them on 500 gig Lacie external hard drive. It seems that hard drives dont last forever, so after period of time I need to back up the external hard drives to new external hard drives. I have 6 questions:

1) Do hard drives degrade over time similar to the was that Hi8 tapes do?

2) Each time I copy one hard drive to another, is some quality of the video/data lost?

3) with what frequency should I be backing up the hard drives-- every 5 years?

4) Is an enterprise class hard drive recommended for this. My cursory research suggests that these are muh higher quality drives than for example a Lacie external hard drive.

5) which external hard drive(and manufacturer) would you recommend. I need have about 2TB of data

6) are SSD drives optimal or are these really not a proven technology yet

0. In general: Whatever answer we give today will not be valid in ten years time.

1. If you look at ten years, who knows. For long term storage like that I would expect that some bits change for no reason whatsoever, and I would want some software that does checksumming at the very least. There's also the question how long the electronics will work. And nobody knows if sitting around and doing nothing is good or bad for a drive. I own some electronics that was high end and expensive in 1993 and that still works.

2. Each time you make a copy there might be errors introduced. There shouldn't be, but there might. That's easily checked by comparing original and copy.

3. Considering that the storage will become cheaper and cheaper over those 20 years, I'd buy a new drive once a year and keep the last few drives around.

4. Enterprise class is built to be used 24/7. You don't need that.

5. I'd take any USB 3 drive. With your long time frame, I would guess that USB 3 will be the interface that will be available for the longest time. And if I recommend company X, that company might not even be there in five years time.

6. Advantage of SSD is speed. Not really important for your purposes.

[phrehdd]

Hard drives are a poor solution as failure rates are hit and miss. One brand x drive could last way beyond its expected life and the same brand x drive could fail the next day.

I highly suggest you look at media discs.

There are a few sites you can explore via Internet that will explain the differences between the discs, the type of materials and so forth. In the past, there were CD discs that were made strictly for archival purposes. They were made different than the typical discs and certainly cost a bit more. I would suggest you check into DVD discs and possibly blu ray (though I suspect the latter has not come as far enough along to be truly archival).

It has been a long while since I have looked at archival discs but here is an old site you can start with (sorry its not a plug for a particular product but it does reference them }- adterrasperaspera.com .

I am sure many will argue that drives are just fine. If you insist on this route, I would, as I would with discs, make multiple copies per a volume and also keep them at different locations.

Best to you.

[rfle500]

Long term storage of data is a particularly challenging problem. Most storage systems are designed with a ten year life span, including hard drives, ssd, DVDR etc. I actually research magnetic materials so can definitively help with how long they last and why. The data is stored on a number of magnetic grains which are designed to retain their written state for a period of around ten years. Thermal activation leads to a loss of this information over time, and so the data will certainly not be there forever. For hard disks a more common problem is the mechanical arm and head wearing out, which is typically on the three-five year timeframe. In industrial settings archives are generally stored on magnetic tape (digital) which are much simpler in operation and likely to age much better. I think they are also specified with 20+ year lifetimes but I am not sure. However, they are also much more expensive than disks.

Practically I would use a simple RAID 5/6 set up of multiple disks which I would completely reformat every 2-3 years. This covers the degradation of data with time and also the possibility of a failing disk, but obviously not fire/theft. Regarding other technologies SSDs are unproven and certainly higher capacity versions are more likely to degrade over time due to smaller cell sizes. DVDs are subject to rot, particularly DVDRs as the materials used tend to react with the atmosphere overtime, making the data unreadable. This probably also applies to BluRay discs as well.

1) Do hard drives degrade over time similar to the was that Hi8 tapes do?

Yes.

2) Each time I copy one hard drive to another, is some quality of the video/data lost?

HDDs include advanced error correction with corrected errors typically less than 10-14

3) with what frequency should I be backing up the hard drives-- every 5 years?

That's a good idea, more frequently is better

4) Is an enterprise class hard drive recommended for this. My cursory research suggests that these are muh higher quality drives than for example a Lacie external hard drive.

If it is for archive purposes then there is little advantage, but multiple disks in a RAID setup is a good idea

5) which external hard drive(and manufacturer) would you recommend. I need have about 2TB of data

I always liked Samsung drives, but WD have always been good for us

6) are SSD drives optimal or are these really not a proven technology yet[/QUOTE]

Very expensive as archive storage, and unproven for extended periods of time.

[RedTomato]

Originally Posted by rfle500

Practically I would use a simple RAID 5/6 set up of multiple disks
...
If it is for archive purposes then there is little advantage, but multiple disks in a RAID setup is a good idea

I have no argument with the rest of your post, but I have to ask - have you ever actually worked with a RAID 5/6 set-up?

I have, and it is not pretty. RAID 5/6 is complex and answers a specific enterprise need. It is NOT for home use for the non-technical. You are introducing multiple points of failure. The OP is clearly non-technical, and looking after a RAID setup, especially after not having used it for two years, when you've forgotten the details of how you set it up, is full of traps. I've repaired RAID systems that have silently buggered up through user error, and it is not pretty and not for the faint of heart.

My advice? Keep it simple, cheap, and replicated. The OP says his data is 2TB. That's a small HDD. As a MINIMUM, keep 3 copies of the data in 2 different places. The original tapes form one copy, though as tape readers are becoming hard to find, it might be wise to not count on that.

Purchase 3 HDDs with 5 year warranties, minimum of 2 TB each, though 3 TB is cheap nowadays. USB3 as mentioned above is fine. Record a copy on each HDD. Give thought as to what format to store the video in - and store copies of applications and even operating systems that are able to read the video i.e. an install copy of Win 7 (or 8), an up to date Linux, OSX 10.8. At least 2 OS installers. And several apps that can show the video.

Consider buying two small computers (e.g. mac mini, or PC equivalent - though mac minis seem to last almost forever; my old workplace is still happily using a 7-year old mac mini G4 as a server) that can show the video, setting them up to work properly, then just storing them with the HDDs.

Two copies can be at your house or your office. The third copy needs to be in a physically different place (in case of fire or burglary) - across town or in a different city.

Also consider cloud storage. At the moment, 2TB is quite expensive to put on the cloud (though I hear Amazon Glacier is quite cheap). Prices will fall. That's a useful fourth copy in a non-physical place.

http://aws.amazon.com/glacier/

Budget for testing and replacing the HDDs every few years. And the computers too, though they will last longer.

In 15-20 years time, storage will be completely different. A 3TB USB3 HDD will seem as quaint and limited as an old typewriter ribbon - but with no typewriter to put it in.

[rfle500]

Originally Posted by RedTomato

I have no argument with the rest of your post, but I have to ask - have you ever actually worked with a RAID 5/6 set-up?

I have - at least the software RAID via Linux variety. However I appreciate your point and agree with your suggestions. In fact you can get simple plug-in sata USB adapters.

http://www.amazon.co.uk/dp/B0033AF5WW/

Just be sure to store your drives somewhere cool and dry.

[Apple fanboy]

Originally Posted by gnasher729

6. Advantage of SSD is speed. Not really important for your purposes.

No moving parts should also increase lifespan no?

[Pakaku]

Originally Posted by Fishrrman

Have you considered M-disc technology:
http://www.milleniata.com/

Looks like an ordinary DVD, but uses a different kind of data-recording technology that is claimed to last a century...

'Claim' is the key-word, since they have obviously never been around long enough to back up their claims

Originally Posted by Apple fanboy

No moving parts should also increase lifespan no?

You have different things to worry about, like bad sectors.

[rfle500]

Originally Posted by Apple fanboy

No moving parts should also increase lifespan no?

Yes in principle, but the storage mechanism involves the confining of charge to a small physical volume. The problem is, if left for a long period of time, the charge may leak out, turning a 1into a 0, and so losing data. Both thermal effects and quantum mechanical tunnelling would contribute to leakage, and MLC flash is much more likely to degrade than enterprise SLC.

[gnasher729]

Originally Posted by Apple fanboy

No moving parts should also increase lifespan no?

No moving parts means no moving parts stop moving

There are other things that fail on SSD drives. And the hard drives we are talking about are not moving. Most of the time they are just sitting still, not connected to any computer. Do you know what happens if you unplug an SSD drive and leave it locked in a fire-proof safe for a year? I don't.

Originally Posted by RedTomato

Consider buying two small computers (e.g. mac mini, or PC equivalent - though mac minis seem to last almost forever; my old workplace is still happily using a 7-year old mac mini G4 as a server) that can show the video, setting them up to work properly, then just storing them with the HDDs.

Two copies can be at your house or your office. The third copy needs to be in a physically different place (in case of fire or burglary) - across town or in a different city.

Two excellent items. Your video data is of no use whatsoever if you don't have a computer that can read it and no software to process it. And to different places is important.

(Considering two different places: What would you guys think about having a Time Capsule in your neighbour's loft, and their time capsule is in your loft, only wireless access? Would require a major disaster to destroy both data and backup, and if there's a fire that burns down both your and your neighbour's home, your data backup is the least of your worries).

[Apple fanboy]

Originally Posted by gnasher729

No moving parts means no moving parts stop moving

There are other things that fail on SSD drives. And the hard drives we are talking about are not moving. Most of the time they are just sitting still, not connected to any computer. Do you know what happens if you unplug an SSD drive and leave it locked in a fire-proof safe for a year? I don't.

No but I have old USB flash drives that have 5 year old stuff on them that still work fine. Isn't the technology the same? I have one which is only 256mb I found which had photos on from years back. It hadn't been used for t least 2 years and the photos were fine.

[SDDave2007]

Another thing not mentioned here....

What computer interface is going to be around 10 years from now to connect the "device" to?

What was state of the art 10 years AGO is mostly defunct today.... so perhaps todays media device interfaces will be obsolete 10 years from now.

Be pretty bad to have a stack of whatever type of media and nothing to read it with.

[sagnier]

i second the 'any suggestion will be invalid in 5yrs' argument. Of course, you still need a solution...

You maybe thought that your tapes were the best solution at one point - why imagine that any current solution will not be subject to the same revision? two or three HDDs should do the job for the meantime, and fairly cheaply if you assume a lifespan of maybe five years. Whats the yearly cost if you were to change them all after that period? Not much.

[Old Muley]

Originally Posted by SDDave2007

Another thing not mentioned here....

What computer interface is going to be around 10 years from now to connect the "device" to?

What was state of the art 10 years AGO is mostly defunct today.... so perhaps todays media device interfaces will be obsolete 10 years from now.

Be pretty bad to have a stack of whatever type of media and nothing to read it with.

Where is the SCSI port on my iMac? I need to plug in my SyQuest drive...

[old-wiz]

Aside from the question of the HDD or SSD or any media decaying over time, there's also the question of will you have a computer that will even read the drive via the interface that's on the drive. You would really need to re-copy to the newest interface on a regular basis.

[ShiftClick]

For what its worth, hard drives; mechanical or solid state, are not considered a proper way to archive any video for companies like Sony, Warner Bros. Most television productions shoot digital with only a handful of shows still shooting on film.

The only acceptable forms of archive to these companies is on LTO tape or in Sony's case they will accept XDCAM archives for long term storage as well, but XDCAMs last I worked with were limited to only 48GB a disc and archive times are very slow.

I would suggest putting all your video on a hard drive, contacting a company that can archive it all to an LTO tape. LTO5 tapes that can hold 1.5 TB uncompressed cost about $70~ And my guess they would charge $50 - $100 for the time. But atleast with LTO tapes (i would suggest having two archives in different locations) you KNOW its going to be okay and LTO tech isn't going anywhere for a long time its too valuable of stable of a way to archive data.

[WesCole]

Have you thought about using an online backup service like CrashPlan? You get unlimited storage off-site for ~$5/month (less if you pay for a year or more in advance). You could just save all your video to a single hard drive (or RAID 1 setup), then have CrashPlan back up all the movies automatically to their servers. Seems like this might be the easiest way to ensure the longevity of your data.

[paulrbeers]

Originally Posted by WesCole

Have you thought about using an online backup service like CrashPlan? You get unlimited storage off-site for ~$5/month (less if you pay for a year or more in advance). You could just save all your video to a single hard drive (or RAID 1 setup), then have CrashPlan back up all the movies automatically to their servers. Seems like this might be the easiest way to ensure the longevity of your data.

But (let me know if I am wrong about CrashPlan specifically) most of the "cloud" based storage I've used requires you to upload everything (you can't just send them a hard drive). I don't know about you, but around here both of our broadband providers only give 1mb/s upload. To even upload 1GB of storage means it would hog my entire upload bandwidth (rending my internet access virtually useless) for over 2 hours. Multiply that by 2048 (for 2TB) and you are dedicating 6months of your internet access to just uploading your library! At even $40 a month for internet, that's an investment of $240 in internet access fees just to get it uploaded (not to mention the additional $30 for crashplan).

Further, I bought 3TB drives on Black Friday for $100. I could buy two of them, (which is cheaper and faster than taking 6 months to upload), and I have the backups in my hands. If CrashPlan went belly up, I would have to find someone else and do it all over again yes/no?

To the point about not being able to hook up the drive in 10-20 years. That is a very serious problem. Years ago, we all backed everything up to tape. Problem is, tapes tended to use proprietary formats that now you can't even get the tape drives that your tape went into. With that said, SATA is not a proprietary format. Will it be around in 10-20 years? Probably not, but PATA which was around for well over a decade before SATA and is still possible to hook to your computer (I just had to buy a USB to PATA cable recently to make it work). My point being, is that if you use SATA drives, you should have plenty of time to move it to the "next big thing" before you have to worry about not being able to access your archives.

DVDR's have been proven to be unreliable (can't speak to blu-ray as I haven't looked into it). Even the best, have been found to disintegrate in less than 7 years even in the best of storage locations.

I always recommend archiving to hard drives and checking them at least once a year to make sure they do not need replaced.

[gnasher729]

Originally Posted by Apple fanboy

No but I have old USB flash drives that have 5 year old stuff on them that still work fine. Isn't the technology the same? I have one which is only 256mb I found which had photos on from years back. It hadn't been used for t least 2 years and the photos were fine.

The individual cells have shrunk enormously in the last five years. Might be just 100 electrons nowadays that store a bit. I would trust SSD drives less for long term storage today than five years ago.

That said, the magnetic areas for bits on a hard drive have shrunk as well. Anyway, nobody actually _knows_ what happens if you store data on a brand new SSD drive and leave it for five years because nobody has tried it yet. _And_ 2 TB of SSD drive is awfully expensive. For the same money you get 20 2TB hard drives.