Archiving thoughts

[dimme]

I just finished a 2+ year project scanning and cataloging our family photos my father took. The collection spans 50+ years and is over 8000 photos. I have the photos uploaded on flicker for the whole family to view. So not I thinking about long term storage of the digital files. Half the photos are tiffs and the other jpegs. Right now they are on my mac mini server drive which is backed up with crash plan. I also have 2 local backups. I started reading about bit rot and wondering if that is something I need to worry about. The drive the main copy is on is a WD my book running 24/7 I figure I would change out the drive every 3 years or so. The local backups are updated every week along with other files that are on the mini server.
Can anyone offer and advise or is my setup pretty sound.
Thanks

 

[dmax35]

I would buy a cheap drive and carbon copy your image files to it as a master copy and put it in a bank vault or safety deposit box offsite. God forbid any natural disaster to your house you always will have the master to recover from.

 

[dimme]

I would buy a cheap drive and carbon copy your image files to it as a master copy and put it in a bank vault or safety deposit box offsite. God forbid any natural disaster to your house you always will have the master to recover from.

Good Idea, I was wondering what maintenance the hard drive would need. I thinking of recopying the files drive every year or two. Will that be sufficient?

 

[swordio777]

Good Idea, I was wondering what maintenance the hard drive would need. I thinking of recopying the files drive every year or two. Will that be sufficient?

Depending on where you're storing your drives that's probably far more regularly than you need to. HDDs are obviously prone to failure, so updating it could potentially present more of a risk than any possible bit rot in a safety deposit box. If you're looking for the maximum security on a physical medium then I'd probably suggest copying to a SSD and only recopying the data once every 6-8 years or so.

An online backup / storage could be an even better solution - that way it'll be saved to a massive server array where where faulty discs are replaced with no loss of data. That option also ticks the "offsite backup" box.

With all that said - taking into account the speed that the computing market is moving and Moore's Law, it's almost impossible to know what kind of storage options will be available to us in the next 4-5 years. Rather than actually recopying the data onto the same disc in 4 years' time, you'll probably find it's easier just to buy a completely new disc to copy it to, and then dispose of the old one.

While bit rot is a very real, I honestly don't think you need to be overly concerned unless you're storing important information on optical media (CD-R / DVD-R discs definitely DO degrade quickly).

Hope that helps

 

[dimme]

I just finished a 2+ year project scanning and cataloging our family photos my father took. The collection spans 50+ years and is over 8000 photos. I have the photos uploaded on flicker for the whole family to view. So not I thinking about long term storage of the digital files. Half the photos are tiffs and the other jpegs. Right now they are on my mac mini server drive which is backed up with crash plan. I also have 2 local backups. I started reading about bit rot and wondering if that is something I need to worry about. The drive the main copy is on is a WD my book running 24/7 I figure I would change out the drive every 3 years or so. The local backups are updated every week along with other files that are on the mini server.
Can anyone offer and advise or is my setup pretty sound.
Thanks

I wanted to resurrect this thread form a few years ago to see if any options have changed. My archiving procedure is still pretty much the same as above. I have updated the harddrives to newer ones, and retired the old ones. Still with crashplan (pro version now). I also run a checksum program weekly called Fixity https://www.weareavp.com/products/fixity/.
Any new insights would be appreciated.
Thanks

 

[Apple fanboy]

I wanted to resurrect this thread form a few years ago to see if any options have changed. My archiving procedure is still pretty much the same as above. I have updated the harddrives to newer ones, and retired the old ones. Still with crashplan (pro version now). I also run a checksum program weekly called Fixity https://www.weareavp.com/products/fixity/.
Any new insights would be appreciated.
Thanks

I have my iMac HD, time machine back up and an offsite CCC back up in my drawer at work.
I figure that's safe enough. I don't worry about disc failure as I have three copies. If one goes, I have two left.

 

[kenoh]

I have 2 mirrored redundant arrays in the house then backup to a connected drive locally and then also belt and braces out to AWS Glacier. Good enough for my stuff.

Which reminds me it is time to check again that I can recover files from Glacier.

 

[Clix Pix]

Even as I type on one computer, the other one is busy serving as the go-between as two external drives are busy at work. I am in the process of copying from one to the other so that the latter can go off to my safe deposit box at the bank later today. I have several copies here at home and do regular updates, but what I've been working on more recently is an archival project, and that one definitely needs to be in the bank for safekeeping! It won't be coming back home for updates the way my other drives that I use for ongoing backups do.

 

[dimme]

Even as I type on one computer, the other one is busy serving as the go-between as two external drives are busy at work. I am in the process of copying from one to the other so that the latter can go off to my safe deposit box at the bank later today. I have several copies here at home and do regular updates, but what I've been working on more recently is an archival project, and that one definitely needs to be in the bank for safekeeping! It won't be coming back home for updates the way my other drives that I use for ongoing backups do.

So it sounds like we all are protected for hard disk failure and fire, or theft. But what about the integrity of the data? I have some files that I may not access for years. I have been running a program called fixity that creates a checksum file and I have it set to run weekly to check for bitrot. I was looking to see if anyone else has any insight on data integrity? My fear is I have all these backups that I keep updating, but if the data is bad what good are they

 

[kenoh]

So it sounds like we all are protected for hard disk failure and fire, or theft. But what about the integrity of the data? I have some files that I may not access for years. I have been running a program called fixity that creates a checksum file and I have it set to run weekly to check for bitrot. I was looking to see if anyone else has any insight on data integrity? My fear is I have all these backups that I keep updating, but if the data is bad what good are they

Thats where my redundant arrays come in. Striped with parity and mirrored.

 

[Clix Pix]

Over the past several years I have begun using SSDs for my external drives, for current backups. Now the archived images with which I have been working this past month or longer are also going to be on SSDs rather than platter drives. Two reasons for that: one is that SSDs are much faster to get all the folders and files onto in the first place, and secondly, the SSDs are physically smaller than my mobile platter drives, so they fit much better in my fairly narrow safe deposit box.

I actually started this project some time ago, transferring image files from older external drives, some which still used USB 2, others which used Firewire, and was greatly relieved to find that apparently there was no evidence of "bit rot" even though some of those drives had been sitting, unused, for at least ten or twelve years. Once I had everything consolidated on a few more current external (platter) drives, I then began the process of reorganizing, sorting, etc., which has taken quite a lot of time. After that was done I shuffled things around among my various drives in order to free up a couple of external SSDs to use to stash one set in the bank while retaining another couple of copies at home for further refinement when I'm ready to do so. I will probably bring the SSDs home every so often to make changes or to ensure that the files are still OK, but I don't really anticipate bit failure problems with the SSDs.

 

[0970373]

I wanted to resurrect this thread form a few years ago to see if any options have changed. My archiving procedure is still pretty much the same as above. I have updated the harddrives to newer ones, and retired the old ones. Still with crashplan (pro version now). I also run a checksum program weekly called Fixity https://www.weareavp.com/products/fixity/.
Any new insights would be appreciated.
Thanks

I have my main portable hard drive that backs up to my NAS + Backblaze.

 

[Ledgem]

I think that creating a PAR archive is probably what you're looking for.

PAR files are basically checksummed data generated from the original files. They can be used to check over a file (whether original or copy) and if any corruption is detected, the PAR file can then be utilized to correct or patch the corruption. I first came across them with files on Usenet; perhaps internet connections were flakey enough back then that you had to worry about file corruption in the upload and download process, but the PAR files worked really nicely to restore corrupted files, or regenerate files that were missing.

The Wikipedia article mentions that they were originally used with Usenet, but are now used for things such as guarding against bit rot. So it sounds like this is exactly what you're looking for!

The program I use is MacPAR Deluxe, which is donationware (free, no features are held back, but there's a donation button). Admittedly I don't know how actively it is being maintained; I opened it for the first time in years and updated, but the update says it requires OS X 10.9, at least. Still seems to be running nicely on macOS 10.13.4.

I suppose the way I envision you using this would be to create PAR files from groupings of your photos, and then every year or so, checksum your files to ensure that the files are still pristine.

A number of people have suggested RAID and network backup solutions. The problem with those solutions is that all of them will copy a file, but they won't know if the file itself has become corrupted. The only way to know would be for you to check, yourself (which won't always be apparent), or to have a checksum of the original generated. PAR files don't just allow you to detect corruption, but to repair it. They're pretty useful.

 

[whiteonline]

I think that creating a PAR archive is probably what you're looking for.

PAR files are basically checksummed data generated from the original files. They can be used to check over a file (whether original or copy) and if any corruption is detected, the PAR file can then be utilized to correct or patch the corruption. I first came across them with files on Usenet; perhaps internet connections were flakey enough back then that you had to worry about file corruption in the upload and download process, but the PAR files worked really nicely to restore corrupted files, or regenerate files that were missing.

The Wikipedia article mentions that they were originally used with Usenet, but are now used for things such as guarding against bit rot. So it sounds like this is exactly what you're looking for!

The program I use is MacPAR Deluxe, which is donationware (free, no features are held back, but there's a donation button). Admittedly I don't know how actively it is being maintained; I opened it for the first time in years and updated, but the update says it requires OS X 10.9, at least. Still seems to be running nicely on macOS 10.13.4.

I suppose the way I envision you using this would be to create PAR files from groupings of your photos, and then every year or so, checksum your files to ensure that the files are still pristine.

A number of people have suggested RAID and network backup solutions. The problem with those solutions is that all of them will copy a file, but they won't know if the file itself has become corrupted. The only way to know would be for you to check, yourself (which won't always be apparent), or to have a checksum of the original generated. PAR files don't just allow you to detect corruption, but to repair it. They're pretty useful.

Good advice on parity files, but filesystems such as ZFS and BRTFS have built in protections against bitrot/file corruption. My suggestion is to use a NAS that utilizes one of those filesystems.