Like a lot of you on this form I have a large archive of photos from over the years. Awhile back I had a program that I would run a few times a year to check for data corruption in my photo archives. (Never had an issue) That program is no longer supported in the current MacOS. After searching and talking with chatGPT I don see much out there. I am wondering if it is even necessary. I do have multiply backups, but if there is corruption on the main archive it just get duplicated to the backups.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Anyone using software to scan their photo archives for bit rot or data corruption.
- Thread starter dimme
- Start date
- Sort by reaction score
how would files get corrupted? if the segment of the drive they're on gets corrupted, right? or the entire drive? or do I miss something.
so checking the actual drive for corruption should do imho.
How do you backup? do you override old backups frequently? I use CCC and it backs up changed files only, having said that, if a file were corrupted - is the actual file corrupted or the directory entry with the pointer to the actual location on the drive? hmmh
I'm not shooting professionally so only have ~ 1.25TB or so of photos, all on my internal SSD of 4 TB. Backup onto another 4 TB SSD every night, backup to 3 different HDDs on a regular basis. I like to recycle those HDDs every 3 years or so, don't trust them for longer.
I have never had a SSD fail on me, neither die nor corruption etc, HDDs, yes those are more problematic imho.
The OP is referring to bit rot, where some bits can get flipped from their intended state over time (0 to 1 or 1 to 0). It can happen to all kinds of storage media. Sometimes it may go unnoticed (i.e. the photo still looks fine, but some pixels have the wrong color); or if certain bits get flipped, it can make the whole file unreadable. If you copy a file that has been affected (i.e. restoring the file, or copying it to new drive for long term storage), you just end up copying the bad bits without knowing it.
Checking the drive for corruption does not detect bit rot because the bits/sectors are working fine...they just have the wrong state, nor do these types of scans know what each bit should be. You need to check the files against their original checksum (hash) to scan for bit rot.
While it is relatively rare to happen, I would say so, especially if these are archived files for longer-term storage (i.e. no longer on your computer and part of your regular/more frequent backups). Better safe than sorry. You could consider using a NAS; they usually handle this automatically. That is what I use for my primary backups and has made it much easier for me to manage, and gives me peace of mind that it's checking my data integrity frequently.
I guess it would depend on how bad the bit rot is. If you can open the file and there a just a few artifacts they could be cleaned up in photoshop. It regards to having a exact copy it will depend on when the coy was made, if it was done when the bit rot all ready occurred they the back would also have bit rot.
This should be done at the file system level, not by a user-run app. My NAS does this on a fixed schedule
About the file corruption being copied to the backup. This is why backups like Time Machine are versioned. The old data should never be overwritten. If your backup system overwrites old data, replace it
One thing you can do is take a snapshot of the entire APFS volume. This has near zero cost and takes near zero space but will record the current state forever. Seems like magic but that is how "COW" or copy on write works. THere is a command line program to do this already on the Mac
However, if you really wanted to check that all files are still readable, simply copy them to /dev/null using the terminal.
Some experience here on archival. The only way you can truly survive bitrot is storing the things in uncompressed TIFF format with forward error correction. Any compressed formats (JPEG/HEIC/RAW) tend to get heavily damaged by single bit errors. And single bit errors are accepted by FLASH manufacturers in a high order (expect 1-2 bits a year corruption in a TB disk every year). For us it was TIFF + FEC and store on DAS and LTO offline. Which is expensive as TIFFs are huge, FEC makes them bigger, both storage tiers make it even more expensive and the software (IBM) make it eyewatering.
However this is impractical for the average user. What is important is you actually verify your data is what it is supposed to be periodically and have at least two backups from which you can assemble a corrected version from. That's how you play the probability game on consumer levels of storage.
Another way to play it is store less data. You are less likely to suffer from probabilistic corruption if you have less data. So delete all those crappy photos - they are increasing your risk of single bit errors. Granted they might appear on one of those crappy photos, but you don't want to then have to recover it and find out it was a crappy photo.
Simple statement:
1. Keep your data small and tidy.
2. Keep multiple copies of it
3. Regularly verify your data is what you expect it to be.
-or-
1. Just accept the risk and wing it on time machine backups which is what I do despite knowing all this.
However this is impractical for the average user. What is important is you actually verify your data is what it is supposed to be periodically and have at least two backups from which you can assemble a corrected version from. That's how you play the probability game on consumer levels of storage.
Another way to play it is store less data. You are less likely to suffer from probabilistic corruption if you have less data. So delete all those crappy photos - they are increasing your risk of single bit errors. Granted they might appear on one of those crappy photos, but you don't want to then have to recover it and find out it was a crappy photo.
Simple statement:
1. Keep your data small and tidy.
2. Keep multiple copies of it
3. Regularly verify your data is what you expect it to be.
-or-
1. Just accept the risk and wing it on time machine backups which is what I do despite knowing all this.