Aperture crash, lost master files

mousemd · Jul 12, 2011

I had a HORRIBLE crash with my Mac resulting in the rebuilding of my iMac OS directory with Diskwarrior (thank you man!). Unfortunately, it resulted in the inability to open aperture (due to plug-in failure from what I could gather from the internet/forums). Nonetheless, I reinstalled aperture 3 and was able to recover my aperture library. Now, here is the tough part:

When I opened Aperture now, there are many BLANKS where the pictures used to exist before. It appears that the master files are gone. I believe I kept my library as a MANAGED (default) not referenced but I could be wrong. Nonetheless, I can NOT find the pics that are missing in the library. I managed to find SOME of the pics on my hard drive under a "RECOVERED" folder, but not all of my pics are there. I tried REBUILDING the library, but it did not help with those files that were missing master pics. In fact, instead of showing the # of pics that should be in those projects, it now just shows a BIG FAT 0 for what used to be there.

I THOUGHT I had a backup using Time Machine, but apparently TIME MACHINE never made any back ups (ie the Library (sans some of the files weighs in at 60gig+, the Time Machine backups are like 5 gig).

So, am I totally screwed now?

Thanks for your input guys. I put a question up on the Apple Forums, but didn't get much of a response.

compuwar · Jul 12, 2011

You are likely screwed, which is why I always use REFERENCED files- disk recovery software can recover referenced files, for managed files, it needs to recover the database, and that's less likely- I'd rather have as many of my files as possible recoverable. I also back up my files to a separate disk when I upload them to the computer. That disk gets backed up to an external that goes off-site. I'm actually considering cloud storage as an additional layer of backup.

Why am I telling you this? Because as you have found out the hard way, it's up to you to ensure your files are backed up. Too many backups is way better than too few. Also, you need to TEST your backups at some point when you still have all the originals, not after you've lost them. Hopefully others can learn from your mistakes, and hopefully going forward you'll know what to do.

Paul

mousemd · Jul 12, 2011

Thanks for the response. I have been backing up religiously since a previous problem 6 years ago where I lost some photos as well. I guess it never occurred to me to VERIFY that the file SIZE was correct on the time machine backup. I always assumed that if the folder was there, the file would be correct as well.

In either case, I am on my way of switching to Lightroom and making a cloned drive (thanks Superduper). If by the off chance I don't switch, how can I change my MANAGED files to a REFERENCED File?

Funny thing, on my way to transferring my files OUT of Aperture, someone on the web suggested that I switch out RAW, RAW adjusted, JPG, TIFF, and other files. This method is nice, but the output files are approaching a TERABYTE of data (ok several hundred GIGABYTES). Is this right?!?!?! My Aperture library is less 100GB.

compuwar said:
You are likely screwed, which is why I always use REFERENCED files- disk recovery software can recover referenced files, for managed files, it needs to recover the database, and that's less likely- I'd rather have as many of my files as possible recoverable. I also back up my files to a separate disk when I upload them to the computer. That disk gets backed up to an external that goes off-site. I'm actually considering cloud storage as an additional layer of backup.

Why am I telling you this? Because as you have found out the hard way, it's up to you to ensure your files are backed up. Too many backups is way better than too few. Also, you need to TEST your backups at some point when you still have all the originals, not after you've lost them. Hopefully others can learn from your mistakes, and hopefully going forward you'll know what to do.

Paul

compuwar · Jul 12, 2011

mousemd said:
Thanks for the response. I have been backing up religiously since a previous problem 6 years ago where I lost some photos as well. I guess it never occurred to me to VERIFY that the file SIZE was correct on the time machine backup. I always assumed that if the folder was there, the file would be correct as well.

I've seen enough failed backups that I generally manually back up, if I didn't, I'd want to test restore.

mousemd said:
In either case, I am on my way of switching to Lightroom and making a cloned drive (thanks Superduper). If by the off chance I don't switch, how can I change my MANAGED files to a REFERENCED File?

http://aperture.maccreate.com/2010/07/30/moving-items-between-referenced-and-managed/

mousemd said:
Funny thing, on my way to transferring my files OUT of Aperture, someone on the web suggested that I switch out RAW, RAW adjusted, JPG, TIFF, and other files. This method is nice, but the output files are approaching a TERABYTE of data (ok several hundred GIGABYTES). Is this right?!?!?! My Aperture library is less 100GB.

Exporting adjusted files will result in complete files being written where just changes are stored in the database. Hard drives are cheap though- I use a USB2/SATA dual drive dock, and buy regular bare internal drives- drive capacities keep growing quickly enough that I don't run out of space as I copy drives to new ones and add images. However, I've recently gotten ready to focus stack large images, and potentially panoramas, so at that point, I'm sure I'll have to move to a second drive.

Paul

Clark Kent · Jul 14, 2011

In the future, don't trust Time Machinecand Time Machine alone to back up irreplaceable files like photos.

Use a vault and/or a clone to be safe.

mousemd · Jul 15, 2011

Thanks Clark. I have learned my lesion for the second time now. I am backing up with vault. But in the process of switching to Ligthroom. Exporting the damn files from aperture is nuts (raw, raw a dusted, tiff, jpg) has resulted in hundreds og gigabYtes if not over a terabyte of data. Pain in the ass. So I've had to buy a bunch of hard drives in the process to compensate. My back up plans include FireWire bootable 1tb drive with super duper clone, drobo 8tb drive/time machined on it. And then another backup of the photo folder. I've been debating on using carbonate as well, but seeing how the photos are taking 1-2 days to transfer to the hard drive I can't imagine what it would be like to baack up over the Internet.

I feel like it seems a bit of overkill to have raw, raw adjusted, tiff, and jpg but I can not prove to myself that I don't need ALL of those files. Too many pics. I don't have the time to review each one.

MacProDude · Jul 15, 2011

Your masters are probably still there. It's really just a hidden directory mechanism in OSX. Here's something you can do to find all files with a certain file extension over a certain size in your Aperture Library.

Right-click on your Aperture library and select "Show Package Contents". It'll open a finder window that shows the contents just like any other directory.

Go into the Masters subdirectory. Your master files should all be in there somewhere. Mine appears to be by year, then by month, then by import date/time. I assume that's the way it is for everyone.

If not, then you can look for large files in your Aperture library with the file name extension for your masters (*.CRW, *.CR2, *.NEF, whatever). Something like:

find . -name "*.CR2" -exec ls -l {} \; | awk '{if ($5 > 20000000) print $5" "$9}'

Replace "." with the path to your Aperture library.
Replace "CR2" with the file extension for the files you are looking for
Replace 20000000 with a number smaller than the minimum file size for your camera (20 million bytes in this case) but larger than most other files (eliminates thumbnails, etc if you're searching for .png or .jpg, etc).

It could take a while to run if your library is large. And you might need a bit more help figuring out what to do with the results, but that's a start for you, and maybe we can all help you find your originals. I certainly hope you don't have to try this second method.

Good luck.

MacProDude · Jul 15, 2011

Oh, sorry, I didn't read how you had to recover the files via diskwarrior. They might not be there, but that probably has nothing to do with referenced vs managed. If the filesystem is hosed, it can take out either of them because in either case, they're just simple files on the file system.

mousemd · Jul 15, 2011

Macprodude

I can assure you that have done the former with opening my library and looking for the master files. The dates are there but there are 0 files under the file folders for the .cr2 files or whatever.

The latter method intrigues me because I have never thought about doing a search like that before. Is this in terminal via command line? Or some other method?

mousemd · Jul 15, 2011

@MacProDude -

I still don't understand how my system just died the way it did. I appreciate your input re: Diskwarrior recovery. I didn't think that it would kill the files in the folder. I thought it was just restore the directory.

MacProDude said:
Oh, sorry, I didn't read how you had to recover the files via diskwarrior. They might not be there, but that probably has nothing to do with referenced vs managed. If the filesystem is hosed, it can take out either of them because in either case, they're just simple files on the file system.

MacProDude · Jul 15, 2011

Yeah, it's in the terminal, aka from the shell.

find . -name "*.CR2" -exec ls -l {} \; | awk '{if ($5 > 20000000) print $5" "$9}'

Find from currently directory (.) files that fit "*.CR2". For each file found, do an "ls -l". if the fifth string on the "ls -l" result (which is the file size) is over 20,000,000 bytes, then print the fifth string (the size) a space, and then the ninth string (the path/filename).

Honestly, I forgot to remove the printing of the fifth string, as I did that just to verify the command did what I wanted. This would be fine:

find . -name "*.CR2" -exec ls -l {} \; | awk '{if ($5 > 20000000) print $9}'

MacProDude · Jul 15, 2011

mousemd said:
@MacProDude -

I still don't understand how my system just died the way it did. I appreciate your input re: Diskwarrior recovery. I didn't think that it would kill the files in the folder. I thought it was just restore the directory.

I can't say I do, either.

I didn't mean to imply that diskwarrior hosed anything. It's possible that the filesystem was hosed to the point that diskwarrior simply couldn't recover everything/anything.

Ruahrc · Jul 15, 2011

I'm not understanding why if you had a big disk failure but had a TM backup, you didn't just reinstall the OS clean and do a Time Machine restore, which should have put everything back to a working state? Rather than try to recover what seems like a severely damaged disk using Diskwarrior and then trying to cobble things back together from there? If this didn't work, then the TM backup would have been corrupted in which case it would not be the first time I have heard such reports.

Do you still have an unadulterated TM backup (i.e. have you been backing up to TM since the restore or is your TM backup still as it is pre-crash)? If so I'd try wiping your disk, installing the OS again, and restoring from TM backup.

What kind of crash exactly was it that you suffered?

Ruahrc

mousemd · Jul 16, 2011

Yes I do have a time machine backup...sort of. Time machine FAILED to back up my Aperture library. Yes there was a copy of the file/folder, but it was INCOMPLETE. For example the damaged folder (the one I am trying to cobble together) is around 60 to 100 gb give or take. The backups on time ,aching were ~5 to 6 gb. Trust me, all I wanted to do was do a clean install and start from scratch but that would've meant even MORE lost photos.

Of course, when I rebuilt with diskwarrior, I didn't think that it would've deleted some of my photo files either. I guess the point is I didnt trust Time Machine enough just the WIPE my hard drive from th beginning and do the Time Machine restore. In retrospect, I'm glad I didn't domthatdue to the failed backups.

As far as what kind of crash I suffered, I have no idea. I was trying to boot camp into windows and suddenly I got the blue screen of death. Something about issing some files which was preventing windows to load properly. So I thought I would just boot into Mac os, but there was NO OPTION to do so at start up. I tried to use the disk utility program that came with the install disk on my iMac and the response I got with that program said that the directory was seriously damaged and that I should they and back up was many files I could and then reinstall the the OS. However I couldn't access the drive to do so. In retrospect, when I booted th iMac in target mode via FireWire using my macbook pro I should've tried and looked at the directories to see if I could recover some files, but i thought hey I have doskwarrrior and it had fixed the problem before on my other macs along with my drobo. Plus I also have a time machine backup (or so I thought). So I still have no idea how the directories became corrupted in the first place.

Ruahrc said:
I'm not understanding why if you had a big disk failure but had a TM backup, you didn't just reinstall the OS clean and do a Time Machine restore, which should have put everything back to a working state? Rather than try to recover what seems like a severely damaged disk using Diskwarrior and then trying to cobble things back together from there? If this didn't work, then the TM backup would have been corrupted in which case it would not be the first time I have heard such reports.

Do you still have an unadulterated TM backup (i.e. have you been backing up to TM since the restore or is your TM backup still as it is pre-crash)? If so I'd try wiping your disk, installing the OS again, and restoring from TM backup.

What kind of crash exactly was it that you suffered?

Ruahrc

dampsquid · Jul 23, 2011

Mousemd, I know how you feel. Yesterday I did a complete restore to revert from a Lion upgrade. Of course given a Lion upgrade was a major change, I made sure my TM backup was good and current.
Post restore the first thing I noticed was my desktop background had changed to a stock Apple one. A couple of other settings were changed too. A little odd. At some point I went to Aperture to get my preferred desktop image - and discovered it couldn't find any masters. I tried rebuilding the database etc. to no avail. I even restored an earlier Aperture library - same. Then I restored an earlier library and the actual App (it had changed recently to 3.1.3, so I made sure the app was 3.1.2 to work with a library backed up when the app was 3.1.2 - same.
Next it hit me, my library was around 14GB instead of 200 or 300GB. Right clicking on the library shows the folder structure for the masters is there - just no images, at all! And it's the same for every version Time Machine has. I have absolutely no idea why this is. Initially I blamed Lion, the irony of course is on Lion APerture and it's library was there, and working just fine.
I have already used File Juicer (on another machine) to pull the thumbnail/preview images from the library. So I have a misnamed, mis-dated version of every picture at least, but they only go to 1024 pixels.
Overall, I have lost 19,000 images covering 100 years of family history and 25 years of personal world travel. I actually feel physically sick. All of this on an Apple machine, using an Apple app, backed up with an Apple product.
As of now I'm scanning all the drives in my machine with file recovery software hoping I can at least find something to salvage. We'll see.
So, my friend, all I can offer it solidarity.

mousemd · Jul 23, 2011

@dampsquid...
I feel your pain. Let me know what program u used to recover some of those photos if you do recover any. I have given up with what I was able to recover but then again I didn't try those programs you had. My only fear is that if you tried to restore and or replace your database with an older version without trying to recover first you are likely in the same boat as I am...with GONE files. I also assume u did what I did with Managed instead of a REFERENCED library. I am in the process of switching to Lightroom. It is slower in some ways and I am not used to the way it organizes the photos.I am not sure I will stick with Lightroom but after my experience with the time machine back up fiasco I am wary of the program again. I have backed now with the Vault program which is the ONLY way to back up Aperture apparenty. In any case good luck with your attempts at recovery!

bunnkristie · Jan 21, 2013

getting my photos which is lost in a dramatic crash down of my operating system... it was cutting my nerves down.. remo photo recovery software i had used to get them back.. still i had not recovered my data back.. i only worked on the jpeg file format...

Bear · Jan 22, 2013

mousemd said:
...
I THOUGHT I had a backup using Time Machine, but apparently TIME MACHINE never made any back ups (ie the Library (sans some of the files weighs in at 60gig+, the Time Machine backups are like 5 gig).
...

Just to be clear, did you check multiple backups?

Did you set Time Machine preferences to not back up any files?

----------

Clark Kent said:
In the future, don't trust Time Machine and Time Machine alone to back up irreplaceable files like photos.

Use a vault and/or a clone to be safe.

Yes, it's sometimes good to use different backup methods.

No matter what backup methods are used, one should occasionally examine backups after they're made to make sure they look okay.

Laird Knox · Jan 22, 2013

I suspect the OP has moved on by now.

wolfpuppies3 · Jan 23, 2013

Ya think? After two years?

Aperture crash, lost master files

macrumors member

macrumors 601

macrumors member

macrumors 601

macrumors member

macrumors member

macrumors member

macrumors member

macrumors member

macrumors member

macrumors member

macrumors member

macrumors 65816

macrumors member

macrumors newbie

macrumors member

macrumors newbie

macrumors G3

macrumors 68000

macrumors 6502

Our Staff