Efficient Archival Strategies ( or, how I can't get anything right)

Discussion in 'Digital Photography' started by rroback, Feb 3, 2009.

  1. rroback macrumors member

    Jun 29, 2007
    I'd really like an efficient way of storing my photos that both backs them up, and leaves them easily locatable, and usable. Initially, I would just copy them to different folders, and had all the folders properly labeled. Seemed easy enough, worked ok, but ended up with lots, and lots of folders. Then, I said, what the heck, aperture ( v.1 at that point) sounds great, but it was slow, and I couldn't get used to its features. I ended up with a combo of folders, and the aperture library. Finally upgraded to v2 which was better, but I can't get the system down. I'm dealing with something like ~10,000 raw images +5,000 jpg.

    The computer is a powermac g5 quad 2.5, 2 500gb drives, with a 1.5 gb mybook setup for time machine.

    My biggest issue is the the aperture organization works on when I go from camera-->card reader when I'm home, but I get screwed when I download photos on the road to my laptop, and then get boggled reorganizing them on my tower. I hate the folder/project titles too, and "smart" options. Do I just need to suck it up, and learn ap more, or is there really a better option ( lightroom?). The photos all go through extensive post-processing btw.

    thanks for your suggestions.
  2. ajpl macrumors regular

    Oct 9, 2008
    You need to have a good filing system that is independent of any programme or OS and once that is done you can add keywords that allow for fast searching.
    Belt and braces.

    I use date and description for folders. And will do more than one folder per day as well to keep different jobs/subjects separate.
    --2009-01-01 New Years Day at Home
    --2009-01-04 Snow in Peak District
    --2009-01-04 Product shot
    --2009-02-02 Grid Lock due to snow

    Also by doing this, you will then tend to remember when things happen better than before. The benefit o this system is that I can use any software to look through my files and it works on any OS.
    Though you have to label the files like I've done year then month, then day to order correctly and I also do 2009-01-January, not 2009-January, so files will order correctly. The dashes are to aid reading as is using the month's names. This also avoids the confusion of 10-08-2009 which could be August or October [depending on where you are from] and that way of labelling doesn't order correctly either.
    I also name files '2009-01-04 Snow in Peak District 001.CR2' again it orders correctly [with date at front], it's easy to find folder to place it in if you move or copy files.
    Some people use '20090104snow_in_peak_district_001.CR2' which is much harder for humans to read. If you need underscores for files destined for web - when you save them, they canbe added then. Done automatically if using Save for Web in PS.
    I've tried numerous variations of file/date naming and now this is the only one that actually works well.

    A date heirachy works for any photographer as you can add subject to date by keywording. Filing by subject simply does not work, as you end up with pictures all over the place and most pictures can be many subjects.
    You can also use a File Browser like Bridge/PathFinder/Finder to look through and sort/manage your files. Database apps like Aperture are not so good for that sort of thing. LR at least will display your actual folders and hard drives, so you don't have to guess where things are. I also use Bridge a lot as it is much better than LR for some things and being able to use both a File Broser and a Database to manage your files is much easier than the database only method, which is the Aperture/iPhoto method. It also means I can change my software at any time, without worrying about my filing system. How likely are Aperture or Lightroom to be around in 20/30/40yrs time?

    For professional work, you may choose to do things differently and file by job/client. e.g. a Wedding Photographer may file by Couple's name as each job is discrete and if having to refernce it in future, they will probably be asked for the work by name. And if using something like Lightroom you would have a new catalogue for each wedding.

    Aperture or iPhoto are awful [IMHO] when it comes to file management as they like to do things their own way and this does not play so nicely with the sort of filing system I suggested.

    Also you can back up your files very easily to different HDs or Optical discs using a file browser, disc burner or back up software if using this dated folders method. And any software can access the files.
  3. OreoCookie macrumors 68030

    Apr 14, 2001
    Sendai, Japan
    The problem is that you have a hybrid solution and get mixed up in the middle. It sounds to me that, independently of Aperture, you don't have a proper strategy figured out on how to store your photos. Then, you apparently don't use Aperture properly. First of all, if you want to use it, install it on your laptop. Apple allows this explicitly in the license agreement (one desktop, one laptop, they may not run concurrently).

    So when you import photos onto your laptop, do it in Aperture. Work in Aperture and if you are ready to transfer the files to your desktop, export them as a project. This means it will bundle internal Aperture-specific information (edits you have done to the images, tags, etc.) with the image files themselves. If you were to use Lightroom, you'd have to do the same (transfer not just the files, but the edits you've done with the files).

    The second thing regards your overall file management. This is a matter of taste and how you work: if you do a lot of edits from within Aperture and add a lot of tags, having the files at your disposal means nothing, the work is in the edits and tags. I love tagging, rating and sorting my images. This is not (and in many instances cannot) be reflected in the file and folder structure, so I let Aperture manage my image files for me. These files are easy enough to extract if I need to.

    Unlike what ajpl has suggested, you can also choose to manage your image files manually in Aperture or mix and match at will. But then, you must maintain discipline and order, because no one is helping you here. It's up to you to sort your photos in a sensible folder structure and if you fail to do so, it's your mess ;) Again, this is independent of which app you actually use, iView, Bridge, Lightroom or Aperture.

    How have you sorted images before using Aperture? What is the structure you have in mind? Mine is [Continent] > [year] > [year].[month] > [year].[month].[day] - [name of event]. With the exception of Continent, it was essentially the same I used before switching to Aperture (I used iView which needs you to sort your image files in folders by hand), so Aperture didn't change anything in that respect. If I were to use Lightroom, that wouldn't be any different, either.

    Now I'm a bit more flexible, because Aperture allows me to create subfolders and subjugated objectes to a specific project (e. g. books, albums and web galleries). I'm letting Aperture manage my files, because (i) I don't want to do things twice (create a project and an associated folder by hand), (ii) most of my work is done in tags, books and edits -- which aren't reflected in the file structure at all and (iii) I can be sure that backups of my Aperture Library (be it via Time Machine or Vaults, I use both) contain all of my images. (Aperture vaults also save image files you have deleted.)
  4. GotMyOrangeCrus macrumors regular

    I recommend you organize outside of Applications like Aperture or iPhoto and then you can reorganize once your in those Apps. Personally I organize them on DVD and use Light Impressions archival storage to store the DVD's. I then use Extensis Portfolio as a database for those organized DVD's. I can then organize them however I want in Aperture without having to worry. You just have to figure out a system that works best for you as everyone is different in this regard. I will say that you should quickly invest some time into figuring out what methods you want to use as the longer you wait, the bigger headache it is when you decide to switch it up. Trust me, I know all about it as I have switched up on a number of occasions and with over 5000 DVD's burned it is a nightmare. I can say that the system I now use is perfect for me so thankfully I wont have to start from scratch ever again.
  5. ChrisA macrumors G4

    Jan 5, 2006
    Redondo Beach, California
    My opinion....

    Don't work so hard. Your Aperture license allows you to install it on one notebook and one desktop. Do that. Now when you are on the road put the images into an Aperture "Project" (Yes project, not folder or album) when you get home it is very easy to move a project because projects are, in Aperture, self contained (they contain the thumbnails, previews meta data ans so on) Export then inpot the project using your network.

    As for over all organization. Apertures "Project" concept works well. create a new project for each well, new project or "shoot". You might keep the projects inside Aperture folders and keep you family vacation snapshots in a folder and paid wedding photography in another folder.

    When you first import the card or direct from camera you need to create or select a Project and then, right away tag the images with meta-data. Once you have tagged the image files you no longer need to care where they are physically on the disk.

    Backup: There are a couple rules you must follow. Any backup system that follows then is OK.

    (1) Your files must always exist on at least three different physical media. (And critically, this include durring a backup operation. So if your backup system wipes a disk clean you will need four copies.)

    (2) Your files should exist in at least two geographically different locations. (Rotate one backup set to your office or home. Just make sure to deliver one copy before yu take the other back so one copy is always at the off site location. You will need at least two backups sets to do a rotation.

    The best way to implement this is to use Time Machine and just lest in run 24x7 on the largest disk or RAID you have (Yes get the largest disk you own to TM) and then buy a few other disks and rotate those. I keep one off site, one in a fire safe and one near the computer and rotate which is where. When one disk gets to small I replace it with the largest size disk then on the market and retail the oldest one.

    How to copy data to a backup? Use Apertures "Vault" system it works well and is fast. I frequently update the Vault on whichever disk is nearest the computer. The disks at the office and fire safe or more out of date but the TimeMachine backup is always up tio the hour

    Notice I did not talk about how to organiz files into folders. Just don't do that. It's a useless hold over from the 1980's Just put them all into Aperture or if you like, some other asset management system. Pull the images out using meta-data tags.
  6. GotMyOrangeCrus macrumors regular

    Sure thats a good system if you dont have a lot of data however it becomes excessively expensive as your collection gets bigger and bigger. I guess it would depend on just how much he is planning on shooting and what his financial capabilities are. IMO the cheapest and safest way to back up your pictures are to DVD's. DVD's cost 25 cents per 4.3 Gig disc. I burn 2 discs per back-up so its 50c per 4.3 gigs. With HD's your going to be paying much more and HD's are not as reliable as Discs. I passed 5000 DVD Back-Ups (10,000 Discs) a few months back and if I were to keep those backed up on HD, I would have to have a massive RAID setup that would be massively expensive. Now obviously he doesn't have that much to store but my point is as time goes on and you shoot a lot, that type of backup system is going to become expensive and he might be forced to switch up his entire organization system and as I explained earlier, switching up and starting from scratch is hell. Best to find a system that will work for you long term and stay with it. If he can afford the type of system your talking about long term then its certainly a viable option.

    If you start your organization with backed up DVD's, your growth potential is unlimited. All you have to do is buy more cases to store them in. Once you have them backed up on to DVD, you can pick and choose which ones you want to keep stored on your computer and then organize those pictures in Aperture, ,iPhoto or something like Portfolio which is what I use. Then you dont have to worry about anything but how you want them organized. They are already backed up so you dont have to worry about running Time Machine or even having a hard drive dedicated to time machine.

    DVD's IMO are the single best option with which to start your organization and back up plan with. They are cheaper, more reliable and again they offer you unlimited growth at just 25c intervals or more like 20 dollar intervals for 100 packs of DVD's. As for storing the DVD's there are a multitude of options. You can use any DVD storage device out there. Again I use Light Impressions Archival boxes for my really important discs and then use Gator 300 DVD soft cases for the rest of them.

    Again your method is certainly a viable option, its just not one that I would use or recommend someone use. Pretty much every photographer I know, including myself, started out using a system like the one you explain and then ended up switching to the kind of system I explained.
  7. OreoCookie macrumors 68030

    Apr 14, 2001
    Sendai, Japan
    Current harddrives cost about 12-15 Euro cents/GB. That's twice as much. However, work time costs several times that. Ditto for storage. Backing up to multiple harddrives is the better option IMHO. DVDs are also useless if you want to back up changes you've made to your files.
  8. GotMyOrangeCrus macrumors regular

    Yes consumer level HD's cost around 2 times more, actually more like 2.5 times more, yet higher quality HD's which I would use if I were backing up my pictures cost around 3-4 times as much. DVD's give you 430 Gigs of back up space for 20-25 dollars. You simply cant beat that and as time goes on that savings becomes SUBSTANTIAL. Again If I were to use the kind of system you guys are talking about I would have easily spent another 5-6 thousand dollars, probably more. Plus when you use DVD's you dont have to keep all of your pictures on your computer thus it alleviates alot of the clutter you get with large collections.

    I will openly admit that if your not going to be shooting a lot and you have the money to burn then the backing up on HDs is a viable option. If you shoot a lot its simply not a good option and trust me I found out the hard way on this one. It took me almost 3 months, like 3 hours a day when I decided to switch my organization system from HD's to Discs.
  9. OreoCookie macrumors 68030

    Apr 14, 2001
    Sendai, Japan
    One hour of my work time costs about as much as a harddrive. Two, three years ago, I've burnt a complete backup of my data onto ~50-75 DVDs. Took me three weekends.

    Also, what's a quality drive to you? SAS? To me, that's not really necessary as I'd rather have the data on three `cheap' drives (internal harddrive + 2 backups) rather than on two expensive ones.

    Even worse, backups to DVDs cannot be incremental (well, at least not effectively), so they can't do the job anyway.
  10. ChrisA macrumors G4

    Jan 5, 2006
    Redondo Beach, California
    Even worse, how do you test a stack of DVDs to know if the backup set is still OK? Easy, you read and verify every DVD once every month. With a hard drive testing is MUCH easier because you use the drive in rotation and periodically re-write the backup set to the drive. This process also will detect a failed drive within one rotation period of failing.

    A backup that is defective is worse than useless. Your backup system should have some kind of check designed into it to verify that you are not writing data into a black hole.

    As for cost, DVDs about equal hard drives now. I bought a 1TB drive for $100 and I bought a box of 100 blank DVDs for $34 the DVDs hold about 4GB each so 100 DVDs hold 400GB. I'd need about three boxes to hold 1TB. Roughly the same price.
  11. GotMyOrangeCrus macrumors regular

    One hour of work time? Backing up discs doesn't take any time at all. I do my backups while I am already at my computer. It only takes around 30 seconds for me to set up a burn and catalog it. Once it starts burning I go back to doing what I was before whether its photoshop of surfing the net. The time you spend on these forums you could be backing up discs while you were here. Again burning and cataloging my disc takes about 30 seconds of my time, nothing more and it certainly wouldn't have to take any more for you guys as well.

    No I used Lacie Hard Drives for my backups back in the day. They used to run about twice as much as consumer drives but they have come down in price. Again I have saved at least 5-6000 thousand, probably much more, by switching to discs and its a much better system as again it allows you to pick and choose what pictures to put on your computer and gets rid of the clutter pictures that you never use.

    Again almost every single photographer I know has gone through the same exact cycle that I did. Starting out using the HD method and then as your collection gets much larger over the years ends up switching to a disc based back up and every one of them including myself wished we had gone with discs from the get go as it would have saved us a ton of money and a ton of time that was spent switching over.

    Again if you dont shoot often or you have a small collection of photographs and by small I mean less than 2-3 TB, then HD's are certainly a viable option. For people that shoot a lot, its a bad idea, lol. Dont get me wrong I still store a lot of pictures on my computer. I just dont back them up on my computer. I now use a Lacie 7.5 Gig RAID for the pictures I do keep on my computer and even that is running out of space. Maybe you dont shoot a lot so you dont see just how big the space requirement becomes over time.
  12. GotMyOrangeCrus macrumors regular

    LOL, testing DVD's every month. Thats funny. I make 2 backups of each disc and I have yet to have a single problem. You dont have to test your back ups because DVD reliability blows that of HD's away. The odds that both of your DVD's are gong to go bad are astronomical.

    Testing DVD's, lol. The only thing you have to worry about with discs is in the burning. Once its verified your golden.
  13. dlegend macrumors 6502

    Jan 11, 2009
    Northern VA (outside DC)
    I personally do a hybrid of all of the above systems. I backup my computer onto an external hard drive and burn 2 CDs (or DVD's pending on the size). I check to make sure they work on another computer before I put them away. One copy stays here, another goes to my parents. I find it works well, but I don't shoot TONS of pictures like some of you.
  14. rroback thread starter macrumors member

    Jun 29, 2007
    I guess I need to give aperture a real try. I believe it just copied my files to its special location, and managed them, but I wasn't into labeling them, so they confused me. A little about me:


    My photo work is similar to sports photographers I guess, as in I shoot lots of photos in short intervals ( ~150 in 5-6 minutes), end up editing about ~15, and deliver 1-3. I do like keep nearly all of them, except the most clearly unusable ( i.e. totally blurry).
  15. OreoCookie macrumors 68030

    Apr 14, 2001
    Sendai, Japan
    Nice pics. How to you take them, on a helicopter?

    To be honest, if I were you, I'd start by thinking how you want to store pictures in the end, and I mean the specific hierarchical structure here. How you realize this in the end is another matter. As I said, the logical structure of the data doesn't change whether you let Aperture handle file storage automatically or you end up using only the Finder and Photoshop. Once that is in place, we can help you make a workflow adjusted to that.

    Regarding storage, I know some pro photographers who simply bill their client for that, e. g. one of them bills them for a harddrive and then puts one job (which is bigger than in your case) onto that one harddrive.
  16. ChrisA macrumors G4

    Jan 5, 2006
    Redondo Beach, California

    No, there is a big difference. Using only the finder you must have your photos in only one specific hierarchical structure. Most people choose to go with dates or maybe "job number". But in any case you must choose only ONE

    Aperture and others let you put a photos in as many places as you like. The photo of (say) Mary in San Diego taken in Feb 2008 can go into three folders one called "Mary" another called "San Diago" and then in the "2008-Feb-12" folder. and then it might also show up inside the "four stars and better" or into a folder named for the designed of the dress she wore. The image file of course is physically on the disk only once but is linked to make places.

    The best think here is that you can change your mind later, re-define the definition of the folders the the file sort them selves into the newly re-defined folders. So your file system can evolve over time and changing needed with out a lot of effort. But if you have 10,000 images in finder folders there are not likely to be re-organized often
  17. OreoCookie macrumors 68030

    Apr 14, 2001
    Sendai, Japan
    I don't really disagree with you here that this is possible (and desirable), but from what I could gather, the OP's problems were more basic than that. I think he should start with a `pseudo file-folder structure' and then work his way up (e. g. by adding other ways to sort pictures, e. g. smart albums which filter by type or location). Also, he needs a certain project structure as files reside only in one project (although they may reside in multiple albums, galleries, etc.).

    I guess what I'm saying is that the OP should start simple.
  18. ajpl macrumors regular

    Oct 9, 2008
    And you are stuck using Aperture until you die or Aperture ceases to exist.
    You keep spouting this quite frankly naive advice and IIRC last time you suggested this, peolple were queing up to say "Don't do it!"
    There is nothing useless about a good filing structure that is application and OS independent. Depending on a single programe to organize and archive your file is simply stupid. Programmes get dropped, replaced, companies go bankrupt - Apple have nearly gone under once already and as current times are showing, no company is ever completely safe. Even Adobe have laid off a lot of people recently.
  19. OreoCookie macrumors 68030

    Apr 14, 2001
    Sendai, Japan
    In almost all cases, the work is not contained in the file structure and you simply have to export when you migrate. Same goes for Lightroom. Image edits (crops, changes in exposure, etc.) are lost, as are photo albums, web galleries, books and so forth. Since most people nowadays shoot RAW, all the file-folder structure contains is the undeveloped RAW file!

    Being able to read it right away with another program does zilch. If you migrate, you export your projects with the changes. Obviously, this comment does not depend on whether you use Aperture, Lightroom or any other app.
  20. rroback thread starter macrumors member

    Jun 29, 2007
    I'm not sure aperture really is for me. I wish there was a more seamless way I could import all my raws, immediately have small jpg's created ( for web gallery), some large jpgs, and easy access to edit the photos. I'm sure some combination of aperture and automator could get it done, but I have yet to find the right path.
  21. ajpl macrumors regular

    Oct 9, 2008
    Finder is crap anyway and only a buffoon would use that to organize anything.

    {quote]Aperture and others let you put a photos in as many places as you like. The photo of (say) Mary in San Diego taken in Feb 2008 can go into three folders one called "Mary" another called "San Diago" and then in the "2008-Feb-12" folder. and then it might also show up inside the "four stars and better" or into a folder named for the designed of the dress she wore. The image file of course is physically on the disk only once but is linked to make places.[/quote] I can do that in Bridge, even though it is a mere File Browser.

    You don't quite understand filing it seems and using Finder is certainly painful and not a tool I ever use if I can help it.

    You can file your images into your folders [as I said above, I do that by date+ description] and then add you can add metadata for smart searches. That way you can search by folder or metadata. Best of both worlds. Files in my date folders can also be in numerous subject folders, it is not exclusive folders or metadata as you can happily use both. LR originally went down the Aperture route of not bothering with folders and during the public beta it was quickly realised, it didn't work that well, so now you have folders and metadata.

    I'm working on some images at the moment in Bridge, they are part of a very long 5 week shoot with about 4-5hrs sleep a night, so there's a lot of work to be gotten through. When I go through my folders and I find shots I like I simply add them to an approriately named collection. I can add the shots to as many collections as I want, despite the fact that they are in folders. I can also do smart collections so I give the smart collection some criteria and anything that matches the criteria will end up in the apposite collection. On a long shoot like this, filing by date is very useful as I tend to know when things happen, as on certain days of the week, certain events would happen.
    I can also use LR to work on same job and although LR is a database programme like Aperture, both Bridge and LR can work on the same images as they are filed sensibly so either approach can be used.
    It also means that I can use Capture One or Phocus for Phase One or Hasselbald Cameras without any difficulty or changing my filing system.

    The problem with metadata filing [which is what Aperture does] is that it is very, very, very time consuming to do properly and when you repeatedly photograph the same subject or do long shoots over several years, filing just by metadata can become next to useless as you end up with an awful lot of images with the same tags. But if you sorted by folders first, then you have two options for searching and can use a variety of programmes to do so.
    Aperture seems quite easy at first with a few images, but if you are a busy shooter then, it ain't so simple. A chap who shoots for Alamy, spends his time shooting and his wife spends all her time adding metadata to the images so they can be found - it's a full time job for her.

    But whatever method you use for any filing, a good logical heirachy is a must, unless you only have a handfull of categories. Working out a subject heirachy for keywording is not an easy task, if you want to do it properly.
    I've yet to finish working out my metadata heirachy and I guess it will end up with a few thousand labels, at least, a painful task.
  22. ajpl macrumors regular

    Oct 9, 2008
    Not true, In LR you can save the changes to an .xmp file, within a DNG or inside the file itself. This means you can use Bridge and look at the developed photos and they wil be the same as in LR.
    As the .xmp file is simply a text file listing what has been done and not proprietry, it would be relatively simply for other software to then read it and apply suitable settings in their software. Obviously it would have to have all the features used to do it properly, but in a few years time the software will have settled down and copied all each other's tricks.
    It's the same as Adobe reverse engineering RAW files to enable LR/ACR to do it's job.

    This .xmp file is another very good reason why a good filing system is a must as if your catalogue goes a bit squiffy, your edits are not trapped in the programme's database, but are with the file in it's folder.
  23. ajpl macrumors regular

    Oct 9, 2008
    Easy to do that in either LR or Bridge. In LR you hit export and tick the relevant boxes for what you want doing to your images such as resizing and saving and in Bridge you use the image processor - 'Tools/Photoshop/Image Processor and again choose what you want doing - from Bridge you can also use PS's action in the processor or just actions on images selected from Bridge.
    But in both LR and Bridge you can simply use your RAW files to go create a web gallery, without having to leave the programme.
    Aperture can also produce webpages with pictures on as well as books.
  24. OreoCookie macrumors 68030

    Apr 14, 2001
    Sendai, Japan
    Small jpgs are immediately created. Simply drag and drop your RAW images from Aperture into the Finder or onto the Desktop and Aperture will create lower-res jpgs.

    Or you can use the export function with a preset of your liking -- which also is very fast (File > Export > Export Working Copy (not sure about the last one, I use the German localization)). You also have easy access to Photoshop from within Aperture, simply right-click an image you want to edit and select to Open it with Photoshop (or whatever image editing app you happen to use). Do not save as in Photoshop, simply save. Aperture recognizes the fact that you have updated the file. Also, you never, ever edit the original, Aperture will always create a `working copy'.

    Lightroom works in the same way.
  25. GotMyOrangeCrus macrumors regular

    Just remember that there are certainly other options out there and you dont even have to use Aperture or Lightroom if you dont want too. Everyone is different in regards to the system they prefer. The bottom line is there are a lot of options and you should at least give them all a try before making a final decision. Who knows that one app you didn't try just may end up being exactly what your looking for. The good thing is most apps have free demos that let you test out the software before buying it. Im actually surprised more people dont use some of the other programs that are out there like Extensis Portfolio. I actually prefer it over both Aperture and Lightroom.

    Oh well again to each their own. Everyone has an opinion on this matter and the only thing that matters is what works for you. Again just dont wait too long before coming to a decision as switching later on can be an utter nightmare and again that is something I know all too well.

Share This Page