Master Files

[Razeus]

I'm wondering how to go about creating and archiving "master" files of my photos.

Mind you these wouldn't be the original RAW images, but my the files created from those RAW's with all the color correction, skin softening, conversion to b&w, cropping, etc. that makes the final, full resolution photo.

These "master" files would be used to create prints of all sizes (so I don't have to dig through Lightroom) and to be the "definitive" version of a file, so when I die and someone can just look at the folder and say "this is what he wanted to show" 😎.

Should I save these as .jpegs or tiffs? I'm aware of how big .tiff files are and with my own tests, I simply can't see the difference between a 100% quality (using Lightroom) jpeg or a tiff, yet the jpeg is sooooo much smaller.

 

[simsaladimbamba]

While a JEPG might look exactly like a TIFF, the TIFF stores a lot more information for colour, luminance, colour models and more you do not see. A TIFF is meant for storing image data for later manipulation or print.
A JPEG is used for final delivery in the consumer range, thus if you just want to share the final master and not "allow" future generations that much manipulation, a JPEG is totally fine.

 

[acearchie]

I agree with above.

If you want to print from the images than a TIFF is going to have far more information to get strong vibrant colours realistic to how you have edited them, something that can't quite be said of a JPEG.

A good example that I have seen the difference in before is uploading a TIFF and a JPEG to an image hosting site. The TIFF will hold its detail and colours after being compressed by the host but the JPEG falls apart a bit.

 

[ChrisA]

I'm wondering how to go about creating and archiving "master" files of my photos.

Mind you these wouldn't be the original RAW images, but my the files created from those RAW's with all the color correction, skin softening, conversion to b&w, cropping, etc. that makes the final, full resolution photo.

These "master" files would be used to create prints of all sizes (so I don't have to dig through Lightroom) and to be the "definitive" version of a file, so when I die and someone can just look at the folder and say "this is what he wanted to show" 😎.

Should I save these as .jpegs or tiffs? I'm aware of how big .tiff files are and with my own tests, I simply can't see the difference between a 100% quality (using Lightroom) jpeg or a tiff, yet the jpeg is sooooo much smaller.

Simple. Just buy Aperture. It works as you describe. There is not "hunting for the definitive image. Aperture always shows the edited version of the file. It stores your raw files and stores the edit commands. Aperture is non-destructive so it make a good archive.

 

[Designer Dale]

Simple. Just buy Aperture. It works as you describe. There is not "hunting for the definitive image. Aperture always shows the edited version of the file. It stores your raw files and stores the edit commands. Aperture is non-destructive so it make a good archive.

This won't get him what he wants, which is a single file/folder with nothing but final versions of selected images. He wants to archive these only on a seperate reference drive so he (or his next of kin) doesn't have to dig through everything that Aperture keeps.

Dale

 

[OreoCookie]

This won't get him what he wants, which is a single file/folder with nothing but final versions of selected images. He wants to archive these only on a seperate reference drive so he (or his next of kin) doesn't have to dig through everything that Aperture keeps.

The workflow in Aperture does exist: you create versions and make a pick. The pick is the »master« (which is not a master in any sense). If you collapse the stacks, all you see are the »masters« and you don't have any clutter.

 

[acearchie]

The workflow in Aperture does exist: you create versions and make a pick. The pick is the »master« (which is not a master in any sense). If you collapse the stacks, all you see are the »masters« and you don't have any clutter.

But this requires the end user to have aperture.

I believe Razeus is looking for a more long term solution whereby he can export a high-res copy of his files from lightroom/aperture/photoshop with all the edits made for an archival purpose.

There is no point saving all the raws and .xmps (in lightrooms instance) only for lightroom to stop opening them in 5 years time. Not a very solid archival process.

 

[OreoCookie]

But this requires the end user to have aperture.

This »technique« works also in Lightroom.

I believe Razeus is looking for a more long term solution whereby he can export a high-res copy of his files from lightroom/aperture/photoshop with all the edits made for an archival purpose.

There is no point saving all the raws and .xmps (in lightrooms instance) only for lightroom to stop opening them in 5 years time. Not a very solid archival process.

I don't agree: as long as he continues to use Aperture/Lightroom, the »technique« works (I use quotation marks, because this is how these pieces of software are meant to be used in this instance). When it is time to switch away from Aperture/Lightroom, it is then when the OP should think about rendering his files and writing them to disk. The whole point of Aperture and Lightroom is that they are the tool with which you manage your photos from import to export.

Wanting to export files for archival purposes is of course no different than exporting them for any other reason, but you're duplicating built-in functionality.

 

[acearchie]

I don't agree: as long as he continues to use Aperture/Lightroom, the »technique« works (I use quotation marks, because this is how these pieces of software are meant to be used in this instance). When it is time to switch away from Aperture/Lightroom, it is then when the OP should think about rendering his files and writing them to disk. The whole point of Aperture and Lightroom is that they are the tool with which you manage your photos from import to export.

I think you have missed the point I was trying to make.

Razeus states, 'when I die and someone can just look at the folder and say "this is what he wanted to show"'.

Although a bit sombre the point is valid. If you have these files in Aperture/Lightroom/any other raw editor they can be edited by mistake for a start and if he's anything like me all my top images are mixed in with my OK and only kept for personal reasons photos... Not a great first impression if someone in the future uncovers the great Razeus collection of 2013.

This »technique« works also in Lightroom.

It also works in Capture One, Darktable, Camera Raw, Phocus, etc... but it does not work between the applications. If I have lightroom and you have aperture we cannot exchange raw files with the changes to then be edited in each others applications.

What is the printer has Capture One? How will he get the print.

My interpretation is that Razeus wants a folder where he can export high res images that in essence are "finished" and will remain in that state for ever. For me leaving them as .Cr2 or .nef or whatever in Aperture/Lightroom etc. is not a good idea. Take for example Lightroom updating their process version update with Lightroom 4 which meant that the photo could be processed in a different way. Already in one version change we have two different ways for a photo to be altered. Who's to say that when the next process version comes out support isn't dropped for PV2010. What will happen to the changes then!

A tiff file is a tiff file. It theoretically opens the same on every computer baring monitor and colour space differences.

If Razeus wanted to share his collection with the world as a folder on line leaving it as raws and their supplementary files instantly limits who can open the files.

Leaving it as a tiff means that it can be opened by pretty much anyone with a computer.

In your first quote you mention:

The whole point of Aperture and Lightroom is that they are the tool with which you manage your photos from import to export.

Surely this is what Razeus is trying to accomplish. He has finished his edits and is ready to export?

yet the jpeg is sooooo much smaller.

It's all relative though. So much smaller but realistically it's 15MB-ish per image (I assume). With storage getting cheaper and cheaper I don't really see photo files being such an issue.

Last weekend I shot a short film on the new Blackmagic Cinema Camera. It fills up a 240GB SSD in 34 mins. The rushes for the total 5 minute film totalled nearly 900GB. Suddenly my 250GB 10,000 photo library didn't seem that big!

 

[swordio777]

Hands down, use JPEG! TIFF is an archaic file format from the 80s and hasn't been updated in over 20 years. JPEG isn't pefect, but it's close and certainly meets your needs.

Save your jpegs at maximum quality. If you shoot landscapes and want the best quality then use 16-bit, you pay for this in filesize but it will keep smooth gradations in the skies. This only really makes a difference in areas of very smooth gradation where 8-bit can cause banding - For anything other than cloudless blue sky though, the diffeences between 8bit & 16bit are very hard to see.

Hope that helps!

 

[Razeus]

This won't get him what he wants, which is a single file/folder with nothing but final versions of selected images. He wants to archive these only on a seperate reference drive so he (or his next of kin) doesn't have to dig through everything that Aperture keeps.

Dale

But this requires the end user to have aperture.

I believe Razeus is looking for a more long term solution whereby he can export a high-res copy of his files from lightroom/aperture/photoshop with all the edits made for an archival purpose.

There is no point saving all the raws and .xmps (in lightrooms instance) only for lightroom to stop opening them in 5 years time. Not a very solid archival process.

Correct.

I'm assuming if something should happen, I have a hard drive with a folder called Master images with all the high rez, final edits, it it. I'm assuming my wife or some other non photography computer person would find this and would need an easy way to view these files without the aid of software she has no idea how to use and what all my stars, colors, flags, etc. mean (I use Lightroom). I'm looking at jpeg and tiff since I believe these 2 formats will go on in infinite. Also, these files serve as an easy way to make prints without have to dig through LR everytime and do an export.

 

[Designer Dale]

Correct.

I'm assuming if something should happen, I have a hard drive with a folder called Master images with all the high rez, final edits, it it. I'm assuming my wife or some other non photography computer person would find this and would need an easy way to view these files without the aid of software she has no idea how to use and what all my stars, colors, flags, etc. mean (I use Lightroom). I'm looking at jpeg and tiff since I believe these 2 formats will go on in infinite. Also, these files serve as an easy way to make prints without have to dig through LR everytime and do an export.

This is morbidly true, but worth thinking about. I'm 10 years older than my wife and when I die I'm certain she wouldn't even be able to find my Flickr page, let alone navigate Aperture. Before I discovered the importance of backup, I kept a folder called Processed Files that held print quality copies of everything I posted to this forum. That was when I edited with Photoshop, and I should probably get back to that practice.

And may we all live long and prosper.

Dale

 

[snberk103]

While I am normally the fellow who starts off with "But Lightroom can do...." - - in this case I understand perfectly what you want to do.

A lot of this is going to be hardware dependent. But... I think what you want is a dedicated HDD just for these images. If you have a MacPro, it could be an internal one if you have the space. Otherwise, an external HDD on a fast connection. You don't want to be waiting while Lr/Apertue/Ps/etc is writing the images to the HDD.

I don't know what kind of work you do, but the folder structure needs to make sense to a 3rd party. So maybe organized by theme, or show, or something. I wouldn't save every size for any one image, just the biggest print size. Add a text file to the root of the HDD that explains what your intentions are for the prints in general. You could periodically update the text file if you need to add specific guidelines.

Next... you need to backup this HDD. A couple of cheap USB2 drive is adequate. It just needs to run overnight to clone the primary image HDD. Carbon Copy and SuperDuper have a feature that copies only the changes to produce the clone. One of these backup HDDs should be offsite. It would be unfortunate, but not tragic, if some of your latest work was lost so the rotation cycle on/off site can be a bit more leisurely than say your System Files. At least in my opinion.

I would opt for TIFF, as painful as the space penalty may be. The fellow I use to print on occasion - and he is at truly professionally printing guru - says that Full JPGs are just fine to send him - but then he adds the caveats.... if they don't have an extreme range of colours, if they don't have an extreme range of tones, etc. So for most purposes JPGs are fine, but for the really good stuff...TIFFs are better. If these are your "Master" files.... then I would save the TIFFs. It just means that as one HDD fills you will need to retire and safely store 2 or 3 HDDs (primary plus backups) - and of course buy the corresponding number of new HDDs for the new cycle.

Addendum: Organize your images by Year/Month on the HDD. Add an inventory as a text file to the root of the HDD. The text file could be organized by Date as well, with a description for each image including keywords (not necessarily the same as your DAM). Chances are if you are printing for a show or a commission, they will all be done in the same month. Even if your create Masters over two months for one show, I would 'fudge' it and put them into the same month. The point is to keep similar Master together and easily found.

 

[Razeus]

This is morbidly true, but worth thinking about. I'm 10 years older than my wife and when I die I'm certain she wouldn't even be able to find my Flickr page, let alone navigate Aperture. Before I discovered the importance of backup, I kept a folder called Processed Files that held print quality copies of everything I posted to this forum. That was when I edited with Photoshop, and I should probably get back to that practice.

And may we all live long and prosper.

Dale

True, true. I don't think it's good to tie up all my work in Lightroom as alot of it is unfinished or uploaded to Flickr for some kind of feedback as I try out various images, not necessarily my best stuff. I don't have any other form of a photo other than the RAW itself and since LR does the backend work when it uploads to Flickr, I don't make jpegs of stuff. From time to time I'll make jpegs for emailing or Dropboxing to other people, but those are on a case by case basis and usually at a lower rez.

The point, again, is to have an easily accessible place to store choice files of my work (however bad it may be, but it's what I enjoyed doing in life) so that others may access it, be it a family member or what have you. Heck, they may want to make a few large prints to display during the wake or whatnot. I

It's just something that I can leave behind, but also something I can use presently as I'm about to start actually printing some of my photos in various sizes as items to frame and display around my work area and around the house - something for people to see when they come to my home instead of just viewing things online.

I appreciate the advice and I'll just use 16-bit TIFF files for the "master" copies.

 

[snberk103]

...
The point, again, is to have an easily accessible place to store choice files of my work (however bad it may be, but it's what I enjoyed doing in life) so that others may access it, be it a family member or what have you. Heck, they may want to make a few large prints to display during the wake or whatnot. I
..
I appreciate the advice and I'll just use 16-bit TIFF files for the "master" copies.

Going to add one other thought, which is in addition to my post above. The external HDD (backup or primary) should be clearly marked with a big Red Dot and text that says "Take Me".... or something like that.

I'm thinking of a case of an evacuation (highly unlikely, but then again this whole thread is about dealing with a difficult issues). You look around the office, and the Big Red Dot reminds you that you need to grab that HDD. Also the one with your financial and personal statements... or whatever is important. If it is a family member who is scanning the office, they don't need to know what the contents of the HDDs are ... you have simply told them to grab whatever has the Big Red Dot. They'll have to figure it out from there.

 

[ijohn.8.80]

I'm very late to this conversation, but, I'm mildly surprised no one mentioned DNG's as an option.

If you are dealing with finished flat (not-layered) images I'd be going with TIFF's to maximise compatibility for the future and retaining quality. I use snberk103's type of archiving approach, with annual, monthly and daily/activity type folders used. I prefer to break down the days activities to individual folders so I can find them easily later on. I really like the idea of a text file inventory also, good thing I don't have thousands of keepers at this early stage!

My folder structure e.g.

2013

05.May

24.Beach.Port.Noarlunga.Jetty
24.Macro.Feathers.New.Holland.Honey.Eater
24.Pets.Dogs​

 

[chrfr]

TIFF is an archaic file format from the 80s and hasn't been updated in over 20 years.

This is exactly why TIFF is the preferred archive format.

 

[OreoCookie]

Razeus states, 'when I die and someone can just look at the folder and say "this is what he wanted to show"'.

I did not take this part of the post literally, I thought Razeus meant it figuratively.

Although a bit sombre the point is valid. If you have these files in Aperture/Lightroom/any other raw editor they can be edited by mistake for a start and if he's anything like me all my top images are mixed in with my OK and only kept for personal reasons photos... Not a great first impression if someone in the future uncovers the great Razeus collection of 2013.

You can also create an album in Aperture or Lightroom with your eternal best ofs.

It also works in Capture One, Darktable, Camera Raw, Phocus, etc... but it does not work between the applications. If I have lightroom and you have aperture we cannot exchange raw files with the changes to then be edited in each others applications.

What is the printer has Capture One? How will he get the print.

If someone knows what Capture One is, I suppose he or she will be able to export a file from Aperture or Lightroom as well. Come on.

My interpretation is that Razeus wants a folder where he can export high res images that in essence are "finished" and will remain in that state for ever. For me leaving them as .Cr2 or .nef or whatever in Aperture/Lightroom etc. is not a good idea.

I disagree strongly: the RAW files are the unadulterated negatives and as long as you have the file, you can reprocess them if you have software that can read that format. Having software at hand that can read it is just like storing your negatives and slides properly so that they don't fall victim to mold. Except that it's much easier to store bits than to store pieces of plastic that is sensitive to light and the elements.

What you really store in Aperture/Lightroom/whatever are edits, tons of metadata, relationships between files (e. g. when you create albums that have nothing to do with the project structure, say a »best of 2012« album or a collection of all your architecture shots). This is invariably lost if you don't migrate it, and it is this data that takes hours and hours to complete. Put more succinctly, there is a lot of state that is not contained in the rendered file. Moving this over when switching to another piece of software is always an issue and for that reason, people focus way too much on files. I learnt this the hard way when I switched from iView Media Pro 2 to Aperture (because the former was always corrupting the catalog file after a while), almost all of my work was gone even though iView does not touch the file structure or files themselves.

Take for example Lightroom updating their process version update with Lightroom 4 which meant that the photo could be processed in a different way.

I don't know how Lightroom handles this, but Aperture does take this situation into account: if you process photos with another version of the RAW processor, the update does exactly nothing. Aperture will continue to use the old version to process the photo unless you manually tell it to reprocess the photo. (Again, I don't use Lightroom, so I don't know how Adobe handles this case, but I doubt they haven't thought of that eventuality.)

Surely this is what Razeus is trying to accomplish. He has finished his edits and is ready to export?

He wants something that can be accomplished from within these applications (he wants a best-of album) and all of the woes go away (which format to use, etc.) if he sticks to that. Of course, it's his files, so he can handle them any way he sees fit, just saying.

 

[Designer Dale]

True, true. I don't think it's good to tie up all my work in Lightroom as alot of it is unfinished or uploaded to Flickr for some kind of feedback as I try out various images, not necessarily my best stuff. I don't have any other form of a photo other than the RAW itself and since LR does the backend work when it uploads to Flickr, I don't make jpegs of stuff. From time to time I'll make jpegs for emailing or Dropboxing to other people, but those are on a case by case basis and usually at a lower rez.

The point, again, is to have an easily accessible place to store choice files of my work (however bad it may be, but it's what I enjoyed doing in life) so that others may access it, be it a family member or what have you. Heck, they may want to make a few large prints to display during the wake or whatnot. I

It's just something that I can leave behind, but also something I can use presently as I'm about to start actually printing some of my photos in various sizes as items to frame and display around my work area and around the house - something for people to see when they come to my home instead of just viewing things online.

I appreciate the advice and I'll just use 16-bit TIFF files for the "master" copies.

This got me thinking about the difference between the masters of the film age and those of us here in the digital age. When Ansel Adams and those like him moved on, they left us with a legacy of film negatives that live on to this day. We of the digital age, with all our technical expertise, have muddied the waters and need to get our acts together or all of our fabulous stuff will be lost to future generations of admirers.

Just some musings of an old man...

Dale

 

[ChrisA]

This won't get him what he wants, which is a single file/folder with nothing but final versions of selected images. He wants to archive these only on a seperate reference drive so he (or his next of kin) doesn't have to dig through everything that Aperture keeps.

Dale

OK buy Aperture. Select the photos he wants then click "export versions" and then choose a file format (tiff, psd, jpg,...). It will build a folder with the selected images (with all edits applied) in the selected format. You can have it use the album name as part of the exported folder structure if you like.

----------

This got me thinking about the difference between the masters of the film age and those of us here in the digital age. When Ansel Adams and those like him moved on, they left us with a legacy of film negatives that live on to this day. We of the digital age, with all our technical expertise, have muddied the waters and need to get our acts together or all of our fabulous stuff will be lost to future generations of admirers.

Just some musings of an old man...

Dale

Almost all thefiles will be lost. hardly anyone backs up their data well enough that it could last 50 years, let alone 100.

The trouble with Adam's negative is that he had to apply some very involved corrections by hand to each print. He would choose a paper contrast, a deveopmer and a toner and he would apply dodging and burning by hand. none of this was saved. Yes he made notes but it is not the same. Now today we make edits and we can save the edits. In that way we have it better than he did.

But very few people understand backups

 

[MacCruiskeen]

When Ansel Adams and those like him moved on, they left us with a legacy of film negatives that live on to this day.

I've saved decades' worth of negatives, even though it is unlikely that anyone will ever want to print my summer camp photos from the 1970s. On the other hand, when working on a family photo project for my mother, I was annoyed that _no one else_ in the family ever seems to have saved negatives.

But if you really wanted to, there's nothing preventing you from making physical backup of your files--output the files onto film and store the film. See http://www.cr.nps.gov/hdp/standards/PhotoGuidelines_Nov2011.pdf

 

[Razeus]

Yes, this will apply to my family photos too, but those will be jpeg. So jpeg for family photos, tiff for my more "serious" stuff.

The text file that I will create in the Master Folder will contain login id/passwords for Flickr, which is where I plan to store the photos online, but I still need the "hard copy" masters in a folder on a hard drive.

 

[Cheese&Apple]

This is morbidly true, but worth thinking about. I'm 10 years older than my wife and when I die I'm certain she wouldn't even be able to find my Flickr page, let alone navigate Aperture. Before I discovered the importance of backup, I kept a folder called Processed Files that held print quality copies of everything I posted to this forum. That was when I edited with Photoshop, and I should probably get back to that practice.

And may we all live long and prosper.

Dale

I'm older than my wife as well but I don't worry access to my photos. When I'm gone, my wife will unplug my computer, sell it along with all my camera gear and head-off to the Bahamas with the cabana boy. All the while she'll be saying: "Thank God all that photography foolishness is over". 😀

 

[ApfelKuchen]

...You can also create an album in Aperture or Lightroom with your eternal best ofs...

I'm an Aperture fan, and completely understand where you're coming from, but...

IF I didn't want someone mucking with my final edit, just ONE version of the image (like a painter or composer who destroys all drafts and all works outside the catalog/opus), I'd go with the export-a-TIFF approach, too. If I was really concerned about damage to my legacy, I might arrange to have the entire Aperture library wiped, lest an unedited original finds its way to daylight.

When it comes to legacies, heirs really don't like to figure things out - they're sorting through piles and piles of stuff - "Donate this? Try to sell it to a resale shop? Call in an appraiser and hold an auction?" This is why guests on Antiques Roadshow can come up with such great dumpster finds. Most heirs just don't know what's valuable, or are too exhausted by the process to care.

Sure, there might be another family member (or art historian) who might want to curate the legacy and would like to have the entire Lightroom or Aperture package. The chances of either? Pretty slim.

So yeah, I vote for the "make it easy to find, easy to use, and impossible to muck up" approach. Only disagreement? I'd do both TIFF and JPEG. Almost everything that displays a graphic can display a JPEG. TIFF viewing is not quite so universal. Any barrier, even the smallest, can make a difference. "What is this? I don't know, but the file won't open!"

I might go so far as to create a fully-formated and captioned album suitable for computer display - make casual family viewing dead simple - with instructions that the TIFFs be used for printing or high-quality display. Self-contained albums are far more likely to be appreciated than a folder full of TIFFs, and might lead to the TIFFs being treasured, rather than trashed.

 

[ChrisA]

Yes, this will apply to my family photos too, but those will be jpeg. So jpeg for family photos, tiff for my more "serious" stuff.

The text file that I will create in the Master Folder will contain login id/passwords for Flickr, which is where I plan to store the photos online, but I still need the "hard copy" masters in a folder on a hard drive.

Like I said above, Aperture will build this for you with only a few clicks. Select the photos, choose the target folder and the file format and it will copy out or "export" a bunch of files.

Thetrouble is that no matter how nice your storage media is. Say you buy a new hard USB3 hard drive or even a flash drive (because you KNOW hard drives in storage have a limited live, the bearings freeze up) OK some you have four copies of your work. stored on high-end media. I 20 years someone wiil see iithis as "old computer junk" the toss in in the trash. Just exactly like I do today when I find and old floppy disk. I have no way to look at what is one it and if I did 90% are unreadable. I toss them. Your drives will get chucked out too.

One of the worst problems with digital archives is that the media becomes obsolete, You have to remain active and copy the old data to new media. If you stop then in 15 or so years it's gone.