Understanding how Photo-Editors store my work

[reetcher]

sure let's be hopeful that Darktable has decades of longevity - I personally thought DigiKam was going to be my primary cataloging software.

 

[Ambrosia7177]

sure let's be hopeful that Darktable has decades of longevity - I personally thought DigiKam was going to be my primary cataloging software.

I tabled this product for now, and have already forgotten, but one of the pieces of open-source software I was looking at for something (?) only has like one or two developers on their dev team, and they appear to be in their 60s or 70s. (I think it is maybe DarkTable.)

So in that scenario, if the open-source dev team dies off - and no one takes over - then you aren't any better off than if say Affinity goes 100% subscription, or a proprietary app (e.g. Aperture) stops being developed, or if company XYZ goes out of business.

I guess that was partially @cSalmon point...


Which is why I do so much research, and try to think about all of these scenarios BEFORE I take the plunge!! ;-)

 

[r.harris1]

I tabled this product for now, and have already forgotten, but one of the pieces of open-source software I was looking at for something (?) only has like one or two developers on their dev team, and they appear to be in their 60s or 70s. (I think it is maybe DarkTable.)

So in that scenario, if the open-source dev team dies off - and no one takes over - then you aren't any better off than if say Affinity goes 100% subscription, or a proprietary app (e.g. Aperture) stops being developed, or if company XYZ goes out of business.

I guess that was partially @cSalmon point...


Which is why I do so much research, and try to think about all of these scenarios BEFORE I take the plunge!! ;-)

Of all of the products you're researching, have you sat down and tried any of them? That's literally the best research and I can think of no penalty for doing so for your stated use cases.

 

[Herbert123]

Thanks for the insight.

PhotoMechanic looks like a nice app, although I see they have switched to the nefarious subscription model like so many others. (There is a perpetual license, but it is rather expensive for 1 year of updates.)

XnView MP is the free alternative to PhotoMechanic. At least, free for non-commercial purposes - otherwise a small fee for a traditional software license!

Before choosing a solution, I need help understanding how all of these applications store my work and how they interact with the original photo file.

• DAMS (such as XnView MP and PhotoMechanic) almost always use some kind of local relational database to keep track of thumbnails, meta info, etcetera. This speeds up operations considerably when managing multitudes of images. DAMS generally have tools to filter out duplicates, photos with similar subjects, and allow the user to categorize images.
• Photomanagement apps such as Adobe Bridge, PhotoLine's built-in Browser and mini-browser and others do not depend on a database. Or at least, they may create local XML or other text-based files to add information to folders. Bridge uses XML files.
  
  These apps may also use the XMP metadata in image files to save meta information in. This only works for image file formats that support XMP metadata in their file structure. PhotoLine's image browser allows us to save additional info in images' metadata - even keywords and categories. Some DAMS may also save metadata in images.
• RAW developer software (LightRoom, DarkTable, ...) may also use a local database. Usually this type of software creates sidecar files (often XMP/XML files) when the original RAW files are edited and these files are saved alongside the original RAW files to enable non-destructive RAW development. The original files are left alone. This type of software may have some kind of image management features built-in as well.
• Image editors have built-in RAW development usually too, but the controls and quality may not be on par with dedicated RAW developers. Or at the very least it requires more time and effort (and knowledge/experience) to attain that quality. But it depends on the image editor in question.
  
  Image editors may also have an option to link to your original images and load the original external file. This is similar to how Resolve, Premiere, Adobe InDesign, Blender, and other media creation software behaves.
  
  For example, I can load any file that PhotoLine can read as an external linked file. When the external files are updated, the embedded file(s) in PhotoLine update automatically. This also is a feature of Photoshop via Smart Objects.

I'll complicate matters a tad more

• Image editing software may combine the above in their feature sets to a varying degree.

PhotoLine is quite unique in this respect that it combines all of these options:

1. destructive editing is possible
2. non-destructive editing is possible
3. any imported bitmap layer retains its original resolution, bit depth, image mode, colour profile, ppi, and transparency. These do not change when the image mode, bit depth, etc. is changed of the main file. It does not depend on smart objects and (unlike Affinity) these images are destructively editable. This shares similar behaviour with NLE and FX editing software (like Resolve).
4. images can be placed as external links that update when the external file is changed/edited.
5. an image browser is built-in to manage image files, similar to Bridge.
6. image meta data can be browsed and edited for each image, including categories and keywords. And many more.
7. (!) PhotoLine provides an option to save a native source file alongside edited images.
   
   For example: I open a jpg. I add layers, effects, text, and adjustment layers, and so on. I save. The jpg is saved with all my edits (a flattened jpg file).
   
   The next day I double-click on that jpg, and presto: all my non-destructive edits are still in place!
   Magic! How is that possible?
   Well, PhotoLine saved a sidecar file in its own proprietary PLD format alongside the original JPG. It saved the new version, but the original version is still retained in the sidecar file.
   
   This is not the same as a sidecar file such as the ones generated in RawTherapee or Lightroom. The principle is sort-of the same.

I use this last option all the time. It saves an extraordinary amount of time! No longer do I need to "save for the web" or "save as" or export... I can work with the output file directly and seamlessly yet have all the non-destructive features of PhotoLine made available to me. Very, very powerful and convenient.

This is probably why I have always organized my digital files by creating verbose filenames and building elaborate directory structures. (If those become corrupt then I don't have a computer.)

I will propose a much nicer solution to you in place of bastardizing the file names of your images. Use their metadata containers instead.

I add categories, keywords, and other info to my images, and then use XnView MP's Metadata-->Update Catalog from Files command to teach XnView that it should catalog all that metadata.

Advantages:

1. no reliance on a database for my image info, cataloging, and keyword data
2. all the extra info about an image resides in the image itself. Everywhere it goes, it stays (unless I decide to erase that data manually).
3. DAMs such as XnView MP can easily "learn" about that metadata, so I am not depending on any particular piece of software to ensure my images can be managed.
4. DAMs such as XnView MP can easily edit and write metadata in multiple images, and manage/search these via the metadata.
5. most image processing software can read the metadata. Or at least partially. It's great for archiving images independently from any specific software. And many free tools exist to inspect the metadata in an image. And there exist extensions for operating systems to expose this metadata as well.
6. when taking photos a camera or device already adds tons of useful data anyway.

How about that, he? All your fears - leave them behind.

 

[Ambrosia7177]

Of all of the products you're researching, have you sat down and tried any of them? That's literally the best research and I can think of no penalty for doing so for your stated use cases.

@r.harris1, I am trying to get my hard-drives done by end-of-year (EOY) so temporarily put this research "project" - not "product" - on hold.

Yes, you are right, I do need to dive in and start playing around with few of these.

For managing lots of photos, I am interested in DarkTable since it mimics Lightroom but is free.

For photo-editing, Affinity Studio looks good.

And there have been lots of other suggestions.

 

[Ambrosia7177]

It's funny how old threads seem to "rear their ugly heads" when you least expect it?!

My head was sooo not into this conversation this week until now.

And @Herbert123 always makes my head spin!!

But it is all good, and s-o-m-e-d-a-y I will apply all of this wisdom for a solution!

Here is a short response, since a lot of what you were saying sorta went over my head this a.m. - I was in the middle of a folder-comparison when this thread popped back up.

XnView MP is the free alternative to PhotoMechanic. At least, free for non-commercial purposes - otherwise a small fee for a traditional software license!

PhotoMechanic looks interesting, but the price is obnoxious.

Will check out XnView when I have time.

DarkTable looks like a good solution for me, but I was disappointed in that it doesn't have a built-in file manager. (And it was a sick April Fool's joke that the creator made when he posted online how DarkTable has a new file manager...)

DAMS (such as XnView MP and PhotoMechanic) almost always use some kind of local relational database to keep track of thumbnails, meta info, etcetera. This speeds up operations considerably when managing multitudes of images. DAMS generally have tools to filter out duplicates, photos with similar subjects, and allow the user to categorize images.

Can you dictate anything about the database? Like which database platform it uses? (e.g. PostgreSQL or MySQL)

Can you export said databases so that you have a backup and can restore the data should you ever need to - independent of the application.

Photomanagement apps such as Adobe Bridge, PhotoLine's built-in Browser and mini-browser and others do not depend on a database. Or at least, they may create local XML or other text-based files to add information to folders. Bridge uses XML files.

These apps may also use the XMP metadata in image files to save meta information in. This only works for image file formats that support XMP metadata in their file structure.

So which formats support XMP metadata?

PhotoLine's image browser allows us to save additional info in images' metadata - even keywords and categories. Some DAMS may also save metadata in images.

RAW developer software (LightRoom, DarkTable, ...) may also use a local database. Usually this type of software creates sidecar files (often XMP/XML files) when the original RAW files are edited and these files are saved alongside the original

RAW files to enable non-destructive RAW development. The original files are left alone. This type of software may have some kind of image management features built-in as well.

Based on limited research, one thing that I liked about DarkTable was its use of sidecar files...

[*]Image editors have built-in RAW development usually too, but the controls and quality may not be on par with dedicated RAW developers. Or at the very least it requires more time and effort (and knowledge/experience) to attain that quality. But it depends on the image editor in question.

This is an entire other topic.

However, for the foreseeable future, I will just be shooting native iPhone files, and NOT the Apple ProRAW whatever - I have heard some bad things about Apple's proproietray format, but I will create a NEW thread on this...

Image editors may also have an option to link to your original images and load the original external file. This is similar to how Resolve, Premiere, Adobe InDesign, Blender, and other media creation software behaves.

For example, I can load any file that PhotoLine can read as an external linked file. When the external files are updated, the embedded file(s) in PhotoLine update automatically. This also is a feature of Photoshop via Smart Objects.

Are you talking about sidecar files?

I wasn't able to finish reading up on - and completely understanding - how DarkTable uses sidecar files. As I recall, DarkTable gives the database priority, and the sidecar files are treated more as a "backup" - not sure I liek that.

I'll complicate matters a tad more

*Brain overload* Will come back to this later...

I will propose a much nicer solution to you in place of bastardizing the file names of your images. Use their metadata containers instead.

I add categories, keywords, and other info to my images, and then use XnView MP's Metadata-->Update Catalog from Files command to teach XnView that it should catalog all that metadata.

My iPhones shoot .JPEG for photos and .MOV for video.

Don't both of those formats support adding metadata about the files?

And can't any decent application allow you to embed metadata into those files? (Or does the metadata have to get stored in a sidecar file or in an internal database?)

Knowing how I think and work, moving forward, I have pretty much settled on using this naming convention...

<date>-<time>_<camera>_<camera app>_<description>.jpeg

<date>-<time>_<camera>_<camera app>_<description>.mov

For example...

20251031-1944_iPhone11_iOS_Mary at Halloween Party.jpeg

20251216-0925_iPhone15_CP3_John in Chicago.mov

Advantages:

1. no reliance on a database for my image info, cataloging, and keyword data
2. all the extra info about an image resides in the image itself. Everywhere it goes, it stays (unless I decide to erase that data manually).
3. DAMs such as XnView MP can easily "learn" about that metadata, so I am not depending on any particular piece of software to ensure my images can be managed.
4. DAMs such as XnView MP can easily edit and write metadata in multiple images, and manage/search these via the metadata.
5. most image processing software can read the metadata. Or at least partially. It's great for archiving images independently from any specific software. And many free tools exist to inspect the metadata in an image. And there exist extensions for operating systems to expose this metadata as well.
6. when taking photos a camera or device already adds tons of useful data anyway.

So you are saying you put all of your metadata INSIDE the actual file?

And that was you are not dependent on applications?

And you are also saying that most applications can read a file's own embedded metadata?

How about that, he? All your fears - leave them behind.

You are a wealth of information, @Herbert123!!

 

[Herbert123]

DarkTable looks like a good solution for me, but I was disappointed in that it doesn't have a built-in file manager. (And it was a sick April Fool's joke that the creator made when he posted online how DarkTable has a new file manager...)

I assume you have checked out RawTherapee as well? That is my favourite RAW developer.

Home

About RawTherapee is a powerful, cross-platform raw photo processing system, released as Free Software (GPLv3). It is designed for developing raw files from a broad range of digital cameras and targeted at users ranging from enthusiast newcomers who wish to broaden their understanding of how...

rawtherapee.com

Can you dictate anything about the database [XnView MP uses]? Like which database platform it uses? (e.g. PostgreSQL or MySQL)

Can you export said databases so that you have a backup and can restore the data should you ever need to - independent of the application.

SQLite format 3 - the file is called XnView.db, and can be easily exported, backed up, and of course tracked with your choice of automated backup solution, reducing the chance of a corrupted or lost database.

So which formats support XMP metadata?

Sorry, I should have mentioned that there are three standards.

Exif - Wikipedia

en.wikipedia.org

https://en.wikipedia.org/wiki/Extensible_Metadata_Platform

https://en.wikipedia.org/wiki/IPTC_Information_Interchange_Model

These three are all related to how (design and image software) use metadata. Not all apps support reading all of these. (too many apps out there, so please do not ask - ask ChatGPT AND do your own research per app)

Exif: JPG, PNG, Tiff, wav, webp, DCF digital camera output files

XMP can be used in several file formats such as PDF, JPEG, JPEG 2000, JPEG XR, JPEG XL, GIF, PNG, WebP, HTML, TIFF, Adobe Illustrator, PSD, MP3, MP4, Audio Video Interleave, WAV, RF64, Audio Interchange File Format, PostScript, Encapsulated PostScript, and proposed for DjVu. In a typical edited JPEG file, XMP information is typically included alongside Exif and IPTC Information Interchange Model data.

Based on limited research, one thing that I liked about DarkTable was its use of sidecar files...

...then you will like RawTherapee as well. It uses pp3 sidecar files (which are not XML based, btw).

Reads like:

[PostResizeSharpening]
Enabled=false
Contrast=15
Method=rld
Radius=0.5
Amount=200

Which is much more readable than an XML formatted file.

PhotoLine will add sidecar files in its own proprietary format - which can grow huge compared to the original output file, of course. Different usage context altogether, so comparing apples with pears here.

Don't both of those formats support adding metadata about the files?

See above. JP(e)G does. Movie file formats are different, though, and I do not have as much experience with those myself.

I asked DuckDuckGo's Search Assist for you:

Common Metadata Standards​

Standard Name	Description
Dublin Core	A widely used standard for describing a variety of resources, including videos.
MPEG-7	A standard specifically for multimedia content, providing rich descriptions.
PBCore	A metadata standard for public broadcasting, focusing on audiovisual content.
EBUCore	Developed by the European Broadcasting Union, it standardizes metadata for broadcasting.

And can't any decent application allow you to embed metadata into those files? (Or does the metadata have to get stored in a sidecar file or in an internal database?)

That again is a question that requires a specific answer for each particular software option.

Ask ChatGPT or check the documentation / manual of the software that you are interested in.

I can tell you about my software pipeline:

- PhotoLine for serious image editing
- RawTherapee for serious RAW developing
- XnView MP for image file management
- VectorStyler for serious vector illustrative work and increasingly type setting jobs
- InkScape for certain things, such as bitmap-->vector conversions
- Blender for 3D stuff, VFX, and animation
- OpenToonz for 2d animation
- Open Broadcast Studio for recording screen etc.
- Davinci Resolve (paid edition) for video editing, colour grading, sound improvements, FX
- Audacity for recording audio and simple edits
- Greenshot for grabbing screenshots
- Figma for prototyping work. Also PenPot for personal purposes
- ProMotion NG for Pixel Art
- Clip Studio for digital drawing work
- Krita for digital painting work and genAI
- InDesign for layouts
- Visual Studio Code for general coding work
- Godot for game dev
- LibreOffice for docs and spreadsheets
- Git for version control and GitLab
- Affinity for small specific jobs.
- PDF Exchange Editor for PDF edits and jobs

...and many other smaller and larger tools for very specific tasks. I might have overlooked apps in the above list.

I do have an Adobe CC subscription, because I work with external parties. My own personal work and freelance work I avoid Adobe like the plague. Not because of the rental thing (which I dislike), but because I prefer to work in other software.

o you are saying you put all of your metadata INSIDE the actual file?

And that was you are not dependent on applications?

And you are also saying that most applications can read a file's own embedded metadata?

Yes, all the important information is saved in the actual image file.

No, of course we are still dependent on applications for reading and editing this information, but we are not stuck to one specific application, and the meta data is always readable with some kind of tool. Even command-line tools and independent of the OS and platforms.

Most design applications are able to read at least part of the metadata standards embedded data. But again, not all of them can or will, and definitely only a subset will allow us to change and edit that data or amend the data with new entries.

Again, check if your own software choices do and at what level. And there may be incompatibilities in how each software reads these entries as well.

That is why you carefully consider your own tool chain, depending on your particular context and preferences.

 

[Ambrosia7177]

@Herbert123, I feel like I am in a Ph.D. program with you!!

I assume you have checked out RawTherapee as well? That is my favourite RAW developer.

Home

About RawTherapee is a powerful, cross-platform raw photo processing system, released as Free Software (GPLv3). It is designed for developing raw files from a broad range of digital cameras and targeted at users ranging from enthusiast newcomers who wish to broaden their understanding of how...

rawtherapee.com

Haven't yet, but it is on my To-Do List!

SQLite format 3 - the file is called XnView.db, and can be easily exported, backed up, and of course tracked with your choice of automated backup solution, reducing the chance of a corrupted or lost database.

Okay.

Sorry, I should have mentioned that there are three standards.

Exif - Wikipedia

en.wikipedia.org

https://en.wikipedia.org/wiki/Extensible_Metadata_Platform

https://en.wikipedia.org/wiki/IPTC_Information_Interchange_Model

There appears to be good coverage among common file formats I would use for photos (e.g. JPG, JPEG, TIFF, and AVIF) and video (e.g. MOV, MPEG). Also some support for PNG.

And it appears that DarkTable and RawTherapee support these too - although Affinity is disappointing.

...then you will like RawTherapee as well. It uses pp3 sidecar files (which are not XML based, btw).

Reads like:

[PostResizeSharpening]
Enabled=false
Contrast=15
Method=rld
Radius=0.5
Amount=200

Which is much more readable than an XML formatted file.

That is nice. What do sidecar files from DarkTable look like?


So, if I take your advice and start with adding metadata at a FILE level, then...

1.) How stable is EXIF / XMP / IPTC data in files?

How likely could it become corrupt - especially during editing of a file?


2.) How can you edit EXIF / XMP / IPTC data and not have it change the file timestamp?


3.) Presumably, editing metadata dos not impact file quality - especially with JPEG?


4.) Will applications like DarkTable and RawTherapee not only read EXIF / XMP / IPTC metadata, BUT ALSO import metadata into their internal databases and/or sidecar files so you can have the best of both worlds, including redundancy in case of file metadata failure?

I can tell you about my software pipeline:

- PhotoLine for serious image editing
- RawTherapee for serious RAW developing
- XnView MP for image file management
- VectorStyler for serious vector illustrative work and increasingly type setting jobs
- InkScape for certain things, such as bitmap-->vector conversions
- Blender for 3D stuff, VFX, and animation
- OpenToonz for 2d animation
- Open Broadcast Studio for recording screen etc.
- Davinci Resolve (paid edition) for video editing, colour grading, sound improvements, FX
- Audacity for recording audio and simple edits
- Greenshot for grabbing screenshots
- Figma for prototyping work. Also PenPot for personal purposes
- ProMotion NG for Pixel Art
- Clip Studio for digital drawing work
- Krita for digital painting work and genAI
- InDesign for layouts
- Visual Studio Code for general coding work
- Godot for game dev
- LibreOffice for docs and spreadsheets
- Git for version control and GitLab
- Affinity for small specific jobs.
- PDF Exchange Editor for PDF edits and jobs

...and many other smaller and larger tools for very specific tasks. I might have overlooked apps in the above list.

Quite a stack you have there!!

Most design applications are able to read at least part of the metadata standards embedded data. But again, not all of them can or will, and definitely only a subset will allow us to change and edit that data or amend the data with new entries.

If I added EXIF / XMP / IPTC metadata to a photo file, and then a given DAM or photo-editor did not support one of more of those, would said application in any way hurt the existing metadata? (The assumption is that it just wouldn't read it, but also wouldn't harm it.)

Again, check if your own software choices do and at what level. And there may be incompatibilities in how each software reads these entries as well.

That is why you carefully consider your own tool chain, depending on your particular context and preferences.

Yeah, all of you have introduced so many new moving parts to all of this!

I am particularly appreciative that you brought up the different types of metadata, PLUS the suggestion of embedding metadata at a FILE-LEVEL first.

That seems like solid advice.

Look forward to your responses.

Thanks!!

 

[r.harris1]

If I may, there are not a lot of moving parts in my assessment. But I allow that I can be wrong . Your primary camera here is an iPhone? Or do you have multiple cameras?

 

[Ambrosia7177]

If I may, there are not a lot of moving parts in my assessment. But I allow that I can be wrong

You lost me, @r.harris1 Did I say there were a lot of moving parts - to your comment?

In your last post to me - unless I missed one - you suggested that I just start trying out different applications.

I agreed with your assessment. (But I also said that my primary focus is completing my hard-drive project before EOY.)

Your primary camera here is an iPhone? Or do you have multiple cameras?

Yeah, I say "camera" colloquially because that is how I think of my two iPhones...

No, my dSLR died about 8 years ago, and that is how I ended up getting into iPhones and MOJO (i.e. Mobile Journalism).

That being said, I probably have a lot less need for a fancy video-editing or DAM application right now.

Then again, if my business takes off, I would like to buy a nice mirrorless camera in the next year or so - or possibly a true video-camera, so at that point I will need a more robust application.

(See, this is why I like to PLAN so much! Because I hate to have to do things over. So I am trying to think a couple steps ahead, and look for a solution that would also work if I get a real camera, and maybe even start to shoot "raw".)

Sounds like you are telling me to get to work and stop thinking so much?

 

[Herbert123]

So, if I take your advice and start with adding metadata at a FILE level, then...

1.) How stable is EXIF / XMP / IPTC data in files?

How likely could it become corrupt - especially during editing of a file?


2.) How can you edit EXIF / XMP / IPTC data and not have it change the file timestamp?


3.) Presumably, editing metadata dos not impact file quality - especially with JPEG?

[1] The meta data is part of the file's data structure, so if the file itself corrupts, the risk is high that the metadata is also affected ;-)

Editing cannot corrupt the meta data alone. If corruption occurs, this may or may not affect the entire file's contents or just part(s): some of the data may still be recoverable. Anyway, the metadata is not more 'sensitive' to corruption than the file contents itself.

[2] Ah, that is a good question. It depends. Date and time information can be stored in various ways using the meta data. Creation date is generally left alone, unless we specifically alter the value.

In this example I added my own custom 2024 date, but notice how the design software embeds the 'true' dates as well. Also note the creator tool entry. Adobe software tends to add a LOT of extra metadata, btw.

Metadata may have different standards, but that doesn't mean that developers can't invent their own 'standards'. So in the end we all have to agree how to implement these metadata standards. It's not a magic solution.

Do not confuse the timestamp date kept track of in an operating system with the date(s) embedded in a file's metadata! We can wipe ALL metadata from a file easily - most web export file dialogs have that option to reduce the exported file size a tad (or a lot depending on the software used to generate the image - looking at you, Photoshop!).

When the metadata is wiped, how can an OS still know when it was created and modified?
Answer: these dates (timestamps) are stored and tracked of by the Operating System's File System.

Which brings us round-about back to your preference to manage your images via the file system: their dates are kept in an external database by the OS in the first place, i.e. the File System. You mentioned you dislike having to depend on a database to keep track of image info? Well, you depend on the OS now

Thus, the only method to ensure that metadata of a file is maintained across different file systems and software, is via embedded metadata. Even if one piece of particular software can't read that metadata, it should in theory not wipe it. In theory. Always test before committing to any software pipeline.

Sidecar files are prone to dislodge themselves (read: users forget to copy or archive those alongside their original files), so a tad fragile. Or could be corrupted, while the image remains intact.

Do remember that the allocated max size of metadata in a file is limited. JPG's EXIF is 64kb? Don't quote me on that. Do your own research. Photoshop is known to blow up file sizes due to embedding much larger metadata info in its files.

[3] No, of course not. Only file size is affected.

 

[Ambrosia7177]

[1] The meta data is part of the file's data structure, so if the file itself corrupts, the risk is high that the metadata is also affected ;-)

Editing cannot corrupt the meta data alone. If corruption occurs, this may or may not affect the entire file's contents or just part(s): some of the data may still be recoverable. Anyway, the metadata is not more 'sensitive' to corruption than the file contents itself.

I asked, because it seems to me that back in the 1990's I had a bunch of photos (i.e. thousands) which had their metadata become corrupt, and/or maybe it was the file timestamps. (Nikon.)

They all had dates like 0/0/1900 or 1/1/1900 or something nonsensical.

Just want to understand what I'd be getting into if I decide to spend a whole bunch of time adding metadata to my existing iPhone photos.

[2] Ah, that is a good question. It depends. Date and time information can be stored in various ways using the meta data. Creation date is generally left alone, unless we specifically alter the value.

In this example I added my own custom 2024 date, but notice how the design software embeds the 'true' dates as well. Also note the creator tool entry. Adobe software tends to add a LOT of extra metadata, btw.

Metadata may have different standards, but that doesn't mean that developers can't invent their own 'standards'. So in the end we all have to agree how to implement these metadata standards. It's not a magic solution.

Do not confuse the timestamp date kept track of in an operating system with the date(s) embedded in a file's metadata! We can wipe ALL metadata from a file easily - most web export file dialogs have that option to reduce the exported file size a tad (or a lot depending on the software used to generate the image - looking at you, Photoshop!).

When the metadata is wiped, how can an OS still know when it was created and modified?
Answer: these dates (timestamps) are stored and tracked of by the Operating System's File System.

Okay, so a photo's metadata (e.g. EXIF, MXP, IPTC) is stored in its own separate place within a photo file, and not in the same place as the file "created_on" and "modified_on" fields??

Which brings us round-about back to your preference to manage your images via the file system: their dates are kept in an external database by the OS in the first place, i.e. the File System. You mentioned you dislike having to depend on a database to keep track of image info? Well, you depend on the OS now

Valid point, although I feel that I can trust macOS much more than some secondary metadata fields in a file.

What really worries me is that some APPLICATION (e.g. Photoshop, Affinity, DarkTable) would corrupt the metadata more than the OS.

For example, what is to say that I create extensive photo metadata (e.g. EXIF, MXP, IPTC) in say DarkTable, and all is well, but then I have to open up and work on the photo in say Affinity, and that 2nd application corrupts all of the metadata?!

Again, I trust macOS, but 3rd party applications - not so much so!

(It seems like different application developers implement the various metadata fields quite differently.)

Thus, the only method to ensure that metadata of a file is maintained across different file systems and software, is via embedded metadata. Even if one piece of particular software can't read that metadata, it should in theory not wipe it. In theory. Always test before committing to any software pipeline.

True.

Sidecar files are prone to dislodge themselves (read: users forget to copy or archive those alongside their original files), so a tad fragile. Or could be corrupted, while the image remains intact.

I'm not so worried about that, because I am used to working with my own homegrown "sidecar" files. For instance, when I want to download video from YouTube, I first PDF the YouTube video page - with all of its "metadata" - and then I down the actual video, and then I synch up the file names like...

"Review of Affinity Studio in 2025.pdf"
"Review of Affinity Studio in 2025.mp4"

Do remember that the allocated max size of metadata in a file is limited. JPG's EXIF is 64kb? Don't quote me on that. Do your own research. Photoshop is known to blow up file sizes due to embedding much larger metadata info in its files.

For EXIF data, does that get written over when you make changes, or does it accumulate? (Since EXIF data seems finite, I assume that 64kb isn't an issue - unless it keeps all of the history?)

What about for XMP? (Thought I read that XMP data does keep a history...)

What about for IPTC?

[3] No, of course not. Only file size is affected.

So apparently when you create or modify a photo's metadata, the OS is smart enough to not re-save the entire file?? (Maybe similar to - is it TOUCH (?) - in Linux/Unix where you can modify a file but NOT change the "modified_on" date?)

I was afraid that every time I added or modified metadata, it would recompile JPEG files and then you'd have pixelated photos.

 

[r.harris1]

You lost me, @r.harris1 Did I say there were a lot of moving parts - to your comment?

In your last post to me - unless I missed one - you suggested that I just start trying out different applications.

I agreed with your assessment. (But I also said that my primary focus is completing my hard-drive project before EOY.)




Yeah, I say "camera" colloquially because that is how I think of my two iPhones...

No, my dSLR died about 8 years ago, and that is how I ended up getting into iPhones and MOJO (i.e. Mobile Journalism).

That being said, I probably have a lot less need for a fancy video-editing or DAM application right now.

Then again, if my business takes off, I would like to buy a nice mirrorless camera in the next year or so - or possibly a true video-camera, so at that point I will need a more robust application.

(See, this is why I like to PLAN so much! Because I hate to have to do things over. So I am trying to think a couple steps ahead, and look for a solution that would also work if I get a real camera, and maybe even start to shoot "raw".)

Sounds like you are telling me to get to work and stop thinking so much?

Sorry for the confusion regarding "too many moving parts". I don't have the patience to quote each line of a long post line-by-line . It was in reference to some response you made to someone else. You have a camera or cameras (iPhone, etc), you collect images, you export them as jpegs or videos, you post them. You meed tp be able to find them at will. It's a well-trodden path and well-known processes. And if you're just storing those jpegs and movies on your hard drive(s) and using your hard drive layout as your digital asset management system (lots of people do this), it gives you a ton of flexibility in what program you use. You just need something that can work with jpegs and whatever video format you use and you can "start over" as often as you wish.

All the theory and research in the world can't beat taking a picture with your iphone and going through your flow. A simple flow would be to use Apple Photos for everything. It's free, it's on the iPhone and Mac and you already have it, and you can export what you need. Simple. But affinity can edit jpegs or any of the other tools you've been talking about. daVinci is great for video. Easy-peasy.

And yes, "get to work and stop thinking so much"

 

[Ambrosia7177]

Sorry for the confusion regarding "too many moving parts". I don't have the patience to quote each line of a long post line-by-line

If you hit "Reply" and then copy the header...

[QUOTE="r.harris1, post: 34342068, member: 670793"]

...and then paste it with each new point - and of course close with...

[/quote]

...it is super quick and makes the conversation flow better. ;-)

It was in reference to some response you made to someone else. You have a camera or cameras (iPhone, etc), you collect images, you export them as jpegs or videos, you post them. You need to be able to find them at will. It's a well-trodden path and well-known processes.

Not a well-know process to me... ;-)

And if you're just storing those jpegs and movies on your hard drive(s) and using your hard drive layout as your digital asset management system (lots of people do this), it gives you a ton of flexibility in what program you use. You just need something that can work with jpegs and whatever video format you use and you can "start over" as often as you wish.

I am thinking that sticking with my current formula of primarily organizing things by physical fodlers makes the most sense - along with verbose file names.

But I think @Herbert123 advice of putting metadata inside each file is also a great addition.

And as far as which DAM to use, that is where your advice comes in, and I'll just have to try various applications out.

All the theory and research in the world can't beat taking a picture with your iphone and going through your flow. A simple flow would be to use Apple Photos for everything. It's free, it's on the iPhone and Mac and you already have it, and you can export what you need. Simple.

Not sure Apple Photos is sophisticated enough, but will look into it.

But affinity can edit jpegs or any of the other tools you've been talking about.

Yeah.

daVinci is great for video.

DaVinci Resolve is a dream come true. That is for sure!

Easy-peasy.

Nothing is "easy-peasy" - that is why I spend so much time here asking for advice.

And yes, "get to work and stop thinking so much"

I just logged four 16 hour days, so I think I'm working enough... ;-)

 

[Herbert123]

I asked, because it seems to me that back in the 1990's I had a bunch of photos (i.e. thousands) which had their metadata become corrupt, and/or maybe it was the file timestamps. (Nikon.)

They all had dates like 0/0/1900 or 1/1/1900 or something nonsensical.

Just want to understand what I'd be getting into if I decide to spend a whole bunch of time adding metadata to my existing iPhone photos.

That MUST have been due to a file system issue on the OS level. Metadata embedded in thousands of images does NOT suddenly get 'reset' without reason.

I asked an AI:

File timestamps can reset to 1900 due to limitations in certain file systems or software that use a 32-bit representation for time, which may not handle dates beyond a specific range correctly. This issue can occur in older systems or applications that do not properly account for modern date formats.

You mention back in the 1990s - that definitely sounds like a file system level OS problem to me.

By the way, EXIF's initial release was in 1995, so I doubt if any of your images actually had any EXIF metadata embedded in them.

Okay, so a photo's metadata (e.g. EXIF, MXP, IPTC) is stored in its own separate place within a photo file, and not in the same place as the file "created_on" and "modified_on" fields??

Read up on a jpg's metadata header structure below.

Short answer: it is stored in the Exif metadata. But the date can be changed - XnView MP allows us to change these dates, as does ExifTool and other software.

Exif - Wikipedia

en.wikipedia.org

Valid point, although I feel that I can trust macOS much more than some secondary metadata fields in a file.

What really worries me is that some APPLICATION (e.g. Photoshop, Affinity, DarkTable) would corrupt the metadata more than the OS.

For example, what is to say that I create extensive photo metadata (e.g. EXIF, MXP, IPTC) in say DarkTable, and all is well, but then I have to open up and work on the photo in say Affinity, and that 2nd application corrupts all of the metadata?!

If anything most apps leave existing metadata alone, but at times will amend it with their own info. It depends on the application.

I have NEVER encountered any design app that corrupts existing metadata. Only that (if the user chooses that option) metadata is erased.

Again, if the metadata is corrupted the entire file will be affected as well, i.e. the file itself is corrupted.

Applications can change, amend or create new entries in a file's metadata.

The other thing to understand is that when we convert a JPG to a PNG or a TIFF or the other way around, the metadata is maintained, unless the app doesn't support metadata (most do, btw). This is a GOOD thing, and another boon when using metadata to organize images. When we convert to another file format --as long as that file format supports metadata-- the metadata of the original is preserved.

(It seems like different application developers implement the various metadata fields quite differently.)

The metadata standards are 'set in stone'. It's more a matter of "what is exactly supported" and how an application makes use of metadata in an image.

That is why you should test and compare software yourself, and decide what your needs are.

(No: I am not going to figure that out for you - that is your job!

XnView MP and PhotoLine work quite nicely together, for example. When I set categories, rank images, and so on in XnView, PhotoLine's mini-browser and Browser pick up on those as well. And the OS is also able to use these metadata fields immediately.

Which is one more advantage of using metadata in images: those are identified and usable by the OS and software. Relying on a DAM database or sidecar files is generally limited to 1 specific app only. With metadata that is not the case, even if some software reads only part of the metadata (or even none - it is still 'there' in the files).

For EXIF data, does that get written over when you make changes, or does it accumulate? (Since EXIF data seems finite, I assume that 64kb isn't an issue - unless it keeps all of the history?)

What about for XMP? (Thought I read that XMP data does keep a history...)

What about for IPTC?

I should amend this answer: it will get a tad technical now. Well, you asked.

First, there are other metadata standards, e.g. GPS, Camera Data, Photoshop, XMPICOM, Raw Data, and others, aside from EXIF, IPTC and XMP.

Photoshop actually accumulates a log of edits made to a file. Not really used by anything else than Adobe.

GPS - well, guess what that does. DICOM is used for medical data, i.e.: patient ID, etc. Origin when the original file was created. Camera Data is added by the camera and includes info about camera settings, etc.. Basic contains the most important metadata: author, copyright status and info, etc.

Next, read this:

The Metadata in JPEG files - Exiv2

Redmine

dev.exiv2.org

Done? Great.

So we now know that:

- metadata is saved in segments, delimited by markers which have a 16bit length (65,536 instead of the 64kb I mentioned earlier to be more precise).
- APP markers are used by applications. There is no theoretical maximum to these and certain APP markers can be used multiple times. The EXIF header and XMP blocks are used both by the APP1 marker, for example.
- multiple comment blocks can be created.

Also:

In practice the one that is much more common to result in a large header is specifically the APP2 marker being used to store the ICC color profile for the image. Since some complicated color profiles can be several megabytes, it will actually get split into many APP2 blocks (since each APP block one has a 16bit addressing limit).
reference: https://stackoverflow.com/questions/3248946/what-is-the-maximum-size-of-jpeg-metadata

So, in theory the sky is the limits with metadata in a file. It again depends on the software how much data and of what kind is stored in the metadata. A colour profile might increase a 10kb JPG file into a 3MB behemoth. And we also know that information can accumulate in a file: I have known Photoshop and Premiere files to grow a lot just because of their metadata.

In theory an image editor could save different revisions or even an entire history of edits in the metadata.

So apparently when you create or modify a photo's metadata, the OS is smart enough to not re-save the entire file?? (Maybe similar to - is it TOUCH (?) - in Linux/Unix where you can modify a file but NOT change the "modified_on" date?)

I was afraid that every time I added or modified metadata, it would recompile JPEG files and then you'd have pixelated photos.

Yes, if the app is doing it right: only the metadata is changed, and the image data is left alone. PhotoLine, XnView MP, an OS's metadata view and write functionality, and tools such as Exiftool (important metadata tool to be aware of).

Photoshop is not doing it right, if I recall correctly: it wil overwrite a new encoded image over the old version. Not great with lossy formats.

The ExifTool home page is an excellent resource to learn more about all things metadata:

ExifTool by Phil Harvey

A command-line application and Perl library for reading and writing EXIF, GPS, IPTC, XMP, makernotes and other meta information in image, audio and video files. For Windows, MacOS, and Unix systems.

exiftool.org

The author, Phil Harvey, also published extensive information about the various standards. For example, EXIF tags are listed here: https://exiftool.org/TagNames/EXIF.html

 

[Ambrosia7177]

@Herbert123 , I always appreciate your super-detailed responses! (So much to learn!!)

That MUST have been due to a file system issue on the OS level. Metadata embedded in thousands of images does NOT suddenly get 'reset' without reason.

I asked an AI:

File timestamps can reset to 1900 due to limitations in certain file systems or software that use a 32-bit representation for time, which may not handle dates beyond a specific range correctly. This issue can occur in older systems or applications that do not properly account for modern date formats.

You mention back in the 1990s - that definitely sounds like a file system level OS problem to me.

By the way, EXIF's initial release was in 1995, so I doubt if any of your images actually had any EXIF metadata embedded in them.

Well, this would have happened in the late 1990's (e.g. > 1997) or in the early 2000's, so EXIF did exist - that I know.

Also, this was back during my "dark days" when I was a Windows user! *cringe* (God only knows what Windows '98 could have done to my files?!)

Also, a more likely scenario - but I don't recall with this being like 30 years ago - is that my expensive Nikon camera (or Flash card) may have screwed things up.

I just remember being pissed because I had like 2,000 file that all said 0/0/1900 (or whatever) and so I had no way of knowing when they were shot, and thus being able to re-create timelines of my photoshoots.

That is why I have been cautious about going down this metadata rabbit-hole again...

But upon kicking this suggestion around for a few days, once again @Herbert123 , you have shown your brilliance!! I think your advice fits nicely into my geeky world view, and my need for CONTROL over my data!!

And to @r.harris1 point, I will just have to try things out and see how it goes.

If anything most apps leave existing metadata alone, but at times will amend it with their own info. It depends on the application.

I have NEVER encountered any design app that corrupts existing metadata. Only that (if the user chooses that option) metadata is erased.

Again, if the metadata is corrupted the entire file will be affected as well, i.e. the file itself is corrupted.

Okay.

"Once bitten, twice shy"

The other thing to understand is that when we convert a JPG to a PNG or a TIFF or the other way around, the metadata is maintained, unless the app doesn't support metadata (most do, btw). This is a GOOD thing, and another boon when using metadata to organize images. When we convert to another file format --as long as that file format supports metadata-- the metadata of the original is preserved.

Good to know.

The metadata standards are 'set in stone'. It's more a matter of "what is exactly supported" and how an application makes use of metadata in an image.

Okay.

XnView MP and PhotoLine work quite nicely together, for example. When I set categories, rank images, and so on in XnView, PhotoLine's mini-browser and Browser pick up on those as well. And the OS is also able to use these metadata fields immediately.

Pardon if I am asking questions that you answered earlier, but this is A LOT to process!!

Do I understand correctly that XnView MP is a browser extension or works in some way in your browser?

And is it correct that it does NOT "phone home" and do anything with my photos (or photo's metadata) online?

It sounds like you use XnView MP to manage your photos metadata?


Are there any other open-source metadata editors that you'd recommend?

It sounds like Phil Harvey is the "grandfather" of metadata editors - with his ExifTool.

I would like to find a macOS GUI metadata editor - because using ExifTool would likely be too onerous for me.

How about this one? ExifToolGUI

Any others?

Which is one more advantage of using metadata in images: those are identified and usable by the OS and software. Relying on a DAM database or sidecar files is generally limited to 1 specific app only. With metadata that is not the case, even if some software reads only part of the metadata (or even none - it is still 'there' in the files).

Yeah, I like the idea of adding metadata to each photo file.

Counter to what a lot of my critics on MacRumors probably think, I like to try and keep things SIMPLE when it comes to my computer and my files. (This less moving parts something has, the less likely it is to break!)

So, in theory the sky is the limits with metadata in a file. It again depends on the software how much data and of what kind is stored in the metadata. A colour profile might increase a 10kb JPG file into a 3MB behemoth. And we also know that information can accumulate in a file: I have known Photoshop and Premiere files to grow a lot just because of their metadata. In theory an image editor could save different revisions or even an entire history of edits in the metadata.

Okay.

Yes, if the app is doing it right: only the metadata is changed, and the image data is left alone. PhotoLine, XnView MP, an OS's metadata view and write functionality, and tools such as Exiftool (important metadata tool to be aware of).

Okay.

Photoshop is not doing it right, if I recall correctly: it wil overwrite a new encoded image over the old version. Not great with lossy formats.

Glad I'm not using Adobe...

The ExifTool home page is an excellent resource to learn more about all things metadata:

ExifTool by Phil Harvey

A command-line application and Perl library for reading and writing EXIF, GPS, IPTC, XMP, makernotes and other meta information in image, audio and video files. For Windows, MacOS, and Unix systems.

exiftool.org

The author, Phil Harvey, also published extensive information about the various standards. For example, EXIF tags are listed here: https://exiftool.org/TagNames/EXIF.html

Thanks.

 

[Herbert123]

Pardon if I am asking questions that you answered earlier, but this is A LOT to process!!

Do I understand correctly that XnView MP is a browser extension or works in some way in your browser?

And is it correct that it does NOT "phone home" and do anything with my photos (or photo's metadata) online?

It sounds like you use XnView MP to manage your photos metadata?


Are there any other open-source metadata editors that you'd recommend?

It sounds like Phil Harvey is the "grandfather" of metadata editors - with his ExifTool.

I would like to find a macOS GUI metadata editor - because using ExifTool would likely be too onerous for me.

How about this one? ExifToolGUI

Any others?

XnView MP is a free-for-personal-use cross-platform DAM software. I use it on Windows and Mac.

It has extensive metadata support, and does not send any information online.

On Windows it doesn't even need to be installed - just run it from its folder.

 

[Ambrosia7177]

@Herbert123,

From above...

Are there any other open-source metadata editors that you'd recommend?

It sounds like Phil Harvey is the "grandfather" of metadata editors - with his ExifTool.

I would like to find a macOS GUI metadata editor - because using ExifTool would likely be too onerous for me.

How about this one...

ExifToolGUI

Any others?

 

[Herbert123]

@Herbert123,

From above...

Are there any other open-source metadata editors that you'd recommend?

It sounds like Phil Harvey is the "grandfather" of metadata editors - with his ExifTool.

I would like to find a macOS GUI metadata editor - because using ExifTool would likely be too onerous for me.

How about this one...

ExifToolGUI

Any others?

I mainly use XnView MP on both Mac and Win. Not open source, but has a long history.

I found this one after a quick search: https://alternativeto.net/software/metaimage/about/
Not open source, and Mac only.

There are a few GUIs for ExifTool. You found one, I have used a Windows version in the past. But again: I predominantly just use XnView MP for major metadata edits. PhotoLine's Browser also does quick tagging and so on.

You will have to decide for yourself what works best for your particular usage context.