Warning!! Serious Apple Photos Bug

[dreamweazel]

Warning!!! Don't let Photos update your library especially if you have entrusted it with lots photos! I had 65,000 photos, now I have 58,000 after an "automatic recovery" of my files.

Photos has a serious bug that trashes items once the 64,000 file limit has been reached (the new Photos database structure using sqlite). Any photos added or updated after that limit will/may be lost. The file structure (found in "Show Package Contents") has been changed to a system that renames all of your original files to some ridiculously encoded name and places them back in folders labeled in hexadecimal numerics from 1 to F. Once each hex folder is filled with up to approximately 4000 files, that's it, no further files are written. Starting with older files, to newer files, there is no "room" for newer files to be saved.

The result is that, viewing in the Photos app, all newer photos from a particular date are lost, and yes they are lost. Forever, unless you have an untouched Photos Library backup. This means that you can still recover any new, un-renamed files from an older "originals" backup folder which is not populated with file names like 4C48659C-FAF5-4117-B7DE-CBEO... .jpg. Actual, original files should look like IMG5634.jpg. Don't open your backup files in Photos, or you run the same risk. Recover files from the Finder, and make several separate Photos Libraries of say 25,000 images each until this issue is addressed by Apple.

I tried to recover files from the new "originals" folder only to find that anything after November 2022 was lost. Photos automatically deleted files because the file structure for the database was exceeded and couldn't be written to disk. (It took 2 days and 23 hours to re-import my Library into a new Photos library that resulted in the same loss of files).

Then I discovered the horrid Apple's Photos database design...and wrote this note. Hope this helps.

 

[Coltaine]

This is a serious problem!
Thanks for the write-up.
With about 22.000 photos and 1400 videos, I am still well below that threshold.
But it is good to know about it.
Did you file a bug report to Apple, this problem should be addressed asap!

 

[dreamweazel]

This is a serious problem!
Thanks for the write-up.
With about 22.000 photos and 1400 videos, I am still well below that threshold.
But it is good to know about it.
Did you file a bug report to Apple, this problem should be addressed asap!

Yes, I did

 

[jagolden]

I may be a dinosaur here, but after I import new photos into the Photo app, I copy them over to an external drive and have two separate back-up drives.

 

[dreamweazel]

I may be a dinosaur here, but after I import new photos into the Photo app, I copy them over to an external drive and have two separate back-up drives.

Excellent, I wish I had done the same. I never expected I'd make that mistake - to trust technology.

 

[gilby101]

Photos has a serious bug that trashes items once the 64,000 file limit

Are you sure that there is this limit. A web search about limits, I found a) possibly no limit or 1,000,000, b) people quoting libraries of 80,000 (and one with 6.8TB). I can't find a specified limit.
Asking because it is possible you have a different problem.

 

[HDFan]

A web search about limits,

Same results. If there were a hard limit it likely would have been reported years ago. Suspect something else is happening.

 

[dreamweazel]

Same results. If there were a hard limit it likely would have been reported years ago. Suspect something else is happening.

Are you sure that there is this limit. A web search about limits, I found a) possibly no limit or 1,000,000, b) people quoting libraries of 80,000 (and one with 6.8TB). I can't find a specified limit.
Asking because it is possible you have a different problem.

Yes, it could be something else, but I found other reports of people losing files after seemingly random dates. I find it odd that my library is close to 256^2 files with each of the "original" folders contain roughly 64^2 files and no files after Nov of 2022. That's why I thought I hit a "trip wire." This library has been used and updated since iPhoto first came out. I have had trouble with no thumbnails, and files that would not export before. Maybe that was a sign of something about to go wrong. But to auto trash all of the files since Nov 2022, that is really weird. I do have a files backup from 2020 to Aug of 2023, so my "photographic memory" of family events will have a lapse of about six months.

 

[HDFan]

Certainly could see random photos disappearing for reasons unknown.

If you care about your photos you might want to implement a 3-2-1 backup strategy. One point of failure is a recipe for disaster as I guess you unfortunately know.

 

[JonaM]

I have more than 64,000 files and haven't experienced the loss of any photos ( that I'm aware of).
What was the update that you mentioned, as I'm not aware of any significant updates to the Photos app in recent history, so would be useful to understand more about what the update was?

 

[headlessmike]

I have more than 125,000 photos and a few thousand videos and have never had any issues with Photos. I've never allowed Photos to copy my files into its internal library though. Everything is stored externally and the app just accesses them wherever they are.

 

[AlmightyKang]

I think this is a correlation not a causal situation. I've found when debugging things that assumptions are best not made about cause until the problem is properly understood. I have SQLite databases, the underlying store used for Photos, with 500 gig of just row data in a single table in them...

As for the metadata structure on disk, the folders are prefixed /X/Y/{X}{Y}Z.extension ... this is because it's cheaper to do two directory entry lookups then the file name due to how the filesystem data structures operate. These are 128 bit unique UUIDs which do not overlap. The database contains a mapping back to the original filename.

I think you have some other problem here. I'm not saying it's not a problem but the nature of it might be different to what is assumed here.

 

[RhetTbull]

As others have said, this has nothing to do with a 64,000 file limit. I've worked with Photos libraries that are over 2TB in size with 100s of thousands assets. Something in your library was corrupted.

> The file structure (found in "Show Package Contents") has been changed to a system that renames all of your original files to some ridiculously encoded name and places them back in folders labeled in hexadecimal numerics from 1 to F. Once each hex folder is filled with up to approximately 4000 files, that's it, no further files are written.

As AlmightyKang said, This is done for efficiency in storing/retrieving files as having too many files in a single folder makes access slower. But there is no limit to this scheme.

You mentioned "don't let Photos update your library" -- what did you do to get into this state? Did you upgrade from an older version of MacOS? Do you have a backup of the original library?

If you do not have a backup, then I suggest you try the repair library feature to see if Photos can repair the library. If you do have a backup, then I suggest creating a copy of the backup (so as not to corrupt the backup) and try to open the copy in the Photos.

I am the author of a free open source command line tool that may be able to recover the photos. See osxphotos for more info. After installing you could try: "osxphotos export /path/to/export --export-by-date" and see if it can export the "missing" photos. You can also try "osxphotos query --count" which will give you a total count of assets in the library -- this might be hint as to whether the assets are really still there. Finally, you can run "osxphotos orphans" which will tell you if it finds orphan files (that is, the files are still in the library but they've been deleted from the Photos database). You could then recover these orphan files. This is a command line tool that requires some comfort working in the Terminal so if you don't have experience doing this then maybe find a friend who is to help out.

Good luck!

 

[dreamweazel]

Ok. So far I have done the following:

- made a files backup, by using the Export function of Photos, from the files that I had, to a freshly formatted APFS drive
- files were exported with normal names and were stored in folders by creation date.

- I copied all of these files again to another APFS drive using Carbon Copy Cloner 6

- formatted an external 2TB WD Green drive as APFS

- named the volume "Main Photos Library"
- opened Photos while holding down the Option key
- created a new Photos Library in the new volume and named it "Photos Library"
- set it to be the System Photo Library, here are the other settings
- I don't use iCloud for photos

- I spent a couple of days importing photos from the affected Photos Library, and other "Photos Library" backups, into the new Library
- the new Photos Library now contains 68223 Photos, 1960 Videos, and everything but newer 2023 photos is there. Works well so far.

>>> So I guess this throws out the 65536 file limit that I feared was the problem. Thanks for sharing your own library sizes. Same thing with my SQLite finger pointing. And, thanks for the SQLite explanation AlmightyKang.

- the new Photos Library is somewhat slow to load, alternating between spinner and beach ball. I assume the launch will speed up as the file is used, and caches are created
- no duplicates, nothing in the deleted album
- closed the Photos application and began to make a backup

- The Finder window says this new Photos Library is a tiny 2.9MB in size, but all photos and videos are contained within the library and none are linked. 2.9MB. Something fishy here.

- Get Info window for the drive has the correct information, the Library is using about 703 GB.

- now trying to do a Finder level copy (drag and drop) of this new library to an external 8TB Seagate APFS volume.
- it has been preparing to copy for over 4 hours now.

- the Sonoma 14 problem with external drives seems to be a mounting issue, not a file issue, but a 2.9MB Photos Library with over 60000 photos seems odd?

So that's where I am at right now. I have backups and have a lot more understanding of what I should do.

Thank you everyone for all your comments and suggestions. This is a wonderful community! I especially appreciate everyone taking time to respond and your helpful manner to help this "newbie" from freaking out.

 

[HDFan]

the Sonoma 14 problem with external drives seems to be a mounting issue, not a file issue, but a 2.9MB Photos Library with over 60000 photos seems odd?

That could be it. Copying photos in a library can be slow for some reason since it isn't a simple file lookup. Aperture library backups are much slower than simple folder backups for example.