macOS Sonoma 14.4 Bug 'Destroys Saved Versions in iCloud Drive'

[jchap]

Yeah this update broke a 200 piece orchestral project in logic. All of my Universal audio plugins went haywire. #sh*tshow

This is of interest to me, in that my sub unit (a MacBook Air M1) has Logic installed, and I produce music regularly. (My main unit is still on Ventura, which is what it came installed with, and I am not updating.)

What "broke"? How did the plugins go haywire?

Actually, I also wonder why you updated to 14.4 on your production machine. If you're doing serious music production work with a machine, the rule of thumb is never update until you get confirmation from your peers in the industry that the update is working fine and nothing breaks. Also, it's a very good idea to check the manufacturer's site for every plugin that you run on your production machine before updating, to make sure that everything's compatible... but you probably know that already.

Can you roll back?

 

[jchap]

Are you using any plugins by EastWest Sounds? I'm still on 14.2.1 because I'm working on an album and I'm afraid of breaking my plugins.

I suggest not updating until you get confirmation from the developer that the plugins work fine on the new version. Too risky. As a creative and likely someone who depends on music for their well-being or their livelihood, you need to release that album more than you need to update your OS.

 

[ignatius345]

iCloud in general is a calamity waiting to happen. Some years ago, only foresight and a dose of serendipity prevented it from literally annihilating every single video and photo I'd ever taken. Took ages to repair the damage from locally exported files. Had I not had those as a fallback, I would have lost everything forever. Since then I've disconnected from almost everything 'cloud' except keychain passwords, and I even have those exported onto an encrypted USB stick.

'The Cloud' is someone else's computer. It cannot, and should not, ever be trusted with anything. Ever. Especially with bug-infested software on the local machine.

Doesn't need to be an either/or. I happily use iCloud Drive every day. But I also make sure my primary Mac is set to download everything (no optimized storage) and I also make sure that Mac is backing up every day to a physical hard drive. Two hard drives, actually, one of which lives in a drawer at my office. So I've got the convenience of having all my files available on both my Macs and my iPhone -- and at the same time I have physical possession of everything, backed up with multiple versions for redundancy.

You mention you almost lost everything because of some glitch. If you're in a position where a glitch with any cloud service means you'd "lose everything forever" then you're leaving yourself wide open. And it's not just iCloud. Google Photos has corrupted or downsized people's images before as well.

 

[Jim Lahey]

Doesn't need to be an either/or. I happily use iCloud Drive every day. But I also make sure my primary Mac is set to download everything (no optimized storage) and I also make sure that Mac is backing up every day to a physical hard drive. Two hard drives, actually, one of which lives in a drawer at my office. So I've got the convenience of having all my files available on both my Macs and my iPhone -- and at the same time I have physical possession of everything, backed up with multiple versions for redundancy.

You mention you almost lost everything because of some glitch. If you're in a position where a glitch with any cloud service means you'd "lose everything forever" then you're leaving yourself wide open. And it's not just iCloud. Google Photos has corrupted or downsized people's images before as well.

Yes but the issue is that iCloud and its peers are sold as secure data storage. Thus the majority of laypeople treat it as such.

In reference to my photos issue, the very reason I didn’t lose everything is precisely because I had local hard copies. My iCloud library was hosed. Totally annihilated.

 

[AlmightyKang]

Photos scares the **** out of me. I’ve been on the tipping point of moving my stuff out of it for weeks. Final push I think.

 

[Jim Lahey]

Photos scares the **** out of me. I’ve been on the tipping point of moving my stuff out of it for weeks. Final push I think.

iCloud Photos is ok so long as you do not have Optimized Storage enabled and that you do have hard backups of the data on local disc/s. Personally I have disabled iCloud for almost everything but that's part of a wider shift away from cloud storage in general. Optimized Storage is really the nexus of all iCloud data loss risk. It's sold as 'safe' verbatim in the blurb, but it's a disaster waiting to happen. Of course, I'm sure Apple loves it because it allows them to keep more base storage users permanently haemorrhaging cash into monthly data plans while selling them a false sense of security.

ETA: I will expand a little on the photos backup thing - it should be said that just because you have hard copies doesn’t necessarily mean that an iCloud data problem isn’t going to cause a whole world of pain. I had to jump through a million hoops fixing my library and getting it synced back to the iCloud again. After which I later decided to abandon it completely for the sake of sanity.

 

[AlmightyKang]

iCloud Photos is ok so long as you do not have Optimized Storage enabled and that you do have hard backups of the data on local disc/s. Personally I have disabled iCloud for almost everything but that's part of a wider shift away from cloud storage in general. Optimized Storage is really the nexus of all iCloud data loss risk. It's sold as 'safe' verbatim in the blurb, but it's a disaster waiting to happen. Of course, I'm sure Apple loves it because it allows them to keep more base storage users permanently haemorrhaging cash into monthly data plans while selling them a false sense of security.

ETA: I will expand a little on the photos backup thing - it should be said that just because you have hard copies doesn’t necessarily mean that an iCloud data problem isn’t going to cause a whole world of pain. I had to jump through a million hoops fixing my library and getting it synced back to the iCloud again. After which I later decided to abandon it completely for the sake of sanity.

Well there's a bit of a problem there because it's not quite that simple. Photos.app is completely opaque which means it is very hard to validate the integrity of what is in there. As a point, it's not a backup until it is validated and the restore is tested. The originals are stored verbatim so that's less of a problem but the layers of edits and metadata changes over the top is what you risk losing and those are almost completely impossible to extract from it, even if you have the SQLite data file behind the scenes.

On your second point, I had exactly that as well and had to repair the library. Worrying.

I'm copying everything out into (don't laugh) OneDrive at the moment. It syncs immediately to my PC where I can do a hard backup to an external NTFS volume and can actually validate it against that. I don't need much other than gallery folder names which you can export the structure of with some futzing. I've dumped a hundred or so edits out flattened as well.

There are some photos that would destroy me if I lost them so I am ultra paranoid about this.

 

[ignatius345]

Well there's a bit of a problem there because it's not quite that simple. Photos.app is completely opaque which means it is very hard to validate the integrity of what is in there. As a point, it's not a backup until it is validated and the restore is tested. The originals are stored verbatim so that's less of a problem but the layers of edits and metadata changes over the top is what you risk losing and those are almost completely impossible to extract from it, even if you have the SQLite data file behind the scenes.

True. I don't do a ton of editing in Photos, but I *do* have a lot of organization that's gone into what's on there. In a pinch I could pull every original out of my backup, but as you say I'd lose all that other work I've done.

 

[Jim Lahey]

Well there's a bit of a problem there because it's not quite that simple. Photos.app is completely opaque which means it is very hard to validate the integrity of what is in there. As a point, it's not a backup until it is validated and the restore is tested. The originals are stored verbatim so that's less of a problem but the layers of edits and metadata changes over the top is what you risk losing and those are almost completely impossible to extract from it, even if you have the SQLite data file behind the scenes.

On your second point, I had exactly that as well and had to repair the library. Worrying.

I'm copying everything out into (don't laugh) OneDrive at the moment. It syncs immediately to my PC where I can do a hard backup to an external NTFS volume and can actually validate it against that. I don't need much other than gallery folder names which you can export the structure of with some futzing. I've dumped a hundred or so edits out flattened as well.

There are some photos that would destroy me if I lost them so I am ultra paranoid about this.

Yeah I say ditch it and be happy. I still use the photos app on Mac but always export out jpegs to a separate disc for disaster recovery peace of mind. That is to say I don’t just rely on my library being in Time Machine. I then sync selected albums to my iPhone via a good old fashioned cable 👍

 

[AlmightyKang]

Ok everything exported over SMB to PC which is now syncing with OneDrive overnight. 55 gig going up ...

 

[Wando64]

Well there's a bit of a problem there because it's not quite that simple. Photos.app is completely opaque which means it is very hard to validate the integrity of what is in there. As a point, it's not a backup until it is validated and the restore is tested. The originals are stored verbatim so that's less of a problem but the layers of edits and metadata changes over the top is what you risk losing and those are almost completely impossible to extract from it, even if you have the SQLite data file behind the scenes.

On your second point, I had exactly that as well and had to repair the library. Worrying.

I'm copying everything out into (don't laugh) OneDrive at the moment. It syncs immediately to my PC where I can do a hard backup to an external NTFS volume and can actually validate it against that. I don't need much other than gallery folder names which you can export the structure of with some futzing. I've dumped a hundred or so edits out flattened as well.

There are some photos that would destroy me if I lost them so I am ultra paranoid about this.

Sounds like you are ready to ditch the Mac altogether for your sanity.

I really don’t understand your level of worry.
To keep regular backups is the best thing you can do in any environment, and even then things will go wrong from time to time.
I still remember when I was still on Windows, 15 years ago, finding out that some of my photos original RAWs having become corrupted and all my backups having the corrupted version on them.
When you have thousands of files, this is always a possibility and that’s why these days, once a backup disk becomes full, I often just swap it with a new one and keep the old one as archive.

Opaque or not opaque, files get corrupted from time to time.
It happens on Macs, Windows PCs and Unix workstations, on iCloud, on OneDrive, Dropbox or whatever else.
If your data is important to you, you must have a solid backup strategy.

If you think avoiding Photos, iCloud, Reminders and Notes is going to save you from trouble, good luck to you.

 

[Jim Lahey]

From my perspective the problem with iCloud is not that something can go wrong per se. Rather, it’s that when something does go wrong, the data loss or corruption rapidly propagates across all your devices, destroying everything in its path. The very feature that offered all the convenience suddenly becomes the enemy. Yes local backups may save the day in such an event, but you’re still in a world of pain, and if you were using optimized storage, well…good luck. And therein lies the major issue - cloud services are sold as safe and secure, but in some circumstances they’re a clear and present danger.

 

[sideshowuniqueuser]

I had all of my iCloud Drive files corrupted a few years ago and since then I have never relied on iCloud. It's a complex system, but I often give it as a bad example.

Same. I stopped using iCloud entirely, years ago.

 

[AlmightyKang]

Sounds like you are ready to ditch the Mac altogether for your sanity.

I really don’t understand your level of worry.
To keep regular backups is the best thing you can do in any environment, and even then things will go wrong from time to time.
I still remember when I was still on Windows, 15 years ago, finding out that some of my photos original RAWs having become corrupted and all my backups having the corrupted version on them.
When you have thousands of files, this is always a possibility and that’s why these days, once a backup disk becomes full, I often just swap it with a new one and keep the old one as archive.

Opaque or not opaque, files get corrupted from time to time.
It happens on Macs, Windows PCs and Unix workstations, on iCloud, on OneDrive, Dropbox or whatever else.
If your data is important to you, you must have a solid backup strategy.

If you think avoiding Photos, iCloud, Reminders and Notes is going to save you from trouble, good luck to you.

This is pretty much wrong.

Corruption does not just happen. There is a reason: either hardware or software failure. You must insulate yourself against this by doing integrity checking of your backups (checksums) and testing restores and handling thing when there are integrity issues or restore failures.

This is something you can only do if your data is in tangible identifiable files and a copy of them is on your local system.

Relying on a cloud vendor and trust and assuming that the status quo is ok is not an acceptable position in my mind. That is backed up with a lot of experience past end user storage (we have 2734TB of online storage at work).

There is no luck involved.

 

[Wando64]

Corruption does not just happen. There is a reason: either hardware or software failure.

OK, hardware and software failures just happen.

That is backed up with a lot of experience past end user storage (we have 2734TB of online storage at work).

This is hilarious. I worked for HP for almost 30 years, do you want to have a guess how many TB of online storage they have?
Don’t bother, because it is irrelevant.

Like I said, you must have a solid backup strategy, and we seem to agree.

 

[Wando64]

This is something you can only do if your data is in tangible identifiable files and a copy of them is on your local system.

By the way, I forgotten to mention that this is puzzling.
I don’t think anybody’s has ever attempted to back up any intangible, unidentifiable files that are not on their local system.

 

[abatabia]

When the heck are they updating this to fix these bugs?? It’s been two weeks since I could plug anything into my Mac mini.

 

[AlmightyKang]

By the way, I forgotten to mention that this is puzzling.
I don’t think anybody’s has ever attempted to back up any intangible, unidentifiable files that are not on their local system.

Intangible -> Not everything you have access to on your own machine without turning off SIP

Unidentifiable -> opaque formats. No point in backing up something if you can't restore it or recover it. Case in point CRDT based stores on top of SQLite which is what some of the Apple apps use.

Not on local system -> some stuff is just in iCloud, not local at all.

Apple should provide open file and protocol specifications like Microsoft do to allow interoperability.

 

[Wando64]

Intangible -> Not everything you have access to on your own machine without turning off SIP

Unidentifiable -> opaque formats. No point in backing up something if you can't restore it or recover it. Case in point CRDT based stores on top of SQLite which is what some of the Apple apps use.

Not on local system -> some stuff is just in iCloud, not local at all.

Apple should provide open file and protocol specifications like Microsoft do to allow interoperability.

You can make up definitions as much as you like.

You must have a solid backup strategy, like I said to you far too many times now.
With a good backup you can restore your system and all of your files: tangible or intangible, identifiable or unidentifiable, clear or opaque, with or without “open files and protocol specifications like Microsoft“.

Finally, I don’t know why I even have to state this, but it is pretty obvious that to back up a file it needs to be on your local system. This is a basic concept that must be taken into account whatever cloud system you rely upon.

I am now done with this thread, so… whatever.

 

[Jim Lahey]

Finally, I don’t know why I even have to state this, but it is pretty obvious that to back up a file it needs to be on your local system. This is a basic concept that must be taken into account whatever cloud system you rely upon.

It’s obvious to us, yes, but not to the layperson. Is any of what you describe prominently explained by Apple when one enables Optimized Storage? I mean, it may be, but I don’t remember seeing it. The blurb implies the exact opposite.

 

[AlmightyKang]

It’s obvious to us, yes, but not to the layperson. Is any of what you describe prominently explained by Apple when one enables Optimized Storage? I mean, it may be, but I don’t remember seeing it. The blurb implies the exact opposite.

Well that is exactly my point. Your file is intangible. What you have in Photos.app is a preview, not the actual file if you have optimised storage enabled. Also what you have is an intangible file and a stack of edits stored in a metadata database (in SQLite) applied to it and a generated preview as the outcome. The nature of the problem of "where is my photo" becomes quite complicated to reason about.

So while Wando64 is clearly done with this argument, the definitions I have produced are accurate and the nuances must be clearly identified and considered for everybody. Which is my point.

You can only back up and restore things effectively when all assumptions are removed.

We had a wonderful problem with some commercial CAD software a number of years back when we found out their file format is not portable to a later version of the software. This resulted in us having to maintain 3 CAD machines and the software indefinitely while we slowly ported stuff out. So this brings in the further point that if you have an undocumented abstraction and format, it is more complicated than even just possessing tangible copies of it. You have to actually have the software which can read the format. Or preferably have an open documented standard. Dassault CATIA for reference.

This is analogous to the previous points made about older Keychain versions.

Even having the file on a volume does not guarantee that it's useful or usable.

 

[Jim Lahey]

Well that is exactly my point. Your file is intangible. What you have in Photos.app is a preview, not the actual file if you have optimised storage enabled. Also what you have is an intangible file and a stack of edits stored in a metadata database (in SQLite) applied to it and a generated preview as the outcome. The nature of the problem of "where is my photo" becomes quite complicated to reason about.

So while Wando64 is clearly done with this argument, the definitions I have produced are accurate and the nuances must be clearly identified and considered for everybody. Which is my point.

You can only back up and restore things effectively when all assumptions are removed.

We had a wonderful problem with some commercial CAD software a number of years back when we found out their file format is not portable to a later version of the software. This resulted in us having to maintain 3 CAD machines and the software indefinitely while we slowly ported stuff out. So this brings in the further point that if you have an undocumented abstraction and format, it is more complicated than even just possessing tangible copies of it. You have to actually have the software which can read the format. Or preferably have an open documented standard. Dassault CATIA for reference.

This is analogous to the previous points made about older Keychain versions.

Even having the file on a volume does not guarantee that it's useful or usable.

No argument from me. I don’t consider myself qualified to comment on the minutiae of data structures, so I’ll leave that debate to others 👍

The thrust of my position is simply that cloud services are sold as secure when they are not. In fact, since all data errors are rapidly propagated to all devices, an argument can be made that they’re actually dangerous to the average user under certain conditions. Specifically when they use optimized storage. The blurb for which even explicitly uses the word ‘safely’ 🤦‍♂️

 

[AlmightyKang]

No argument from me. I don’t consider myself qualified to comment on the minutiae of data structures, so I’ll leave that debate to others 👍

The thrust of my position is simply that cloud services are sold as secure when they are not. In fact, since all data errors are rapidly propagated to all devices, an argument can be made that they’re actually dangerous to the average user under certain conditions. Specifically when they use optimized storage. The blurb for which even explicitly uses the word ‘safely’ 🤦‍♂️

100% agree with this. It's a big problem on large scale systems as well.

Some Facebook engineers actually wrote a good paper on it https://arxiv.org/pdf/2102.11245.pdf

 

[RyanFlynn]

This is of interest to me, in that my sub unit (a MacBook Air M1) has Logic installed, and I produce music regularly. (My main unit is still on Ventura, which is what it came installed with, and I am not updating.)

What "broke"? How did the plugins go haywire?

Actually, I also wonder why you updated to 14.4 on your production machine. If you're doing serious music production work with a machine, the rule of thumb is never update until you get confirmation from your peers in the industry that the update is working fine and nothing breaks. Also, it's a very good idea to check the manufacturer's site for every plugin that you run on your production machine before updating, to make sure that everything's compatible... but you probably know that already.

Can you roll back?

Sorry about the delayed response. It was almost all of my Universal audio plugins. I'm running 2x x8ps and an x16 in my studio and rely heavily on their stuff for mixing. Looks like there is a fix now.

 

[jchap]

Sorry about the delayed response. It was almost all of my Universal audio plugins. I'm running 2x x8ps and an x16 in my studio and rely heavily on their stuff for mixing. Looks like there is a fix now.

Hope the fix works for you. Those updates will surely sap your productivity… I’m not so trusting of Apple, when it comes to my main production machine.

When it comes to new features and new OS releases, I adopt the motto: “Do NOT be curious! You are NOT interested in this!” Anything that might be disruptive to a creative workflow should be handled with extreme caution, I think.

This includes hanging out on MacRumors, haha… time to call it a day!