Upgrading to MacOS Ventura erased all Notes ($100k+ damages)

[Mike Boreham]

Since the above is a tested proven way of restoring Notes from a TM Backup, and doesn't seem to be documented anywhere, and is not like restoring other types of data, I thought it worth giving the short version:

1. Turn off iCloud Notes in Settings > Name > iCloud.
2. Restore the folder ~/Library/Group Containers/group.com.apple.notes
3. Open Notes app
4. Copy all restored iCloud notes to On-My-Mac (On-my-Mac may need enabling in Notes > Settings)
5. Turn iCloud Notes back on.
6. Move the restored notes out of On-My-Mac back to iCloud.

If anyone sees this or another method documented anywhere please post.

Another way of backing up and restoring iCloud Notes is with a third party app like iMazing, which can extract Notes from backups separately.

iMazing backups are the same as and compatible with the iDevice backups made by Finder, but it is not possible to extract particular data types from a Finder backup without a third party app such as iMazing.

 

[Queen6]

Where is the blood?

No blood, Apple just decided the imagery wasn't for them late in the day and swapped out the wallpaper. Personally I prefer the OG wallpaper from the disks, it's simply more authentic...

Q-6

 

[roar08]

This is probably the 10th OS update I'm going through and the most damaging one. Following the upgrade all apps seem to have re-opened, as expected, with the exception of Notes. On a close inspection, 1.5 years worth of notes containing both client and personal sensitive data, notes, company, healthcare, interview notes, research, encrypted / unique keys and credentials were removed. Overall, probably in the range of $100k++ worth of losses. The Notes were saved locally, and not through iCloud. They were however displayed under the iCloud 'Notes' section. I tried all methods to restore the data but nothing worked so far. I've ordered an SSD for Time Machine. What are my options?

This would be a clever way to slander a company: post a thread like this then point to it to slam your competition and/or promote your own products/services. Certainly not saying that's what happened here. But it would be clever.

If it is true, it's profoundly unprofessional (trying really hard not to say it's stupid).

 

[NoBoMac]

Add to the 3rd party apps/programs, can use Shortcuts to backup Notes. That's how I'm doing it. Can be something as simple as attached (my solution is a bit more involved: saving timestamp of last save, Automations running every two hours, saving notes individually in specific folders [depending on folder Note is in in Notes app]).

 

[Mike Boreham]

Add to the 3rd party apps/programs, can use Shortcuts to backup Notes. That's how I'm doing it. Can be something as simple as attached (my solution is a bit more involved: saving timestamp of last save, Automations running every two hours, saving notes individually in specific folders [depending on folder Note is in in Notes app]).

This is brilliant! Never used Shortcuts before but within 10 mins and using your screenshot and a few missteps I have a one button click to create a zip archive of all Notes. I didn't bother with the "is after last modified date" ...just makes a fresh archive of all my notes near instantly.
Thanks very much.

 

[MBAir2010]

OP isn't engaging, so just another dead thread...

nah,
that $100k price tag on the headline will keep this active for quite some more time.

 

[Queen6]

nah,
that $100k price tag on the headline will keep this active for quite some more time.

True, my data has a far higher price tag, equally It's backed up across multiple platforms, encrypted Cloud server's and external drives. It is what it is, just another hit and run...

Q-6

 

[mrmister]

“Or why on Earth would people store critical and vital information on Notes…”

A lot of comments like this, and it’s dumb.



Notes is great. I think it’s arguably much better than using an obscure or antiquated app.

The real issue is that you must have robust backup strategies.

 

[lindros2]

This is why I use OneNote.
The $100k price tag boggles me.

 

[Queen6]

“Or why on Earth would people store critical and vital information on Notes…”

A lot of comments like this, and it’s dumb.



Notes is great. I think it’s arguably much better than using an obscure or antiquated app.

The real issue is that you must have robust backup strategies.

Likely as was just convenient, the error was to not backup. Personally I'd defer to a more structured application and run it through my end to end encrypted cloud server. My solution is automated so I don't worry, just check from time to time the backups are processing...

Q-6

 

[Queen6]

This is why I use OneNote.
The $100k price tag boggles me.

TBH it's meaningless in the wider scope. If I sent you my documents they would be of little value unless your in the same field of work.

That said some documents do have a significant commercial cost. Just a fraction of the libary.
..

$100K wont touch this, $250K maybe...

Q-6

 

[BlueGhost]

Wondering if the OP was referring to a non-Apple note taking application? Looks like it is local, but I've seen some "cute" uses of aliases that can be confusing on the surface.

Or perhaps the Lotus Notes application, which has its own database and is a fairly complex business application? I believe the newer versions are called Domino. An auditor tried to get us to install this at a former employer in the 1990s. This is a client-server application that includes email, calendars, to-do-lists, contact management, discussion forums, file sharing, websites, instant messaging, blogs, document libraries, user directories, and custom applications.

https://en.wikipedia.org/wiki/HCL_Domino

If it is Notes/Domino, he may need someone familiar with that application (possibly from Robert Half, Manpower or Indeed). Depending on the implementation, data may be stored on a Windows server, Unix server, Minicomputer or Mainframe (sometimes a combination of these). I think it could also be stored on multiple cloud servers, but I've never worked with or administered the application. Could be simple, maybe just installing and configuring a compatible client on the new desktop/laptop (if no data is stored on the client).

If it is a simpler, non-Apple note taking application, the OP can probably contact the author of the application. There are several non-Apple note taking and business productivity applications. I would not expect the Apple migration software to handle these. The author can likely supply a multi-step process to move from one machine to another.

As others have recommended, I'd talk with Apple support first. I think they also have small and medium business folks at some locations.

 

[dsemf]

This is brilliant! Never used Shortcuts before but within 10 mins and using your screenshot and a few missteps I have a one button click to create a zip archive of all Notes. I didn't bother with the "is after last modified date" ...just makes a fresh archive of all my notes near instantly.
Thanks very much.

I must be doing something wrong. I only get 26 notes (out of 136) in the zip file.

DS

 

[Odessa]

I had a friend that also has all their notes disappear from their macbook air after an os update like 10 ago or so.
Since then, I just assumme that If I use apple stock apps, I have way to rely over the data safety in it unless I use icloud and/or time machine (even then I worry the day I need time machine It won't work for X reason because encryption or whatever)

 

[Queen6]

I must be doing something wrong. I only get 26 notes (out of 136) in the zip file.

DS

I use an end to end encrypted cloud solution, so backup is pretty much automated & seamless. Anything new in the specified folders is automatically sent to the servers and will be downloaded to the other systems once online.

The only downside is if you forget the password your data is done as the provider makes it very clear they are not in position to recover it without the recovery key...

Q-6

 

[Mike Boreham]

This is brilliant! Never used Shortcuts before but within 10 mins and using your screenshot and a few missteps I have a one button click to create a zip archive of all Notes. I didn't bother with the "is after last modified date" ...just makes a fresh archive of all my notes near instantly.
Thanks very much.

@NoBoMac Looks like my enthusiasm last night may have been premature

Two problems:
1. The archive I created last night has not included attachments.
2. This morning the Shortcut fails with message "Couldn't communicate with a helper application". If Notes app was open when I launched the shortcut it then crashes after the message. I have not changed anything between last nights success and today.

Any thoughts? As I said I think I did a simplified version of your shortcut. This is what mine looks like:

 

[rm5]

Yet video editing and music production industry (by extension maybe the entire Hollywood) still prefer Mac over Windows any day.

Why? Because it is a much more stable platform (in my experience). In the audio business, every time you connect an interface on Windows, you have to install some driver for it, which is time-consuming and creates more headaches if the driver somehow fails. On Mac, you can literally just plug in an audio interface and it works (in most cases). I would much prefer a more stable platform (i.e. a Mac), than to have my productions crash every 30 seconds.

 

[Mike Boreham]

@NoBoMac Looks like my enthusiasm last night may have been premature

Two problems:
1. The archive I created last night has not included attachments.
2. This morning the Shortcut fails with message "Couldn't communicate with a helper application". If Notes app was open when I launched the shortcut it then crashes after the message. I have not changed anything between last nights success and today.

Any thoughts? As I said I think I did a simplified version of your shortcut. This is what mine looks like:

View attachment 2140286

Just tried again this afternoon and it worked!? Still no attachments though.

 

[NoBoMac]

“Communication Helper” no idea real reason why, but basically Shortcuts app/subsystem glitching. Seen it in the past and swipe-quit of Shortcuts clears things up. Have not seen that issue in quite a while now (iOS 16 launch?).

But attachments, yeah, forgot to mention that. And encrypted Notes will be blank (though might backup if unlocked, can’t recall). Forgot as I only have like two Notes with attachments and those are backups for other sources/apps storing that same data (read: not really important for my use case).

 

[Mike Boreham]

“Communication Helper” no idea real reason why, but basically Shortcuts app/subsystem glitching. Seen it in the past and swipe-quit of Shortcuts clears things up. Have not seen that issue in quite a while now (iOS 16 launch?).

But attachments, yeah, forgot to mention that. And encrypted Notes will be blank (though might backup if unlocked, can’t recall). Forgot as I only have like two Notes with attachments and those are backups for other sources/apps storing that same data (read: not really important for my use case).

Thanks. I have been doing it in Ventura not iOS/ipadOS 16. Will try on iPad.

Attachments are not too important to me either, because not many of my notes have them and in those that do, the attachments exist elsewhere.

 

[HDFan]

Add to the 3rd party apps/programs, can use Shortcuts to backup Notes.

Since you use Time Machine is there a reason you use a manual solution?

With Monterey the google TM restore instructions I found aren't valid. They say restore

NotesV7.storedata-wal
NotesV7.storedata-shm
NotesV7.storedata

from

~/Library/Containers/com.apple.Notes/Data/Library/Notes/

but the files in Monterey are in:

~/Library/Containers/Notes/Data/Library/Notes/

What makes it confusing is that there are 3 Notes directories in that directory on my system. Only the first one has the files above. The other 2 are only ~500 KB in size vs ~10 MB for the first one. Don't understand how there can be 3 directories with the same name in the same folder.

 

[NoBoMac]

Since you use Time Machine is there a reason you use a manual solution?

1. Mac is not always on
2. Time Machine drive is not always plugged in
3. No need to try figuring out what's what in a Time Machine backup and can restore a single Note vs everything in the backup of the Notes database
4. Backup might be corrupt
5. iPhone is on all the time so Automations to easily make backups on a regular schedule
6. And ultimately, Time Machine will most likely not catch this scenario if 1, 2, and or 4 is in play.
   

   Unfortunately, I deleted a paragraph from a note, not the whole file itself.
   Therefore the file is not in the deleted folder, I lost that paragraph.

#5 might still not catch #6, but hoping have something close to what was there vs possibly even older TM version.

(Add: and more backups, more better, ala 3-2-1 backup practices)

As to locations, yes, appears like a couple of places in play and the Google instructions are old. On Ventura, seems to be the important stuff is in ~/Library/Group Containers/com.apple.notes/NotesStore.* files (those have timestamps of today for me vs ~/Library/Containers/com.apple.Notes/Data/Library/Notes/ contains old/stale data [.storedata file shows Nov 2020 date]).

 

[Flynnsworth]

What are my options?

It sounds like they are gone, but make sure you thoroughly check all of the potential notes directories for the various versions of MacOS. E.g:

Locally stored:
~/Library/Group Containers/group.com.apple.notes/

iCloud:
~/Library/Containers/com.apple.Notes/

Other potential locations:
~/Library/Containers/com.apple.Notes/Data/CloudKit/

~/Library/Containers/com.apple.Notes/Data/Library/Notes/

A deep search for the file named: NoteStore.sqlite

- - -

Not an exhaustive list, and if you’ve upgraded the OS a number of times over the years it’s possible that they aren’t where you think.

The best 3 tools at your disposal:

Terminal: great for deep searching, directory lists etc

Spotlight: if you remember some actual text from any of your notes then there’s a chance Spotlight will find it if it’s on the drive

Data Recovery Tools: a last resort before sending your device to a professional or giving up. Try something like Disk Drill/ EaseUS/ iSkysoft

- - -

Good luck

 

[Mike Boreham]

It sounds like they are gone, but make sure you thoroughly check all of the potential notes directories for the various versions of MacOS. E.g:

Locally stored:
~/Library/Group Containers/group.com.apple.notes/

iCloud:
~/Library/Containers/com.apple.Notes/

Thanks. Interesting.

As I reported in this post I successfully restored both iCloud Notes and On-My-Mac Notes by restoring ~/Library/Group Containers/group.com.apple.notes/..the first of the two above. I did not use anything else.

I have been digging a bit on the topic and find I also have seven Notes related folders in ~/Library/Containers/...(NB not Group Containers)....none of which were used in the successful restore.

Note that these are all tiny except the 8.86GB one. Drilling down shows that the 8.86GB is in a Caches folder. I dont know if there is any way these Caches could be used instead.

The size of my ~/Library/Group Containers/group.com.apple.notes/ is 3.06GB

All this reinforces my belief that Notes are quite different from other types of data...no separate files like .jpegs, .pages, and all in an sqlite database.

Backing Notes up is no problem, the difficult part is restoring. A successful backup strategy includes testing the restore part.

 

[Flynnsworth]

Backing Notes up is no problem, the difficult part is restoring. A successful backup strategy includes testing the restore part.

If your 8.86GB folder contains the following:
- A database named NoteStore.sqlite
- A folder named Media

Then as a restore I would suggest:
- Back up this (8.86GB) folder before making changes
- Also back up ~/Library/Group Containers/group.com.apple.notes before making changes

- Close Notes.app
- Go to ~/Library/Group Containers/group.com.apple.notes and delete the following 3 files:
NoteStore.sqlite
NoteStore.sqlite-shm
NoteStore.sqlite.wal

- Copy NoteStore.sqlite file and the Media folder from your 8.86GB folder to ~/Library/Group Containers/group.com.apple.notes
- Open Notes.app and the files should be restored.