Digging further into this, I found some useful info about SQLite3 write-ahead logs and how they might grow to large sizes.
The SQLite3 docs were helpful in this regard:
So based on the documentation for SQLite, we can see that the write-ahead log files can grow to large size in certain situations. Apple must have created one of these situations by their design of the
Syndication.photoslibrary
& "Shared With You" photos features.
We can inspect the current settings of the SQLite3 database as follows. Note that because SQLite3 supports multiple readers & writers at once, this is fine to do without stopping any of the related Apple services (e.g.
photolibraryd
,
mediaanalysisd
, and
photoanalysisd
).
In a terminal, we can open the file with the
sqlite3
command, and look at the
WAL
&
VACUUM
related settings:
SQL:
sqlite3 ~/Library/Photos/Libraries/Syndication.photoslibrary/database/Photos.sqlite
SQLite version 3.37.0 2021-12-09 01:34:53
Enter ".help" for usage hints.
sqlite> PRAGMA journal_mode;
wal
sqlite> PRAGMA journal_size_limit;
32768
sqlite> PRAGMA wal_autocheckpoint;
1000
sqlite> PRAGMA auto_vacuum;
2
sqlite> PRAGMA freelist_count;
0
sqlite> PRAGMA secure_delete;
2
sqlite> .quit
Here, we see the values of some SQLite settings related to the write-ahead log (a.k.a
journal
), the current auto vacuum setting, and the secure delete setting. These can all be read about further in the
SQLite documentation starting here. You might choose later to change some of these settings, but note that only some of them are persistent after the
sqlite3
.quit
command, while some others such as
wal_autocheckpoint
revert to their old value the next time the database is opened. I won't go into this here, and know that doing so might affect database performance and impact how Apple's daemons interact with the database. However, since Apple's engineers seem to have created a situation where the write-ahead log is growing to extremely obnoxious levels, and my old MacBook Pro is on the latest version of macOS (Monterey) that it's capable of running thanks to Apple's EOL (and planned obsolescence), I'm changing my auto_vacuum setting to
1
/
FULL
. Hopefully this will avoid this
WAL
file growing too large in the future for my old machine, since I can not benefit from any bugfixes that Apple might have made in later OS versions.
Whether or not we choose to change some settings, we can address the
WAL
file & disk space issue by truncating it. So...
Next, we can shutdown the services to ensure that no other process is reading or writing to the SQLite database or write-ahead log files. In some situations, the
VACUUM
or
WAL
-related commands can block read & write for other concurrent processes using the database. Since the
WAL
file has grown to extraordinary size and we need to address that, we might as well avoid conflict or database file write blocking so we can slim down the size of those files.
Bash:
launchctl disable gui/$UID/com.apple.photoanalysisd
launchctl disable gui/$UID/com.apple.photolibraryd
launchctl disable gui/$UID/com.apple.mediaanalysisd
launchctl kill -TERM gui/$UID/com.apple.photoanalysisd
launchctl kill -TERM gui/$UID/com.apple.photolibraryd
launchctl kill -TERM gui/$UID/com.apple.mediaanalysisd
Once those daemons are disabled and killed, we can verify that nothing else is using the SQLite database file:
Bash:
sudo lsof ~/Library/Photos/Libraries/Syndication.photoslibrary/database/Photos.sqlite
# Should return no processes
Next, we can open the SQLite database and change some settings related to the
WAL
file size limit and
wal_autocheckpoint
.
Then, we issue the
PRAGMA wal_checkpoint(TRUNCATE)
command to force the
WAL
file entries to be processed and truncate the file. That should address the immediate issue where the
WAL
file has consumed all disk space. Finally, we can run the VACUUM command to slim down the size of the
Photos.sqlite
database, and issue another forced
WAL
truncate command
PRAGMA wal_checkpoint(TRUNCATE)
, for good measure and to ensure the VACUUM operation hasn't re-increased the WAL file size again.
SQL:
sqlite3 ~/Library/Photos/Libraries/Syndication.photoslibrary/database/Photos.sqlite
sqlite> sqlite> PRAGMA journal_size_limit=0;
0
sqlite> sqlite> PRAGMA wal_autocheckpoint=500;
500
sqlite> PRAGMA wal_checkpoint(TRUNCATE);
0|0|0
sqlite> VACUUM;
sqlite> PRAGMA wal_checkpoint(TRUNCATE);
0|0|0
sqlite> .quit
Last but not least, we can check the size of the
Photos.sqlite-wal
file, to make sure that it's been zereoed out.
Bash:
ls -lh ~/Library/Photos/Libraries/Syndication.photoslibrary/database/
total 2739416
-rw-r--r--@ 1 exampleuser staff 298B Sep 23 2023 DataModelVersion.plist
-rw-r--r--@ 1 exampleuser staff 1.3G Aug 15 12:31 Photos.sqlite
-rw-r--r--@ 1 exampleuser staff 32K Aug 15 12:49 Photos.sqlite-shm
-rw-r--r--@ 1 exampleuser staff 0B Aug 15 12:31 Photos.sqlite-wal
-rw-r--r--@ 1 exampleuser staff 48K Sep 23 2023 metaSchema.db
-rw-r--r--@ 1 exampleuser staff 48K Sep 23 2023 photos.db
-rw-r--r--@ 1 exampleuser staff 0B Sep 23 2023 protection
🎉 Success! So, now the
Photos.sqlite-wal
file has zero size (down from
118G
!!), and the
Photos.sqlite
database file has been vacuumed from
1.7G
down to
1.3G
in size!
Hopefully this helps anyone who encounters a similar issue with these semi-hidden
Syndication.photoslibrary
files, which appear to grow because Apple had originally setup incremental vaccuum mode, and likely had not been running the
PRAGMA incremental_vacuum
command and ensuring "reader gaps" periodically enough so WAL checkpointing can truncate the file.
I'm not 100% sure, but it seems that based on all these daemons reading the SQLite database file (some periodically, and
photolibraryd
persistently), Apple has created a situation that the SQLite docs call "
Checkpoint Starvation":
- Checkpoint starvation. A checkpoint is only able to run to completion, and reset the WAL file, if there are no other database connections using the WAL file. If another connection has a read transaction open, then the checkpoint cannot reset the WAL file because doing so might delete content out from under the reader. The checkpoint will do as much work as it can without upsetting the reader, but it cannot run to completion. The checkpoint will start up again where it left off after the next write transaction. This repeats until some checkpoint is able to complete.
However, if a database has many concurrent overlapping readers and there is always at least one active reader, then no checkpoints will be able to complete and hence the WAL file will grow without bound.
This scenario can be avoided by ensuring that there are "reader gaps": times when no processes are reading from the database and that checkpoints are attempted during those times. In applications with many concurrent readers, one might also consider running manual checkpoints with the SQLITE_CHECKPOINT_RESTART or SQLITE_CHECKPOINT_TRUNCATE option which will ensure that the checkpoint runs to completion before returning. The disadvantage of using SQLITE_CHECKPOINT_RESTART and SQLITE_CHECKPOINT_TRUNCATE is that readers might block while the checkpoint is running.
After dealing with all this, the reader can decide whether it's worth re-enabling these
launchctl
daemons...
EDIT: See note 1
[1]: Evidently, people have been trying to disable these daemons without much success. Here are some of those attempted methods: