Create persistent partition for Photos library and other large files

[brookter1]

I could do with some advice about whether to create a new APFS container (or partition?) or volume to ensure that certain large data files are not wiped out by a fresh OS install (i.e. not an upgrade).

I have an iMac 27" (2017) with 512GB, which isn't enough to store my Photos library, music and other permanent files, so these are stored on an external USB3 drive (included in Time Machine backups to other external drives). This setup has worked well for years.

I've just bought a MBP 14" with 1TB, which allows me to keep the Photos library on the internal drive, which is preferable for a laptop. Obviously that's easy enough to sort out.

However, I don't want to have to go to the trouble of having to reinstall the Photos library every time I reinstall the OS (I usually do a fresh install rather than an upgrade). The obvious answer is to keep the Photos library and other such files on a separate partition which will persist over fresh installs.

This was very easy in the old pre-APFS days, but now it doesn't seem to be quite so clear... AFAICT the size of a volume is maintained dynamically by the OS, which makes me wonder whether it will persist over a fresh install, or not. This seems to suggest that I need to create a new container (which I think is the same as an old style partition).

So the question is: for this scenario (keep a separate 'partition' free from fresh installs), do I create a new container or a new volume? Are there any aspects I need to take into account?

Many thanks for any help.

 

[Mike Boreham]

Your question sounds related to this post, except you want to protect a Photos library and he had a backup volume. It wasn't clear whether his backup volume was a separate container or an APFS space sharing volume.

Either way it seems risky, although there might be other factors we don't know about in his situation.

Personally I always upgrade rather than fresh erase and reinstall. I also keep my Photos Lib in the ~/Photos. It avoids these kind of issues. When I have done an erase and reinstall with migration it takes a little longer because the Photos Lib is being restored, but not something that happens often.

 

[brookter1]

Thanks for the quick reply.

I personally don't like upgrading because it means you're dragging along a lot of cruft (I install and uninstall a lot of programs...) and it's an opportunity for a clean slate install.

In principle there really shouldn't be a problem with hiving off a part of the disk in this way (it's recommended if you want to run a beta OS alongside the normal one, and it's how Boot Camp works on non-M1 devices). It's just that I'm not sure whether a volume is wiped out during a fresh install or not or whether than needs a new container/partition.

Thanks again.

 

[Mike Boreham]

It's just that I'm not sure whether a volume is wiped out during a fresh install or not or whether than needs a new container/partition.

I would think a separate container would be safer, but I don't believe either a container or an extra APFS volume should be wiped out by erasing and reinstalling. Assuming you only erase the system plus data volume group. But M1 Macs have ways of surprising.

 

[brookter1]

Thanks!

 

[chabig]

A few thoughts:

1. Clean installs make little sense any more. The system resides on a sealed volume that cannot be corrupted. All you're doing it wiping out your user data and recreating it. Any leftover files you have from previous apps is not likely to be a concern.

2. With APFS, never create new partitions (now called containers). If you want to logically separate data, create a separate volume.

3. There is no reason to put media onto a separate volume. macOS already knows where it wants to store your photos. Let them reside there. Your fear of "I don't want to have to go to the trouble of having to reinstall the Photos library every time I reinstall the OS" is unfounded. No effort is required. OS updates don't mess with user data.

 

[brookter1]

Thanks for your reply!

I’m not sure I understand points 1 and 3. Clean installs make as much sense now as they did before surely: while System files may not have been corrupted, user configuration files may very well have been - a fresh install prevents this accumulating over time. Similarly, I don’t understand your point that there is no effort required in reinstalling the Photo Library: clearly there is time involved as restoring from a backup is unnecessarily prolonged, and MacOS has been perfectly happy for the libraries to be on external drives for years. This is no different, I think.

What’s behind you second point please? If Apple do not intend you to create new containers, why is the feature provided? What are the risks which mean that you should never do this?

Thanks again.

 

[chabig]

Clean installs make as much sense now as they did before surely

Here is an excellent reference why this is not true. Basically, the system volume is unmodifiable once installed, so it cannot ever become corrupted in any manner. Thus, there is no need to reinstall it. User data can always be erased. You can choose to reinstall all of your apps and data at any time. I simply don't think there is a reason to do so unless you are experiencing undesired behavior.

Monterey 12.0.1, Big Sur 11.6.1 and Catalina Security Update 2021-007: a summary

If you haven’t yet installed any of the updates and upgrades released by Apple on 25 October 2021, here’s a quick summary and some matters arising. Monterey 12.0.1 If you’ve just …

eclecticlight.co

Quote: "One trick which is of little use now is performing a clean re-install. This is because every bootable installation of Monterey (and Big Sur) is perfect throughout its System volume. Before Big Sur introduced the Sealed System Volume, it was easy for parts of the new system to have minor errors which could make a clean re-install a useful tool. Now, when the System has been installed, its integrity is checked and validated thoroughly by the hashes which build up to the Seal at the top – and that has to match the hash set by Apple for that version of macOS, or your Mac won’t boot from it."

Here's another good article on the same blog:

Should you clean install Big Sur or Monterey?

Why would you want to go to the lengths of erasing your startup disk and installing a macOS upgrade clean? Here’s why, and how to do it.

eclecticlight.co

I don’t understand your point that there is no effort required in reinstalling the Photo Library

I don't think I articulated well. I'm advocating that you update in place (i.e., install Monterey over Big Sur). When you do it that way, no user data needs to be reinstalled.

As for Containers versus Volumes, Containers are a feature of the file system that is used by the operating system. Therefore, file system utilities provide access to it. However, for user, that feature has essentially been replaced by the concept of volumes.

Juggling with HFS+ and APFS partitions and volumes: a primer

The differences between HFS+ and APFS volumes explored. What a container is, and how to add new HFS+ and APFS volumes to a disk.

eclecticlight.co

Inside the file system: 3 APFS containers and volumes

What you need to know about the APFS file system, its containers and volumes, so you can understand errors and how to repair them.

eclecticlight.co

When should you use APFS Containers, Volumes, and Folders?

There isn’t as much choice in HFS+: you have just two devices for dividing a disk, volumes and folders. Volumes have fixed allocations of space which can’t be shared, whereas folders sh…

eclecticlight.co

 

[brookter1]

Thank you very much for your comprehensive reply — I really appreciate the time you've taken to find these resources.

I think I may not have explained the reason for a fresh install well enough, though. I do know about the sealed System Volume and the way it protects the system files — it seems to be a useful feature (thought it does increase the size of updates.

But the reason for fresh installs isn't that I'm worried about the system volume data, it's that over time user data can become unnecessarily convoluted. Over the course of a year I install and uninstall a lot of programs, and I do a fair bit of messing around in the terminal with Homebrew, Emacs, Vim and so on. Many of those programs, and their associated configuration files, will never be used again. I *could* upgrade and go through each one and try to remove their detritus, but the reality is I never do.

So, every now and then (usually at a major upgrade), I take the opportunity to do a fresh install and clear it all away to start from a clean basis. What you see as unnecessary work, I see as a benefit ?, and encrypted system volumes don't change that assessment. In fact Howard recognises that in the second of the articles you link to:

So as far as the System volume is concerned, there’s nothing to be gained in a clean over a regular install.

Where there can be benefits are in those system files, including Safari, which are stored not on the SSV, but on your writeable Data volume, and in the others there. Those include your bulging Library folder which is bound to contain the remains of all sorts of apps long since removed. If you see this as a good opportunity to clean those up, and are prepared to spend the time doing so, you may find a clean install worth the extra effort.

I do find it worth the extra effort ?. What has changed with Monterey is the addition of the 'wipe clean feature' — in future I'll do that, rather than do the old usb/internet reinstall, but I'll still get the benefit of a clean slate to install the apps I want at the moment, which upgrading doesn't provide.

When the photos library and the other huge files were on an external drive, this didn't matter as all I had to do was repoint the apps at the correct location. But now I want to move them to the internal drive, I'd have to reimport them from backup which is long and tedious.

Hence the desire to create a new 'permanent' additional drive internally — it will have the same function as the old external drive and it won't be overwritten in the fresh install process.

I'll read the other articles in more detail later to get a better understanding of how the volumes etc work.

Thanks again for your help!

 

[chabig]

But the reason for fresh installs isn't that I'm worried about the system volume data, it's that over time user data can become unnecessarily convoluted. Over the course of a year I install and uninstall a lot of programs, and I do a fair bit of messing around in the terminal with Homebrew, Emacs, Vim and so on. Many of those programs, and their associated configuration files, will never be used again. I *could* upgrade and go through each one and try to remove their detritus, but the reality is I never do.

You've got it. There may be a lot of those old configuration files sitting around in your user Library, never to be used again. But they also aren't causing any problems by their existence. If you prefer to reinstall now and then to make sure none of those files exist, it's ok. I've done it now and then, but generally only when I am experiencing some objective problem.

By the way, Monterey now has an easy way to start over without reinstalling the operating system. Open System Preferences, then select "Erase all Content and Settings". That will erase everything except the sealed system volume, leaving your machine minty fresh, just like it came out of the factory box.

 

[brookter1]

You've got it. There may be a lot of those old configuration files sitting around in your user Library, never to be used again. But they also aren't causing any problems by their existence. If you prefer to reinstall now and then to make sure none of those files exist, it's ok. I've done it now and then, but generally only when I am experiencing some objective problem.

By the way, Monterey now has an easy way to start over without reinstalling the operating system. Open System Preferences, then select "Erase all Content and Settings". That will erase everything except the sealed system volume, leaving your machine minty fresh, just like it came out of the factory box.

Yes, that's the 'wipe clean' feature I've mentioned in my post -- makes the periodic clean slate much easier.

I've been doing the 'fresh install' on a major upgrade since 2004 and have never had a problem with it. The odd time I've upgraded I've never felt that the OS has benefited. If you don't change your apps much then of course an upgrade is fine (and it's the 'easy' option, which Apple does a really good job of facilitating), but in my experience, the fresh install route is well worth it.

I've left a comment on Howard's site to see if he can shed any more light on the Container vs Volume question (in specific relation to these circumstances).

Thanks again for your help.