Memory Management on MacOS?

[AnakChan]

How is memory management handled in MacOS (Big Sur)? I run a program which manages a lot of large photo files and during processing it consumes a lot of memory as pictured below. I have a high chance of my iMac Pro with 64GB of RAM crashing due to this processing. However I noticed that no swapfiles are created by the OS.

Aside from adding more physical memory, is there a way to have MacOS create swap files as I do have disk space.

 

[Ritsuka]

Swap is in a separate apfs volume. Your computer is using 34.45 GB of swap already.
A single app shouldn't try to use more than the physical limit anyway, I think something is going wrong in PixInsight.

 

[gilby101]

Swap is in a separate apfs volume

which macOS also shows as being in /var/vm.

However I noticed that no swapfiles are created by the OS.

Did you notice that /var/vm/sleepimage is 34GB. This is a view of your current swap space.

I run a program which manages a lot of large photo files and during processing it consumes a lot of memory as pictured below.

Is pixinsight still using CPU? If so, it may be that you just have to wait for it to finish processing and that macOS is managing memory as best it can. Your astrophotography files are presumably very large and you need to reduce the sizes that pixinsight is working with. Or there is a bug in it.

Have you asked here https://pixinsight.com/forum/index.php?

You would get a little help by closing Chrome.

 

[AnakChan]

Ah I didn't check to see that /var/vm is on that swap partition. I thought :-
1) the swap partition was separate.
2) sleepimage was only when the iMac goes to sleep and keeps an image of where it's at

OK this makes more sense why it can't grow anymore, it's cos the swap partition has maxed out. I kept expecting more swap files to be dumped into that directory growing as needed.

Yes PixInsight is an astro processing software and I usually dump into it anything from 90 to 220x 122MB raw files for processing. It in turn creates other temp files (but that's ok 'cos I've configured the app to use space in my 32TB external disk for that, plenty of space there).

It's during calculations and processing is where the PixInsight starts to bloat and is consuming memory. I'm not certain if there's a way for me to try to grow that swap partition as I do have space in my root disk.

 

[gilby101]

Ah I didn't check to see that /var/vm is on that swap partition. I thought :-
1) the swap partition was separate.
2) sleepimage was only when the iMac goes to sleep and keeps an image of where it's at

OK this makes more sense why it can't grow anymore, it's cos the swap partition has maxed out. I kept expecting more swap files to be dumped into that directory growing as needed.

Yes PixInsight is an astro processing software and I usually dump into it anything from 90 to 220x 122MB raw files for processing. It in turn creates other temp files (but that's ok 'cos I've configured the app to use space in my 32TB external disk for that, plenty of space there).

It's during calculations and processing is where the PixInsight starts to bloat and is consuming memory. I'm not certain if there's a way for me to try to grow that swap partition as I do have space in my root disk.

The 'swap' volume (VM) is separate from the system volume (notice I use the word 'volume'), but they are both in the same APFS container. The VM will grow further if the system needs it and there is space on the boot/system disk. Do check your free space on the system disk. You can see the VM volume in Disk Utility (with View -> Show All Devices enabled) inside what is probably called Container disk1. Select Container 1 to see all the volumes inside. What you see in /var/vm is just a representation of the content of the VM volume.

The name sleepimage is a bit of a misnomer. But so is swapfile - it is really a pagefile which in the old days was quite distinct from the swapfile.

Your astrophotography is out of my league. Very occasional, up to about 50 (at most) 30 MB raws for which I use Deep Sky Stacker. PI is much more sophisticated!

But I fear you may be at the limit of what you can do with PI and only 64GB RAM. Closing as many other apps as possible will help a bit.

 

[AnakChan]

Hmmm....I actually don't see the VM volume :-

But I know it's there :-
/dev/disk1s4 3908112996 3145752 988403244 1% 3 39081129957 0% /System/Volumes/VM

I'm not certain if it "grows" as needed though but I do know that it crashes my computer and reboots. Next time I'll have a df loop instead.

 

[gilby101]

Hmmm....I actually don't see the VM volume

3rd from there left under the coloured bar. At ~3GB too small to get its own coloured section.

I'm not certain if it "grows" as needed though but I do know that it crashes my computer and reboots.

I'm sure it does grow. But not fast enough (see next para). Crash and reboot is a worry!! And I assume you have closed all apps except PI.

A symptom that worries me about your memory is the 'compressed' memory - 40GB in your first screenshot. That is physical memory containing virtual memory belonging to apps but being compressed until such time as it is needed. macOS avoids paging memory out to the swapfile/VM as long as it can, preferring to compress it and keep it in RAM and only paging out to the swapfile as a last resort. The compressed memory is not actually usable (until it is uncompressed) and that is a serious bottleneck when 40 out of 64GB is compressed. In my view, a 'sensible' OS (starting with a W) would have paged the memory out long ago. That might lead to page thrashing for an app like PI, but the system would not crash.

You are going to have to ask on the PI forums how Mac users tune PI to manage operations on 25GB (~200x122MB) of RAW files. A quick search found this one https://pixinsight.com/forum/index.php?threads/memory-usage-and-system-crash.14601/ And system requirements https://pixinsight.com/sysreq/index.html. That sounds like PI creates lot of temporary files to manage its memory use - do you have these on an external Thunderbolt SSD?

As you can probably tell from the last paragraph I am getting out of my depth trying to understand how PI manages memory. Sorry about that.

 

[ewu]

I think software isn't compatible with big sur.

a few software I am running has new issues running on big sur, and same version of software didn't have problem in Mojave.

you could try to install new Mojave to try.