A Hidden Snow Leopard Feature: HFS+ Compression

[brkirch]

As you may already know, in order to reduce the size of Snow Leopard many of the system files are stored compressed. There are some technical details about this new compression in Ars Technica's Snow Leopard review but no details about how to use it for your own files. So I looked through the Snow Leopard command line tools and found this option for ditto:

     --hfsCompression
                   When copying files or extracting content from an archive,
                   if the destination is an HFS+ volume that supports compres-
                   sion, all the content will be compressed if appropriate.
                   This is only supported on Mac OS X 10.6 or later, and is
                   only intended to be used in installation and backup scenar-
                   ios that involve system files. Since files using HFS+ com-
                   pression are not readable on versions of Mac OS X earlier
                   than 10.6, this flag should not be used when dealing with
                   non-system files or other user-generated content.

After testing it out on some files I found that it does indeed apply HFS+ compression, although it seems that only files that meet certain conditions can have HFS+ compression applied to them. Regardless it can still create HUGE space savings. And there shouldn't be any issues as long as you don't mount a disk with HFS+ compressed files directly using a pre-10.6 system, because the files are decompressed when copied (so to use a compressed file on another computer without Snow Leopard, you simply need to make a copy of the file).

After freeing up several GBs from my Applications folder I was curious exactly how much space I had saved, but I ran across a problem: the Finder still reported exactly the same sizes. The Finder definitely reported more available space on the disk, but there wasn't anything I could find that would tell me the new size of my Applications folder; everything still reported the sizes that the files were before compression.

Anyway, to make a long story short I wrote my own command line utility (afsctool) for identifying HFS+ compressed files and getting the sizes of them. Its output looks something like this:

$ afsctool -v /usr/local/bin/afsctool
/usr/local/bin/afsctool:
File is HFS+ compressed.
File size (uncompressed data fork; reported size by Mac OS 10.6+ Finder): 85372 bytes / 85 KB (kilobytes) / 83 KiB (kibibytes)
File size (compressed data fork - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 24417 bytes / 25 KB (kilobytes) / 24 KiB (kibibytes)
File size (compressed data fork): 24433 bytes / 25 KB (kilobytes) / 24 KiB (kibibytes)
Compression savings: 71.4%
Number of extended attributes: 0
Total size of extended attribute data: 0 bytes
Appoximate overhead of extended attributes: 536 bytes
Appoximate total file size (compressed data fork + EA + EA overhead + file overhead): 25376 bytes / 25 KB (kilobytes) / 25 KiB (kibibytes)

And here's what I get when I run it on my System folder:

$ afsctool -v /System
/System:
Number of HFS+ compressed files: 118930
Total number of files: 141558
Total number of folders: 57257
Total number of items (number of files + number of folders): 198815
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 4207439143 bytes / 4.59 GB (gigabytes) / 4.28 GiB (gibibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 2443700774 bytes / 2.55 GB (gigabytes) / 2.38 GiB (gibibytes)
Folder size (compressed): 2523293328 bytes / 2.63 GB (gigabytes) / 2.45 GiB (gibibytes)
Compression savings: 40.0%
Appoximate total folder size (files + file overhead + folder overhead): 2708108395 bytes / 2.71 GB (gigabytes) / 2.52 GiB (gibibytes)

Edit: I've updated afsctool and added in place HFS+ compression for files and folders, just be warned that you should make a backup before attempting to use it as I haven't had a chance to extensively test it yet.

Edit 2: The in place compression seems not to have any significant issues, but if you are compressing anything important then always include the -k flag just to be safe.

Download afsctool (source included) here:
http://web.me.com/brkirch/brkirchs_Software/afsctool/afsctool.html

Let me know what you think; for me this HFS+ compression feature has been quite useful, giving back 10+ GB of space (not including the space saved in the Snow Leopard install)!

 

[bearcatrp]

Starting to wonder if allot of programs installed with snow leopard are still compressed on the hard drive to save space. Then when opened, they decompress until finished, then compressed again. Would explain the higher heat and shorter battery life on laptops folks have been complaining about. I've read some of the review and seems to point this way.

 

[Riemann Zeta]

I'm not quite sure why they would do this: drive space is so plentiful, it is dirt cheap today. Compression of a file system, however, adds a veritable assload of system overhead, as things have to be dynamically decompressed on every since execution.

Edit: thanks for the little utility brkirch, it was quite useful. It seems that my system folder is yielding a 41.5% compression savings. I don't know if that is a good or bad--we probably won't know until there is more information about the overhead of all this de/re/compressing going on.

 

[brkirch]

Actually, Snow Leopard never recompresses anything. If you change a HFS+ compressed file then it is uncompressed and it says that way. Also it would seem that HFS+ compression is only for smaller files (< 20 MB) that can be loaded entirely into memory. That means that HFS+ compression can only really have a effect on performance when the file is first loaded. And from the tests I have done, initially files in their compressed form read and decompress faster than simply reading the files in their uncompressed form (from what I can tell, decompressing uses a negligible amount of CPU).

 

[bearcatrp]

Is there an certain extention on a compressed file that we can do a search to see if there are in fact compressed files on the hard drive? Or a program that lists whats happening when you open a application?

 

[bearcatrp]

Was just looking at the link to the review again. This caught my eye:

In Snow Leopard, other kinds of files climb on board the compression bandwagon. To give just one example, ninety-seven percent of the executable files in Snow Leopard are compressed.

If this is the case, then it would indeed decompress and recompress when opening and closing an application. Am I wrong?

 

[cmaier]

The time it takes to decompress is small compared to the time saved loading the smaller executable into memory.

 

[adamk77]

 

 

[Catfish_Man]

Disk io is pretty much the worst thing for performance. Unusual tradeoffs start making sense to avoid it.

 

[maflynn]

As Adam points out, the Ars article really gives you the details in how it it works. Part wonderful design, part kludge for using the metadata.

 

[thisguy23]

Good article, however i must have skipped over it but what is the cmd line for compressing apps and their contents to save this disk space?

 

[bearcatrp]

The draw back from using compressed files that require decompressing/compressing when used is the wear and tear on hard drives/SSD drives, generates more heat as your making the processor's and memory modules work more so basically working your computer more. But I would assume by keeping the program open all the time, this would reduce this affect (unless you power off your system). I am all for reducing size of programs and OS but not at the expense of more wear and tear on my system.

 

[cmaier]

The draw back from using compressed files that require decompressing/compressing when used is the wear and tear on hard drives/SSD drives, generates more heat as your making the processor's and memory modules work more so basically working your computer more. But I would assume by keeping the program open all the time, this would reduce this affect (unless you power off your system). I am all for reducing size of programs and OS but not at the expense of more wear and tear on my system.

There would be less wear and tear on hard drives/SSD drives. Less data is read from them (a compressed executable). The decompression happens into RAM, not to the disk (decompressing to disk would defeat the purpose).

The memory modules don't work any harder - whether you compress or not, the full uncompressed version is stored in RAM. (yeah, during decompression you'll have some intermediates stored in RAM).

Your CPU works a little harder, but decompression isn't what we call "heavy lifting," and it won't have any noticeable effect on the lifetime of the CPU. Since the CPU would be idle for milliseconds while waiting for all the uncompressed executable to load, why not spend much less time loading it and give your CPU a little work to do in the mean time.

 

[fishmoose]

I'm not quite sure why they would do this: drive space is so plentiful, it is dirt cheap today. Compression of a file system, however, adds a veritable assload of system overhead, as things have to be dynamically decompressed on every since execution.

Maybe the iTablet wont have a big hard drive

 

[brkirch]

I updated the afsctool to report more size information when the -v option is used, download the updated version from the link in the first post of this topic.

Good article, however i must have skipped over it but what is the cmd line for compressing apps and their contents to save this disk space?

The command line utility ditto can be used to apply HFS+ compression. First you need to make a copy of the file or folder from the terminal with this command:

ditto --hfsCompression <src> <dst>

If the copy is successful then you can move the original file or folder to the trash. And then you can rename the HFS+ compressed version to take the place of the original:

mv <dst> <src>

 

[brkirch]

I updated afsctool with the ability to apply HFS+ compression to files in place. It usually gets even better compression ratios than ditto. (Edit: I've changed it to use the same compression level as ditto, it should actually now produce identical compressed files unless you specify a different compression level) Just redownload afsctool from the first post if you had downloaded it before to get the updated version.

 

[jasonrm]

I updated afsctool with the ability to apply HFS+ compression to files in place. It usually gets even better compression ratios than ditto. Just redownload afsctool from the first post if you had downloaded it before to get the updated version.

Cool idea, however, (there always is a however right?) I can't seem to get in place compression to work...

jason@macbookpro /Applications$afsctool -v Dropbox.app
Dropbox.app:
Folder contains no compressed files
Total number of files: 250
Total number of folders: 90
Total number of items (number of files + number of folders): 340
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Folder size (compressed): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Compression savings: 0.0%
Appoximate total folder size (files + file overhead + folder overhead): 50614948 bytes / 50.6 MB (megabytes) / 48.3 MiB (mebibytes)

jason@macbookpro /Applications$afsctool -c Dropbox.app
Dropbox.app:
No compressable files in folder

jason@macbookpro /Applications$afsctool -v Dropbox.app
Dropbox.app:
Folder contains no compressed files
Total number of files: 250
Total number of folders: 90
Total number of items (number of files + number of folders): 340
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Folder size (compressed): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Compression savings: 0.0%
Appoximate total folder size (files + file overhead + folder overhead): 50614948 bytes / 50.6 MB (megabytes) / 48.3 MiB (mebibytes)

I can use ditto however 'ditto --hfsCompression Dropbox.app Dropbox-new.app' and I get a compression savings of ~75%, and I've tried multiple different applications and folders (however I can compress some files like gifs and tars).

Also, are you using any special build process to compile? I'm getting errors about undefined symbols with just 'gcc afsctool.c -o afsctool', might be something on my end but worth asking.

Very cool idea... thanks for making the utility and code available.

 

[brkirch]

Cool idea, however, (there always is a however right?) I can't seem to get in place compression to work...

Most likely you don't have permission to change the files you are trying to compress, try

$sudo afsctool -c Dropbox.app

instead of just

$afsctool -c Dropbox.app

It probably should display an error message about that, so I added one (redownload to get a version that throws tons of permission denied errors at you ).

Also, are you using any special build process to compile? I'm getting errors about undefined symbols with just 'gcc afsctool.c -o afsctool', might be something on my end but worth asking.

Try this command instead:

gcc -arch x86_64 -arch i386 -lz -o afsctool afsctool.c

 

[jasonrm]

Thanks brkirch. Should have been obvious that I needed the zlib flag (as the errors were related), but it compiles just fine now.

I finally was able to get it working by using absolute paths

sudo afsctool -c /Applications/Dropbox.app

instead of relative,

sudo afsctool -c Dropbox.app

Again, thanks.

 

[brkirch]

It should be fixed now, redownload and let me know if you find anything else broken

 

[dfs]

Time Machine and compression?

When I was about to upgrade to Snow Leopard I first cloned my Mac to the external disk where I keep my Time Machine files. Then I upgraded. A couple of days later, when I was sure that the upgrade had gone right in every respect (it did) I trashed all the clone files on the external. And got back about fifth gb. of space. Since then, Time Machine backups have been eating up a whole lot less disk space than before, in fact virtually none. I've been puzzled about what the heck is going on. Reading this article, I wonder if this new compression scheme is the answer. Have other people who have made the upgrade and use TM noticed the same phenomenon?

 

[SurfSpirit]

Just Amazing!

I have used the afsctool and I recover about 4 Gigabytes already, and I have some folders to run with it yet. My apps seems to open faster, as I have a Macbook I always notice the hard drive was the slower part of the system, Apple has some great ideas. Thank You brkirch for such great app, and cmaier your comment explains HFS compression purpose very well!

 

[LordMelkor]

So, is this not supposed to work with images? I ran "afsctool -c -l iPhoto.app" and I got a huge list of images that supposedly weren't compressed, and when I checked it seemed like it only saved a measly 7.1% of space over the original.

 

[gnasher729]

f this is the case, then it would indeed decompress and recompress when opening and closing an application. Am I wrong?

The bytes on your hard drive would never be modified, so there is no need to re-compress.

 

[brkirch]

So, is this not supposed to work with images? I ran "afsctool -c -l iPhoto.app" and I got a huge list of images that supposedly weren't compressed, and when I checked it seemed like it only saved a measly 7.1% of space over the original.

I just uploaded a new version, download it and then try this:

$afsctool -d iPhoto.app
$afsctool -ckvl 9 0 0 iPhoto.app

Here's what I get after compression has completed:

/Applications/iPhoto.app:
Number of HFS+ compressed files: 12722
Total number of files: 12774
Total number of folders: 2613
Total number of items (number of files + number of folders): 15387
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 401452168 bytes / 430.2 MB (megabytes) / 410.3 MiB (mebibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 180544293 bytes / 187.9 MB (megabytes) / 179.2 MiB (mebibytes)
Folder size (compressed): 195774246 bytes / 203.1 MB (megabytes) / 193.7 MiB (mebibytes)
Compression savings: 51.2%
Appoximate total folder size (files + file overhead + folder overhead): 210705429 bytes / 210.7 MB (megabytes) / 200.9 MiB (mebibytes)

The new version has support for uncompressed data blocks (required when the size of a data block will otherwise increase after compression). This means that images or anything else that wouldn't compress before should now (although there won't always be much if any compression savings). The only files that will not compress now (provided the compression setting 9 0 0) are files that have a resource fork, zero length data fork, are over 2 GB in size, or are not regular files (symbolic links, for example).

Note that I used the -k flag when compressing after decompression; you should always do this when recompressing because Apple's HFS+ compression kernel extension has some cache bugs so it may cause some files to appear corrupted if you do not (if you have this problem then resetting your computer or decompressing using afsctool should fix it).