A Hidden Snow Leopard Feature: HFS+ Compression

Discussion in 'OS X' started by brkirch, Sep 5, 2009.

  1. macrumors regular

    Joined:
    Oct 18, 2001
    #1
    As you may already know, in order to reduce the size of Snow Leopard many of the system files are stored compressed. There are some technical details about this new compression in Ars Technica's Snow Leopard review but no details about how to use it for your own files. So I looked through the Snow Leopard command line tools and found this option for ditto:
    Code:
         --hfsCompression
                       When copying files or extracting content from an archive,
                       if the destination is an HFS+ volume that supports compres-
                       sion, all the content will be compressed if appropriate.
                       This is only supported on Mac OS X 10.6 or later, and is
                       only intended to be used in installation and backup scenar-
                       ios that involve system files. Since files using HFS+ com-
                       pression are not readable on versions of Mac OS X earlier
                       than 10.6, this flag should not be used when dealing with
                       non-system files or other user-generated content.
    After testing it out on some files I found that it does indeed apply HFS+ compression, although it seems that only files that meet certain conditions can have HFS+ compression applied to them. Regardless it can still create HUGE space savings. And there shouldn't be any issues as long as you don't mount a disk with HFS+ compressed files directly using a pre-10.6 system, because the files are decompressed when copied (so to use a compressed file on another computer without Snow Leopard, you simply need to make a copy of the file).

    After freeing up several GBs from my Applications folder I was curious exactly how much space I had saved, but I ran across a problem: the Finder still reported exactly the same sizes. The Finder definitely reported more available space on the disk, but there wasn't anything I could find that would tell me the new size of my Applications folder; everything still reported the sizes that the files were before compression.

    Anyway, to make a long story short I wrote my own command line utility (afsctool) for identifying HFS+ compressed files and getting the sizes of them. Its output looks something like this:
    Code:
    $ afsctool -v /usr/local/bin/afsctool
    /usr/local/bin/afsctool:
    File is HFS+ compressed.
    File size (uncompressed data fork; reported size by Mac OS 10.6+ Finder): 85372 bytes / 85 KB (kilobytes) / 83 KiB (kibibytes)
    File size (compressed data fork - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 24417 bytes / 25 KB (kilobytes) / 24 KiB (kibibytes)
    File size (compressed data fork): 24433 bytes / 25 KB (kilobytes) / 24 KiB (kibibytes)
    Compression savings: 71.4%
    Number of extended attributes: 0
    Total size of extended attribute data: 0 bytes
    Appoximate overhead of extended attributes: 536 bytes
    Appoximate total file size (compressed data fork + EA + EA overhead + file overhead): 25376 bytes / 25 KB (kilobytes) / 25 KiB (kibibytes)
    And here's what I get when I run it on my System folder:
    Code:
    $ afsctool -v /System
    /System:
    Number of HFS+ compressed files: 118930
    Total number of files: 141558
    Total number of folders: 57257
    Total number of items (number of files + number of folders): 198815
    Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 4207439143 bytes / 4.59 GB (gigabytes) / 4.28 GiB (gibibytes)
    Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 2443700774 bytes / 2.55 GB (gigabytes) / 2.38 GiB (gibibytes)
    Folder size (compressed): 2523293328 bytes / 2.63 GB (gigabytes) / 2.45 GiB (gibibytes)
    Compression savings: 40.0%
    Appoximate total folder size (files + file overhead + folder overhead): 2708108395 bytes / 2.71 GB (gigabytes) / 2.52 GiB (gibibytes)
    Edit: I've updated afsctool and added in place HFS+ compression for files and folders, just be warned that you should make a backup before attempting to use it as I haven't had a chance to extensively test it yet.

    Edit 2: The in place compression seems not to have any significant issues, but if you are compressing anything important then always include the -k flag just to be safe.

    Download afsctool (source included) here:
    http://web.me.com/brkirch/brkirchs_Software/afsctool/afsctool.html

    Let me know what you think; for me this HFS+ compression feature has been quite useful, giving back 10+ GB of space (not including the space saved in the Snow Leopard install)!
     
  2. macrumors 68000

    Joined:
    Sep 24, 2008
    Location:
    Boon Docks USA
    #2
    Starting to wonder if allot of programs installed with snow leopard are still compressed on the hard drive to save space. Then when opened, they decompress until finished, then compressed again. Would explain the higher heat and shorter battery life on laptops folks have been complaining about. I've read some of the review and seems to point this way.
     
  3. macrumors 6502a

    Joined:
    Feb 12, 2008
    #3
    I'm not quite sure why they would do this: drive space is so plentiful, it is dirt cheap today. Compression of a file system, however, adds a veritable assload of system overhead, as things have to be dynamically decompressed on every since execution.

    Edit: thanks for the little utility brkirch, it was quite useful. It seems that my system folder is yielding a 41.5% compression savings. I don't know if that is a good or bad--we probably won't know until there is more information about the overhead of all this de/re/compressing going on.
     
  4. thread starter macrumors regular

    Joined:
    Oct 18, 2001
    #4
    Actually, Snow Leopard never recompresses anything. If you change a HFS+ compressed file then it is uncompressed and it says that way. Also it would seem that HFS+ compression is only for smaller files (< 20 MB) that can be loaded entirely into memory. That means that HFS+ compression can only really have a effect on performance when the file is first loaded. And from the tests I have done, initially files in their compressed form read and decompress faster than simply reading the files in their uncompressed form (from what I can tell, decompressing uses a negligible amount of CPU).
     
  5. macrumors 68000

    Joined:
    Sep 24, 2008
    Location:
    Boon Docks USA
    #5
    Is there an certain extention on a compressed file that we can do a search to see if there are in fact compressed files on the hard drive? Or a program that lists whats happening when you open a application?
     
  6. macrumors 68000

    Joined:
    Sep 24, 2008
    Location:
    Boon Docks USA
    #6
    Was just looking at the link to the review again. This caught my eye:

    In Snow Leopard, other kinds of files climb on board the compression bandwagon. To give just one example, ninety-seven percent of the executable files in Snow Leopard are compressed.

    If this is the case, then it would indeed decompress and recompress when opening and closing an application. Am I wrong?
     
  7. macrumors G3

    Joined:
    Jul 25, 2007
    Location:
    California
    #7
    The time it takes to decompress is small compared to the time saved loading the smaller executable into memory.
     
  8. macrumors 6502

    Joined:
    Jan 6, 2008
    #8
    The Ars review on Snow Leopard explained it very well --

    But compression isn't just about saving disk space. It's also a classic example of trading CPU cycles for decreased I/O latency and bandwidth. Over the past few decades, CPU performance has gotten better (and computing resources more plentiful—more on that later) at a much faster rate than disk performance has increased. Modern hard disk seek times and rotational delays are still measured in milliseconds. In one millisecond, a 2 GHz CPU goes through two million cycles. And then, of course, there's still the actual data transfer time to consider.
     
  9. macrumors 68030

    Catfish_Man

    Joined:
    Sep 13, 2001
    Location:
    Portland, OR
    #9
    Disk io is pretty much the worst thing for performance. Unusual tradeoffs start making sense to avoid it.
     
  10. Moderator

    maflynn

    Staff Member

    Joined:
    May 3, 2009
    Location:
    Boston
    #10
    As Adam points out, the Ars article really gives you the details in how it it works. Part wonderful design, part kludge for using the metadata.
     
  11. macrumors newbie

    Joined:
    Nov 16, 2008
    Location:
    VB, VA
    #11
    Good article, however i must have skipped over it but what is the cmd line for compressing apps and their contents to save this disk space?
     
  12. macrumors 68000

    Joined:
    Sep 24, 2008
    Location:
    Boon Docks USA
    #12
    The draw back from using compressed files that require decompressing/compressing when used is the wear and tear on hard drives/SSD drives, generates more heat as your making the processor's and memory modules work more so basically working your computer more. But I would assume by keeping the program open all the time, this would reduce this affect (unless you power off your system). I am all for reducing size of programs and OS but not at the expense of more wear and tear on my system.
     
  13. macrumors G3

    Joined:
    Jul 25, 2007
    Location:
    California
    #13
    There would be less wear and tear on hard drives/SSD drives. Less data is read from them (a compressed executable). The decompression happens into RAM, not to the disk (decompressing to disk would defeat the purpose).

    The memory modules don't work any harder - whether you compress or not, the full uncompressed version is stored in RAM. (yeah, during decompression you'll have some intermediates stored in RAM).

    Your CPU works a little harder, but decompression isn't what we call "heavy lifting," and it won't have any noticeable effect on the lifetime of the CPU. Since the CPU would be idle for milliseconds while waiting for all the uncompressed executable to load, why not spend much less time loading it and give your CPU a little work to do in the mean time.
     
  14. macrumors 68000

    Joined:
    Jul 1, 2008
    Location:
    Sweden
    #14
    Maybe the iTablet wont have a big hard drive ;)
     
  15. thread starter macrumors regular

    Joined:
    Oct 18, 2001
    #15
    I updated the afsctool to report more size information when the -v option is used, download the updated version from the link in the first post of this topic.

    The command line utility ditto can be used to apply HFS+ compression. First you need to make a copy of the file or folder from the terminal with this command:
    Code:
    ditto --hfsCompression <src> <dst>
    If the copy is successful then you can move the original file or folder to the trash. And then you can rename the HFS+ compressed version to take the place of the original:
    Code:
    mv <dst> <src>
     
  16. thread starter macrumors regular

    Joined:
    Oct 18, 2001
    #16
    I updated afsctool with the ability to apply HFS+ compression to files in place. :cool: It usually gets even better compression ratios than ditto. (Edit: I've changed it to use the same compression level as ditto, it should actually now produce identical compressed files unless you specify a different compression level) Just redownload afsctool from the first post if you had downloaded it before to get the updated version.
     
  17. macrumors member

    jasonrm

    Joined:
    Jan 26, 2008
    Location:
    Arizona, U.S.A.
    #17
    Cool idea, however, (there always is a however right?) I can't seem to get in place compression to work...

    Code:
    jason@macbookpro /Applications$afsctool -v Dropbox.app
    Dropbox.app:
    Folder contains no compressed files
    Total number of files: 250
    Total number of folders: 90
    Total number of items (number of files + number of folders): 340
    Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
    Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
    Folder size (compressed): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
    Compression savings: 0.0%
    Appoximate total folder size (files + file overhead + folder overhead): 50614948 bytes / 50.6 MB (megabytes) / 48.3 MiB (mebibytes)
    
    jason@macbookpro /Applications$afsctool -c Dropbox.app
    Dropbox.app:
    No compressable files in folder
    
    jason@macbookpro /Applications$afsctool -v Dropbox.app
    Dropbox.app:
    Folder contains no compressed files
    Total number of files: 250
    Total number of folders: 90
    Total number of items (number of files + number of folders): 340
    Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
    Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
    Folder size (compressed): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
    Compression savings: 0.0%
    Appoximate total folder size (files + file overhead + folder overhead): 50614948 bytes / 50.6 MB (megabytes) / 48.3 MiB (mebibytes)
    
    I can use ditto however 'ditto --hfsCompression Dropbox.app Dropbox-new.app' and I get a compression savings of ~75%, and I've tried multiple different applications and folders (however I can compress some files like gifs and tars).

    Also, are you using any special build process to compile? I'm getting errors about undefined symbols with just 'gcc afsctool.c -o afsctool', might be something on my end but worth asking.

    Very cool idea... thanks for making the utility and code available.
     
  18. thread starter macrumors regular

    Joined:
    Oct 18, 2001
    #18
    Most likely you don't have permission to change the files you are trying to compress, try
    Code:
    $sudo afsctool -c Dropbox.app
    instead of just
    Code:
    $afsctool -c Dropbox.app
    It probably should display an error message about that, so I added one (redownload to get a version that throws tons of permission denied errors at you ;)).

    Try this command instead:
    Code:
    gcc -arch x86_64 -arch i386 -lz -o afsctool afsctool.c
     
  19. macrumors member

    jasonrm

    Joined:
    Jan 26, 2008
    Location:
    Arizona, U.S.A.
    #19
    Thanks brkirch. Should have been obvious that I needed the zlib flag (as the errors were related), but it compiles just fine now.

    I finally was able to get it working by using absolute paths
    Code:
    sudo afsctool -c /Applications/Dropbox.app
    instead of relative,
    Code:
    sudo afsctool -c Dropbox.app
    Again, thanks.
     
  20. thread starter macrumors regular

    Joined:
    Oct 18, 2001
    #20
    It should be fixed now, redownload and let me know if you find anything else broken ;)
     
  21. dfs
    macrumors regular

    Joined:
    Sep 17, 2008
    Location:
    California
    #21
    Time Machine and compression?

    When I was about to upgrade to Snow Leopard I first cloned my Mac to the external disk where I keep my Time Machine files. Then I upgraded. A couple of days later, when I was sure that the upgrade had gone right in every respect (it did) I trashed all the clone files on the external. And got back about fifth gb. of space. Since then, Time Machine backups have been eating up a whole lot less disk space than before, in fact virtually none. I've been puzzled about what the heck is going on. Reading this article, I wonder if this new compression scheme is the answer. Have other people who have made the upgrade and use TM noticed the same phenomenon?
     
  22. macrumors member

    Joined:
    Mar 11, 2006
    #22
    Just Amazing!

    I have used the afsctool and I recover about 4 Gigabytes already, and I have some folders to run with it yet. My apps seems to open faster, as I have a Macbook I always notice the hard drive was the slower part of the system, Apple has some great ideas. Thank You brkirch for such great app, and cmaier your comment explains HFS compression purpose very well!
     
  23. macrumors newbie

    Joined:
    Aug 6, 2008
    #23
    So, is this not supposed to work with images? I ran "afsctool -c -l iPhoto.app" and I got a huge list of images that supposedly weren't compressed, and when I checked it seemed like it only saved a measly 7.1% of space over the original.
     
  24. macrumors G5

    gnasher729

    Joined:
    Nov 25, 2005
    #24
    The bytes on your hard drive would never be modified, so there is no need to re-compress.
     
  25. thread starter macrumors regular

    Joined:
    Oct 18, 2001
    #25
    I just uploaded a new version, download it and then try this:
    Code:
    $afsctool -d iPhoto.app
    $afsctool -ckvl 9 0 0 iPhoto.app
    Here's what I get after compression has completed:
    Code:
    /Applications/iPhoto.app:
    Number of HFS+ compressed files: 12722
    Total number of files: 12774
    Total number of folders: 2613
    Total number of items (number of files + number of folders): 15387
    Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 401452168 bytes / 430.2 MB (megabytes) / 410.3 MiB (mebibytes)
    Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 180544293 bytes / 187.9 MB (megabytes) / 179.2 MiB (mebibytes)
    Folder size (compressed): 195774246 bytes / 203.1 MB (megabytes) / 193.7 MiB (mebibytes)
    Compression savings: 51.2%
    Appoximate total folder size (files + file overhead + folder overhead): 210705429 bytes / 210.7 MB (megabytes) / 200.9 MiB (mebibytes)
    The new version has support for uncompressed data blocks (required when the size of a data block will otherwise increase after compression). This means that images or anything else that wouldn't compress before should now (although there won't always be much if any compression savings). The only files that will not compress now (provided the compression setting 9 0 0) are files that have a resource fork, zero length data fork, are over 2 GB in size, or are not regular files (symbolic links, for example).

    Note that I used the -k flag when compressing after decompression; you should always do this when recompressing because Apple's HFS+ compression kernel extension has some cache bugs so it may cause some files to appear corrupted if you do not (if you have this problem then resetting your computer or decompressing using afsctool should fix it).
     

Share This Page