Go Back   MacRumors Forums > Apple Systems and Services > OS X

Reply
 
Thread Tools Search this Thread Display Modes
Old Sep 5, 2009, 07:49 PM   #1
brkirch
macrumors regular
 
Join Date: Oct 2001
A Hidden Snow Leopard Feature: HFS+ Compression

As you may already know, in order to reduce the size of Snow Leopard many of the system files are stored compressed. There are some technical details about this new compression in Ars Technica's Snow Leopard review but no details about how to use it for your own files. So I looked through the Snow Leopard command line tools and found this option for ditto:
Code:
     --hfsCompression
                   When copying files or extracting content from an archive,
                   if the destination is an HFS+ volume that supports compres-
                   sion, all the content will be compressed if appropriate.
                   This is only supported on Mac OS X 10.6 or later, and is
                   only intended to be used in installation and backup scenar-
                   ios that involve system files. Since files using HFS+ com-
                   pression are not readable on versions of Mac OS X earlier
                   than 10.6, this flag should not be used when dealing with
                   non-system files or other user-generated content.
After testing it out on some files I found that it does indeed apply HFS+ compression, although it seems that only files that meet certain conditions can have HFS+ compression applied to them. Regardless it can still create HUGE space savings. And there shouldn't be any issues as long as you don't mount a disk with HFS+ compressed files directly using a pre-10.6 system, because the files are decompressed when copied (so to use a compressed file on another computer without Snow Leopard, you simply need to make a copy of the file).

After freeing up several GBs from my Applications folder I was curious exactly how much space I had saved, but I ran across a problem: the Finder still reported exactly the same sizes. The Finder definitely reported more available space on the disk, but there wasn't anything I could find that would tell me the new size of my Applications folder; everything still reported the sizes that the files were before compression.

Anyway, to make a long story short I wrote my own command line utility (afsctool) for identifying HFS+ compressed files and getting the sizes of them. Its output looks something like this:
Code:
$ afsctool -v /usr/local/bin/afsctool
/usr/local/bin/afsctool:
File is HFS+ compressed.
File size (uncompressed data fork; reported size by Mac OS 10.6+ Finder): 85372 bytes / 85 KB (kilobytes) / 83 KiB (kibibytes)
File size (compressed data fork - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 24417 bytes / 25 KB (kilobytes) / 24 KiB (kibibytes)
File size (compressed data fork): 24433 bytes / 25 KB (kilobytes) / 24 KiB (kibibytes)
Compression savings: 71.4%
Number of extended attributes: 0
Total size of extended attribute data: 0 bytes
Appoximate overhead of extended attributes: 536 bytes
Appoximate total file size (compressed data fork + EA + EA overhead + file overhead): 25376 bytes / 25 KB (kilobytes) / 25 KiB (kibibytes)
And here's what I get when I run it on my System folder:
Code:
$ afsctool -v /System
/System:
Number of HFS+ compressed files: 118930
Total number of files: 141558
Total number of folders: 57257
Total number of items (number of files + number of folders): 198815
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 4207439143 bytes / 4.59 GB (gigabytes) / 4.28 GiB (gibibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 2443700774 bytes / 2.55 GB (gigabytes) / 2.38 GiB (gibibytes)
Folder size (compressed): 2523293328 bytes / 2.63 GB (gigabytes) / 2.45 GiB (gibibytes)
Compression savings: 40.0%
Appoximate total folder size (files + file overhead + folder overhead): 2708108395 bytes / 2.71 GB (gigabytes) / 2.52 GiB (gibibytes)
Edit: I've updated afsctool and added in place HFS+ compression for files and folders, just be warned that you should make a backup before attempting to use it as I haven't had a chance to extensively test it yet.

Edit 2: The in place compression seems not to have any significant issues, but if you are compressing anything important then always include the -k flag just to be safe.

Download afsctool (source included) here:
http://web.me.com/brkirch/brkirchs_S.../afsctool.html

Let me know what you think; for me this HFS+ compression feature has been quite useful, giving back 10+ GB of space (not including the space saved in the Snow Leopard install)!

Last edited by brkirch; Nov 17, 2009 at 07:58 PM.
brkirch is offline   2 Reply With Quote
Old Sep 5, 2009, 08:32 PM   #2
bearcatrp
macrumors 68000
 
Join Date: Sep 2008
Location: Boon Docks USA
Starting to wonder if allot of programs installed with snow leopard are still compressed on the hard drive to save space. Then when opened, they decompress until finished, then compressed again. Would explain the higher heat and shorter battery life on laptops folks have been complaining about. I've read some of the review and seems to point this way.
__________________
2010 2.66 Mac Mini(SL), ATV3, iPhone 6P, Dual X5670 2.93 Westmere Win7 PC w/48gb ram & 4930K w/32gb ram Win7.
bearcatrp is offline   0 Reply With Quote
Old Sep 5, 2009, 09:02 PM   #3
Riemann Zeta
macrumors 6502a
 
Join Date: Feb 2008
I'm not quite sure why they would do this: drive space is so plentiful, it is dirt cheap today. Compression of a file system, however, adds a veritable assload of system overhead, as things have to be dynamically decompressed on every since execution.

Edit: thanks for the little utility brkirch, it was quite useful. It seems that my system folder is yielding a 41.5% compression savings. I don't know if that is a good or bad--we probably won't know until there is more information about the overhead of all this de/re/compressing going on.
Riemann Zeta is offline   0 Reply With Quote
Old Sep 5, 2009, 10:42 PM   #4
brkirch
Thread Starter
macrumors regular
 
Join Date: Oct 2001
Actually, Snow Leopard never recompresses anything. If you change a HFS+ compressed file then it is uncompressed and it says that way. Also it would seem that HFS+ compression is only for smaller files (< 20 MB) that can be loaded entirely into memory. That means that HFS+ compression can only really have a effect on performance when the file is first loaded. And from the tests I have done, initially files in their compressed form read and decompress faster than simply reading the files in their uncompressed form (from what I can tell, decompressing uses a negligible amount of CPU).
brkirch is offline   0 Reply With Quote
Old Sep 5, 2009, 10:54 PM   #5
bearcatrp
macrumors 68000
 
Join Date: Sep 2008
Location: Boon Docks USA
Is there an certain extention on a compressed file that we can do a search to see if there are in fact compressed files on the hard drive? Or a program that lists whats happening when you open a application?
__________________
2010 2.66 Mac Mini(SL), ATV3, iPhone 6P, Dual X5670 2.93 Westmere Win7 PC w/48gb ram & 4930K w/32gb ram Win7.
bearcatrp is offline   0 Reply With Quote
Old Sep 5, 2009, 11:06 PM   #6
bearcatrp
macrumors 68000
 
Join Date: Sep 2008
Location: Boon Docks USA
Was just looking at the link to the review again. This caught my eye:

In Snow Leopard, other kinds of files climb on board the compression bandwagon. To give just one example, ninety-seven percent of the executable files in Snow Leopard are compressed.

If this is the case, then it would indeed decompress and recompress when opening and closing an application. Am I wrong?
__________________
2010 2.66 Mac Mini(SL), ATV3, iPhone 6P, Dual X5670 2.93 Westmere Win7 PC w/48gb ram & 4930K w/32gb ram Win7.
bearcatrp is offline   0 Reply With Quote
Old Sep 5, 2009, 11:15 PM   #7
cmaier
macrumors G3
 
Join Date: Jul 2007
Location: California
The time it takes to decompress is small compared to the time saved loading the smaller executable into memory.
cmaier is offline   1 Reply With Quote
Old Sep 5, 2009, 11:55 PM   #8
adamk77
macrumors regular
 
Join Date: Jan 2008
Quote:
Originally Posted by Riemann Zeta View Post
I'm not quite sure why they would do this: drive space is so plentiful, it is dirt cheap today. Compression of a file system, however, adds a veritable assload of system overhead, as things have to be dynamically decompressed on every since execution.

Edit: thanks for the little utility brkirch, it was quite useful. It seems that my system folder is yielding a 41.5% compression savings. I don't know if that is a good or bad--we probably won't know until there is more information about the overhead of all this de/re/compressing going on.
The Ars review on Snow Leopard explained it very well --

But compression isn't just about saving disk space. It's also a classic example of trading CPU cycles for decreased I/O latency and bandwidth. Over the past few decades, CPU performance has gotten better (and computing resources more plentifulómore on that later) at a much faster rate than disk performance has increased. Modern hard disk seek times and rotational delays are still measured in milliseconds. In one millisecond, a 2 GHz CPU goes through two million cycles. And then, of course, there's still the actual data transfer time to consider.
__________________
http://www.uchidacoonga.com
"DO IT UNTIL IT BECOMES BORING, THEN KEEP DOING IT UNTIL IT BECOMES BEAUTIFUL."
adamk77 is offline   0 Reply With Quote
Old Sep 6, 2009, 12:50 AM   #9
Catfish_Man
macrumors 68030
 
Catfish_Man's Avatar
 
Join Date: Sep 2001
Location: Portland, OR
Send a message via AIM to Catfish_Man
Disk io is pretty much the worst thing for performance. Unusual tradeoffs start making sense to avoid it.
Catfish_Man is offline   0 Reply With Quote
Old Sep 6, 2009, 05:43 AM   #10
maflynn
Moderator
 
maflynn's Avatar
 
Join Date: May 2009
Location: Boston
As Adam points out, the Ars article really gives you the details in how it it works. Part wonderful design, part kludge for using the metadata.
__________________
~Mike Flynn
maflynn is offline   0 Reply With Quote
Old Sep 6, 2009, 08:43 AM   #11
thisguy23
macrumors newbie
 
Join Date: Nov 2008
Location: VB, VA
Good article, however i must have skipped over it but what is the cmd line for compressing apps and their contents to save this disk space?
__________________
15'' MBP 2.33GHz, 3GB RAM, 320GB Scropio Black
13'' UMB 2.4GHz, 4GB RAM, 500GB Seagate Momentus
Original 1998 iMac.. oh the classics
2 Black 16 GB iPhone 3G's
thisguy23 is offline   0 Reply With Quote
Old Sep 6, 2009, 10:18 AM   #12
bearcatrp
macrumors 68000
 
Join Date: Sep 2008
Location: Boon Docks USA
The draw back from using compressed files that require decompressing/compressing when used is the wear and tear on hard drives/SSD drives, generates more heat as your making the processor's and memory modules work more so basically working your computer more. But I would assume by keeping the program open all the time, this would reduce this affect (unless you power off your system). I am all for reducing size of programs and OS but not at the expense of more wear and tear on my system.
__________________
2010 2.66 Mac Mini(SL), ATV3, iPhone 6P, Dual X5670 2.93 Westmere Win7 PC w/48gb ram & 4930K w/32gb ram Win7.
bearcatrp is offline   0 Reply With Quote
Old Sep 6, 2009, 10:32 AM   #13
cmaier
macrumors G3
 
Join Date: Jul 2007
Location: California
Quote:
Originally Posted by bearcatrp View Post
The draw back from using compressed files that require decompressing/compressing when used is the wear and tear on hard drives/SSD drives, generates more heat as your making the processor's and memory modules work more so basically working your computer more. But I would assume by keeping the program open all the time, this would reduce this affect (unless you power off your system). I am all for reducing size of programs and OS but not at the expense of more wear and tear on my system.
There would be less wear and tear on hard drives/SSD drives. Less data is read from them (a compressed executable). The decompression happens into RAM, not to the disk (decompressing to disk would defeat the purpose).

The memory modules don't work any harder - whether you compress or not, the full uncompressed version is stored in RAM. (yeah, during decompression you'll have some intermediates stored in RAM).

Your CPU works a little harder, but decompression isn't what we call "heavy lifting," and it won't have any noticeable effect on the lifetime of the CPU. Since the CPU would be idle for milliseconds while waiting for all the uncompressed executable to load, why not spend much less time loading it and give your CPU a little work to do in the mean time.
__________________
applenews.zone
Discussion of microprocessors and other technology, and reviews of tech products.
cmaier is offline   0 Reply With Quote
Old Sep 6, 2009, 10:56 AM   #14
fishmoose
macrumors 68000
 
Join Date: Jul 2008
Location: Sweden
Quote:
Originally Posted by Riemann Zeta View Post
I'm not quite sure why they would do this: drive space is so plentiful, it is dirt cheap today. Compression of a file system, however, adds a veritable assload of system overhead, as things have to be dynamically decompressed on every since execution.
Maybe the iTablet wont have a big hard drive
fishmoose is offline   0 Reply With Quote
Old Sep 7, 2009, 11:27 PM   #15
brkirch
Thread Starter
macrumors regular
 
Join Date: Oct 2001
I updated the afsctool to report more size information when the -v option is used, download the updated version from the link in the first post of this topic.

Quote:
Originally Posted by thisguy23 View Post
Good article, however i must have skipped over it but what is the cmd line for compressing apps and their contents to save this disk space?
The command line utility ditto can be used to apply HFS+ compression. First you need to make a copy of the file or folder from the terminal with this command:
Code:
ditto --hfsCompression <src> <dst>
If the copy is successful then you can move the original file or folder to the trash. And then you can rename the HFS+ compressed version to take the place of the original:
Code:
mv <dst> <src>
brkirch is offline   0 Reply With Quote
Old Sep 10, 2009, 02:00 AM   #16
brkirch
Thread Starter
macrumors regular
 
Join Date: Oct 2001
I updated afsctool with the ability to apply HFS+ compression to files in place. It usually gets even better compression ratios than ditto. (Edit: I've changed it to use the same compression level as ditto, it should actually now produce identical compressed files unless you specify a different compression level) Just redownload afsctool from the first post if you had downloaded it before to get the updated version.

Last edited by brkirch; Sep 15, 2009 at 10:09 AM.
brkirch is offline   0 Reply With Quote
Old Sep 15, 2009, 02:02 AM   #17
jasonrm
macrumors member
 
Join Date: Jan 2008
Location: Arizona, U.S.A.
Quote:
Originally Posted by brkirch View Post
I updated afsctool with the ability to apply HFS+ compression to files in place. It usually gets even better compression ratios than ditto. Just redownload afsctool from the first post if you had downloaded it before to get the updated version.
Cool idea, however, (there always is a however right?) I can't seem to get in place compression to work...

Code:
jason@macbookpro /Applications$afsctool -v Dropbox.app
Dropbox.app:
Folder contains no compressed files
Total number of files: 250
Total number of folders: 90
Total number of items (number of files + number of folders): 340
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Folder size (compressed): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Compression savings: 0.0%
Appoximate total folder size (files + file overhead + folder overhead): 50614948 bytes / 50.6 MB (megabytes) / 48.3 MiB (mebibytes)

jason@macbookpro /Applications$afsctool -c Dropbox.app
Dropbox.app:
No compressable files in folder

jason@macbookpro /Applications$afsctool -v Dropbox.app
Dropbox.app:
Folder contains no compressed files
Total number of files: 250
Total number of folders: 90
Total number of items (number of files + number of folders): 340
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Folder size (compressed): 49934346 bytes / 50.5 MB (megabytes) / 48.2 MiB (mebibytes)
Compression savings: 0.0%
Appoximate total folder size (files + file overhead + folder overhead): 50614948 bytes / 50.6 MB (megabytes) / 48.3 MiB (mebibytes)
I can use ditto however 'ditto --hfsCompression Dropbox.app Dropbox-new.app' and I get a compression savings of ~75%, and I've tried multiple different applications and folders (however I can compress some files like gifs and tars).

Also, are you using any special build process to compile? I'm getting errors about undefined symbols with just 'gcc afsctool.c -o afsctool', might be something on my end but worth asking.

Very cool idea... thanks for making the utility and code available.
jasonrm is offline   0 Reply With Quote
Old Sep 15, 2009, 10:04 AM   #18
brkirch
Thread Starter
macrumors regular
 
Join Date: Oct 2001
Quote:
Originally Posted by jasonrm View Post
Cool idea, however, (there always is a however right?) I can't seem to get in place compression to work...
Most likely you don't have permission to change the files you are trying to compress, try
Code:
$sudo afsctool -c Dropbox.app
instead of just
Code:
$afsctool -c Dropbox.app
It probably should display an error message about that, so I added one (redownload to get a version that throws tons of permission denied errors at you ).

Quote:
Originally Posted by jasonrm View Post
Also, are you using any special build process to compile? I'm getting errors about undefined symbols with just 'gcc afsctool.c -o afsctool', might be something on my end but worth asking.
Try this command instead:
Code:
gcc -arch x86_64 -arch i386 -lz -o afsctool afsctool.c

Last edited by brkirch; Sep 15, 2009 at 10:10 AM.
brkirch is offline   0 Reply With Quote
Old Sep 15, 2009, 10:42 AM   #19
jasonrm
macrumors member
 
Join Date: Jan 2008
Location: Arizona, U.S.A.
Thanks brkirch. Should have been obvious that I needed the zlib flag (as the errors were related), but it compiles just fine now.

I finally was able to get it working by using absolute paths
Code:
sudo afsctool -c /Applications/Dropbox.app
instead of relative,
Code:
sudo afsctool -c Dropbox.app
Again, thanks.
jasonrm is offline   0 Reply With Quote
Old Sep 15, 2009, 02:35 PM   #20
brkirch
Thread Starter
macrumors regular
 
Join Date: Oct 2001
It should be fixed now, redownload and let me know if you find anything else broken
brkirch is offline   0 Reply With Quote
Old Sep 24, 2009, 05:28 AM   #21
dfs
macrumors member
 
Join Date: Sep 2008
Location: California
Time Machine and compression?

When I was about to upgrade to Snow Leopard I first cloned my Mac to the external disk where I keep my Time Machine files. Then I upgraded. A couple of days later, when I was sure that the upgrade had gone right in every respect (it did) I trashed all the clone files on the external. And got back about fifth gb. of space. Since then, Time Machine backups have been eating up a whole lot less disk space than before, in fact virtually none. I've been puzzled about what the heck is going on. Reading this article, I wonder if this new compression scheme is the answer. Have other people who have made the upgrade and use TM noticed the same phenomenon?
dfs is offline   0 Reply With Quote
Old Oct 7, 2009, 06:40 PM   #22
SurfSpirit
macrumors member
 
Join Date: Mar 2006
Just Amazing!

I have used the afsctool and I recover about 4 Gigabytes already, and I have some folders to run with it yet. My apps seems to open faster, as I have a Macbook I always notice the hard drive was the slower part of the system, Apple has some great ideas. Thank You brkirch for such great app, and cmaier your comment explains HFS compression purpose very well!
SurfSpirit is offline   0 Reply With Quote
Old Oct 8, 2009, 04:36 PM   #23
LordMelkor
macrumors newbie
 
Join Date: Aug 2008
So, is this not supposed to work with images? I ran "afsctool -c -l iPhoto.app" and I got a huge list of images that supposedly weren't compressed, and when I checked it seemed like it only saved a measly 7.1% of space over the original.
LordMelkor is offline   0 Reply With Quote
Old Oct 8, 2009, 04:53 PM   #24
gnasher729
macrumors G5
 
gnasher729's Avatar
 
Join Date: Nov 2005
Quote:
Originally Posted by bearcatrp View Post
f this is the case, then it would indeed decompress and recompress when opening and closing an application. Am I wrong?
The bytes on your hard drive would never be modified, so there is no need to re-compress.
gnasher729 is offline   0 Reply With Quote
Old Oct 9, 2009, 08:37 AM   #25
brkirch
Thread Starter
macrumors regular
 
Join Date: Oct 2001
Quote:
Originally Posted by LordMelkor View Post
So, is this not supposed to work with images? I ran "afsctool -c -l iPhoto.app" and I got a huge list of images that supposedly weren't compressed, and when I checked it seemed like it only saved a measly 7.1% of space over the original.
I just uploaded a new version, download it and then try this:
Code:
$afsctool -d iPhoto.app
$afsctool -ckvl 9 0 0 iPhoto.app
Here's what I get after compression has completed:
Code:
/Applications/iPhoto.app:
Number of HFS+ compressed files: 12722
Total number of files: 12774
Total number of folders: 2613
Total number of items (number of files + number of folders): 15387
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 401452168 bytes / 430.2 MB (megabytes) / 410.3 MiB (mebibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 180544293 bytes / 187.9 MB (megabytes) / 179.2 MiB (mebibytes)
Folder size (compressed): 195774246 bytes / 203.1 MB (megabytes) / 193.7 MiB (mebibytes)
Compression savings: 51.2%
Appoximate total folder size (files + file overhead + folder overhead): 210705429 bytes / 210.7 MB (megabytes) / 200.9 MiB (mebibytes)
The new version has support for uncompressed data blocks (required when the size of a data block will otherwise increase after compression). This means that images or anything else that wouldn't compress before should now (although there won't always be much if any compression savings). The only files that will not compress now (provided the compression setting 9 0 0) are files that have a resource fork, zero length data fork, are over 2 GB in size, or are not regular files (symbolic links, for example).

Note that I used the -k flag when compressing after decompression; you should always do this when recompressing because Apple's HFS+ compression kernel extension has some cache bugs so it may cause some files to appear corrupted if you do not (if you have this problem then resetting your computer or decompressing using afsctool should fix it).

Last edited by brkirch; Oct 9, 2009 at 02:02 PM.
brkirch is offline   0 Reply With Quote

Reply
MacRumors Forums > Apple Systems and Services > OS X

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Similar Threads
thread Thread Starter Forum Replies Last Post
Clusters - HFS+ Compression MrGimper Mac Applications and Mac App Store 0 Sep 29, 2013 11:30 AM
5s - hidden focus feature not working for me lnferno iPhone 1 Sep 22, 2013 03:37 AM
Hidden feature on iPhone 5 Myiphone7 iPhone 38 Apr 4, 2013 03:37 PM
Hidden feature: email to groups Eharel iPhone Tips, Help and Troubleshooting 0 Oct 28, 2012 06:22 PM
mail in snow leopard - hidden address database? toronado455 Mac Applications and Mac App Store 2 Jun 8, 2012 04:58 PM

Forum Jump

All times are GMT -5. The time now is 07:47 PM.

Mac Rumors | Mac | iPhone | iPhone Game Reviews | iPhone Apps

Mobile Version | Fixed | Fluid | Fluid HD
Copyright 2002-2013, MacRumors.com, LLC