Disk usage mismatch: disk full, yet I'm only using 70%

mcvos · Apr 27, 2014

My Mac regularly complains that my startup disk is full. Originally, the problem seemed to be that /var/vm/ collected a crazy amount of swapfiles, which can apparently be preventing by typing `sudo dynamic_pager -L 1073741824` every once in a while.

Now, however, that has stopped working. According to some graphical disk usage utility (CleanGenius), I'm using 85 GB out of my 120 GB SSD, and yet Finder claims I've only got 500 MB free. What could be the problem here?

Some relevant stuff:

Code:

> sudo du -hxd1 /
 75M	/.DocumentRevisions-V100
113M	/.fseventsd
1.0G	/.Spotlight-V100
  0B	/.Trashes
  0B	/.vol
 12G	/Applications
4.1M	/bin
  0B	/cores
5.0K	/dev
1.0K	/home
5.4G	/Library
1.0K	/net
  0B	/Network
9.1G	/private
2.3M	/sbin
3.3G	/System
 48G	/Users
2.4G	/usr
5.0G	/Volumes
 86G	/

86 GB? Sounds like I should have plenty available. So CleanGenius wasn't wrong.

Code:

> ls -lkia /Volumes
total 5242884
   10723 drwxrwxrwt@  4 root  admin         136 Apr 27 13:13 .
       2 drwxr-xr-x  30 root  wheel        1088 Mar  1 03:26 ..
35278630 lrwxr-xr-x   1 root  admin           1 Feb 28 22:55 Mac OS X System -> /
       1 drwxrwxr--@  1 mcv   staff  5368709120 Apr 27 13:30 WualaDrive

Is that 5 GB for Wuala? That's a lot (considering it's empty), but doesn't explain the mismatch.

What else can I try?

Edit:

Code:

> df -h
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2   111Gi  110Gi  889Mi   100%    /
devfs          202Ki  202Ki    0Bi   100%    /dev
map -hosts       0Bi    0Bi    0Bi   100%    /net
map auto_home    0Bi    0Bi    0Bi   100%    /home
Wuala          5.0Gi    0Bi  5.0Gi     0%    /Volumes/WualaDrive

So df (which I assume is similar to what Finder does) says my disk is full, while du (which is presumably what CleanGenius uses) says it's not full. Why that discrepancy?

maflynn · Apr 27, 2014

Download and use OmniDiskSweeper. It will provide a sorted list of what's consuming your space. I'd prefer this over the terminal commands.

subsonix · Apr 27, 2014

I would also question why you trust what one 3rd party application says over Finder.

McGiord · Apr 27, 2014

Check if TimeMachine is ON and you haven't done a backup recently it creates a local snapshot and takes space out of your drive.
You can disable the local snapshots also.
Also run the maintenance scripts in the terminal (sorry if those programs you use perform it), I Don't remember them so google them.

mcvos · Apr 27, 2014

maflynn said:
Download and use OmniDiskSweeper. It will provide a sorted list of what's consuming your space. I'd prefer this over the terminal commands.

Is OmniDiskSweeper more thorough than du? Because even du says I've got space free.

McGiord said:
Check if TimeMachine is ON and you haven't done a backup recently it creates a local snapshot and takes space out of your drive.
You can disable the local snapshots also.
Also run the maintenance scripts in the terminal (sorry if those programs you use perform it), I Don't remember them so google them.

TimeMachine says it's off. Yeah, I know, I should be backing up regularly, but in this particular case, it doesn't seem to explain the missing space.

Where would TimeMachine store those snapshots?

Is there anything else that can take up space in a way that's invisible to du?

maflynn · Apr 27, 2014

mcvos said:
Is OmniDiskSweeper more thorough than du? Because even du says I've got space free.

I prefer OmnidiskSweeper because of how the information is presented. Its easier to see where the space is, and drill down if needed.

chown33 · Apr 27, 2014

Run Disk Utility.app (it's in Utilities).

Select your startup disk, then click "Verify Disk".
(Note: "Verify Disk", not "Verify Permissions".)

When it finishes, copy and paste the output in a post here.

If it finds any errors, you may want to do a "Repair Disk", but only do this AFTER doing a backup. And unless you're experienced in interpreting disk errors, you might want to postpone that until after confirming the findings here.

The reason for doing a backup is that the cause of the errors may be a damaged file-system structure that leads to further file problems. Or the cause may be a slowly failing disk drive that SMART Status doesn't detect. Or the cause may have been a one-time event that won't recur. Safety first, unless you don't value your data.

For the file-system savvy, it's possible to have unused blocks that aren't in any file, nor are they correctly represented in the free-space pool. The result is that counting blocks assigned to files ('du') adds up to a different number than the totals stored for free-space ('df'). Basically, 'df' and 'du' look at different things to arrive at their results.

McGiord · Apr 27, 2014

Run these: http://www.thexlab.com/faqs/maintscripts.html

About the snapshots:
http://pondini.org/TM/30.html

mcvos · Apr 27, 2014

maflynn said:
I prefer OmnidiskSweeper because of how the information is presented. Its easier to see where the space is, and drill down if needed.

I like CleanGenius for the same reason. I'll have a look at OmniDiskSweeper, but it sounds like it'll probably give the same result as du or CleanGenius.

chown33 said:
Run Disk Utility.app (it's in Utilities).

Select your startup disk, then click "Verify Disk".
(Note: "Verify Disk", not "Verify Permissions".)

When it finishes, copy and paste the output in a post here.

Verifying volume Mac OS X System
Checking file systemPerforming live verification.
Checking Journaled HFS Plus volume.
Checking extents overflow file.
Checking catalog file.
Checking multi-linked files.
Checking catalog hierarchy.
Checking extended attributes file.
Checking volume bitmap.
Checking volume information.
The volume Mac OS X System appears to be OK.

Nothing wrong, it seems. And yet the totals from du and df don't match.

If it finds any errors, you may want to do a "Repair Disk", but only do this AFTER doing a backup. And unless you're experienced in interpreting disk errors, you might want to postpone that until after confirming the findings here.

There's nothing to repair, it seems. Fortunately? Unfortunately? No idea. Just finding a clear cause for this mysterious mismatch would be very welcome.

For the file-system savvy, it's possible to have unused blocks that aren't in any file, nor are they correctly represented in the free-space pool. The result is that counting blocks assigned to files ('du') adds up to a different number than the totals stored for free-space ('df'). Basically, 'df' and 'du' look at different things to arrive at their results.

Thanks, that does explain something. But should "verify disk" detect those blocks? It doesn't. Does that make sense? How would I be able to get access to those blocks?

And if this isn't it, then what other explanation can there be for the mismatch?

chown33 · Apr 27, 2014

mcvos said:
Thanks, that does explain something. But should "verify disk" detect those blocks? It doesn't. Does that make sense? How would I be able to get access to those blocks?

Yes, "Verify Disk" should detect missing blocks.

And if this isn't it, then what other explanation can there be for the mismatch?

Off the top of my head, I don't know.

chown33 · Apr 27, 2014

Going back to your original post, something odd is going on. I'm going to snip and hilite to point out the things that seem odd to me.

Code:

sudo du -hxd1 /
 ..snip..
[B]5.0G	/Volumes[/B]
 86G	/
..snip..

> df -h
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2   111Gi  110Gi  889Mi   100%    /
devfs          202Ki  202Ki    0Bi   100%    /dev
map -hosts       0Bi    0Bi    0Bi   100%    /net
map auto_home    0Bi    0Bi    0Bi   100%    /home
[B]Wuala          5.0Gi    0Bi  5.0Gi     0%    /Volumes/WualaDrive[/B]

First, the posted 'du' command has the -x option. This should prevent 'du' from crossing into any other mounted file-systems. In particular, it should prevent calculating or reporting disk-usage for any external drives, secondary partitions, mounted disk-images, etc.

Second, the report from 'df' shows there's a file-system named "Wuala" mounted on "/Volumes/Wualadrive". Its reported size is identical to du's calculated size for "/Volumes". Furthermore, the 'ls' command shows nothing unexplained in "/Volumes": both the symlink to "/" and a directory named "WualaDrive" are expected.

The odd thing is that -x is supposed to prevent 'du' from crossing into another mounted file-system. So why is "/Volumes/WualaDrive" being counted by du? For now, consider that a question for further investigation.

When I google Wuala, I learn that it's a cloud-storage file-system. This may provide a basis for an explanation, because any cloud-storage file-system also maintains a locally-stored component that consumes local disk space.

I suggest unmounting (ejecting) WualaDrive, and running the same 'du' command as before, along with the same 'df' command, and posting the results.

It's impossible for me to say exactly what WualaDrive is doing, and I'd rather not guess. So the simplest thing is to take it out of the picture, rerun the tests, and see what the results are.

Some of your overall SSD space will be consumed by partitioning information, a recovery partition, and an EFI partition. None of those will normally appear in Disk Utility, nor do they appear as mounted file-systems, because they're not mounted. If a file-system isn't mounted, 'df' has no way to show it, and neither will 'du'.

To learn the sizes of all the partitions, use this Terminal command:

Code:

diskutil list

Post the complete output. Please use CODE tags, so the blank-padding works.

mcvos · Apr 27, 2014

Yeah, Wuala is cloud storage, similar to Dropbox, except that it encrypts everything before it's transmitted. Not sure why it needs its own filesystem, but I guess there's a good reason for that.

With Wuala turned off, I get:

Code:

> df -h
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2   111Gi  110Gi  923Mi   100%    /
devfs          202Ki  202Ki    0Bi   100%    /dev
map -hosts       0Bi    0Bi    0Bi   100%    /net
map auto_home    0Bi    0Bi    0Bi   100%    /home

Code:

> ls -lkia /Volumes
total 4
   10723 drwxrwxrwt@  3 root  admin   102 Apr 28 08:20 .
       2 drwxr-xr-x  30 root  wheel  1088 Mar  1 03:26 ..
35278630 lrwxr-xr-x   1 root  admin     1 Feb 28 22:55 Mac OS X System -> /

Code:

> sudo du -hxd1 /
Password:
 75M	/.DocumentRevisions-V100
113M	/.fseventsd
1011M	/.Spotlight-V100
  0B	/.Trashes
  0B	/.vol
 12G	/Applications
4.1M	/bin
  0B	/cores
5.0K	/dev
1.0K	/home
5.4G	/Library
1.0K	/net
  0B	/Network
9.1G	/private
2.3M	/sbin
3.3G	/System
 48G	/Users
2.4G	/usr
4.0K	/Volumes
 81G	/

Still the same mismatch between df and du, although du now reports 5 GB less now that Wuala is turned off, while df doesn't report 5 GB freed. That is certainly somewhat mysterious.

chown33 · Apr 28, 2014

mcvos said:
Yeah, Wuala is cload storage, similar to Dropbox, except that it encrypts everything before it's transmitted. Not sure why it needs its own filesystem, but I guess there's a good reason for that.

Maybe it's so it can detect changes on the local storage, so it knows what to sync without having to scan everything. If you're a programmer, look up the FSEvents API:
http://en.wikipedia.org/wiki/FSEvents
https://developer.apple.com/library...ference/FSEvents_Ref/Reference/reference.html

Still the same mismatch between df and du, although du now reports 5 GB less now that Wuala is turned off, while df doesn't report 5 GB freed. That is certainly somewhat mysterious.

Suppose that Wuala has a locally stored disk-image. That will imply a disk-image file stored somewhere on the startup disk. That further implies it will show up as an allocated file of a particular size in 'du', and will also appear as a file-system of that size in 'df' when it's mounted. Unmounting the disk-image will remove the storage from 'df' (file-system disappears), but not from 'du' (disk-image file remains intact).

I'm not saying this is definitely happening, but it's plausible, it fits with how cloud storage is usually done, and it precisely accounts for the observed results. You could use one of the graphical utilities for examining disk-space consumption, and look for large files about 5Gi in size. However, local storage might be a sparse bundle, so it might not show up as a single large file, but as a directory with a bunch of 8MB files:
http://en.wikipedia.org/wiki/Sparse_image

However, I still don't have a plausible explanation for disparity in the other sizes. I have to run some tests, so I won't be able to post any results for several hours (let's say 8, just to cover things).

Meanwhile, please post the output of this command:

Code:

diskutil list

Alrescha · Apr 28, 2014

For what it is worth, there are times when some disk tools may not see (and therefore not count) an open file. If you have not restarted your Mac as part of debugging, it is worth a try. If this is happening you will either get your space back or the file should become visible to tools after a restart.

A.

mcvos · Apr 28, 2014

chown33 said:
Suppose that Wuala has a locally stored disk-image. That will imply a disk-image file stored somewhere on the startup disk. That further implies it will show up as an allocated file of a particular size in 'du', and will also appear as a file-system of that size in 'df' when it's mounted. Unmounting the disk-image will remove the storage from 'df' (file-system disappears), but not from 'du' (disk-image file remains intact).

But the exact opposite happened in my case: After quitting Wuala, du didn't show it anymore, but df was unchanged.

I don't think it matters much though; Wuala (5GB) is much smaller than the space I'm missing (~35GB).

Code:
Meanwhile, please post the output of this command:

Code:

diskutil list

Code:

> diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *120.0 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS Mac OS X System         119.2 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     Apple_partition_scheme                        *18.8 MB    disk1
   1:        Apple_partition_map                         32.3 KB    disk1s1
   2:                  Apple_HFS Flash Player            18.8 MB    disk1s2
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     Apple_partition_scheme                        *16.1 MB    disk2
   1:        Apple_partition_map                         32.3 KB    disk2s1
   2:                  Apple_HFS Flash Player            16.1 MB    disk2s2
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     Apple_partition_scheme                        *16.1 MB    disk3
   1:        Apple_partition_map                         32.3 KB    disk3s1
   2:                  Apple_HFS Flash Player            16.1 MB    disk3s2

/dev/disk0 looks like my SSD. Not sure what the others are, but again they're too small to add up to 35GB.

Alrescha said:
For what it is worth, there are times when some disk tools may not see (and therefore not count) a new file still open for write. If you have not restarted your Mac as part of debugging, it is worth a try. If this is happening you will either get your space back or the file should become visible to tools after a restart.

Reboot? Now there's an idea. I hardly ever reboot. My Macbook is my mobile workstation, and I've always got way too much stuff open that I don't want to close just yet. And so far, it has always been stable enough to handle it. But if a lack of booting could mean that 35GB of free disk space is not recognized as such by `df`, then I guess I'd better give that a try. I'll report back when I've found the time and opportunity to do that.

chown33 · Apr 28, 2014

mcvos said:
But the exact opposite happened in my case: After quitting Wuala, du didn't show it anymore, but df was unchanged.

Sorry about that. My use of df and du got confused.

I did some more testing, and couldn't come up with anywhere the space could have gone.

My first thought was resource-forks. But in my test, those were correctly counted by 'du'.

My other thought was xattr's. It turns out that 'du' DOESN'T count those, but I don't think it's likely that xattr's could have consumed that much space, unless there's something completely out of control.

mcvos · Apr 29, 2014

chown33 said:
My other thought was xattr's. It turns out that 'du' DOESN'T count those, but I don't think it's likely that xattr's could have consumed that much space, unless there's something completely out of control.

What are xattrs? Is there any way I can check if they're the problem?

chown33 · Apr 29, 2014

mcvos said:
What are xattrs? Is there any way I can check if they're the problem?

http://en.wikipedia.org/wiki/Extended_file_attributes

If you haven't rebooted yet, do that first. Checking all the xattr's won't be simple. So do the easy thing first and see if it fixes the problem.

chrfr · Apr 29, 2014

mcvos said:
But if a lack of booting could mean that 35GB of free disk space is not recognized as such by `df`, then I guess I'd better give that a try. I'll report back when I've found the time and opportunity to do that.

The fact that you have 3 Flash Player update images mounted tells me that the computer hasn't been restarted in a long time. Rebooting is the first logical place to start with any troubleshooting.

mcvos · Apr 30, 2014

What do you know. Rebooting fixed the problem. I guess from now on I need to reboot every time I run out of disk space.

It would be nice if there was a way to reset whatever rebooting resets, without having to reboot. I grew up with unix machines that never really needed a reboot, though I guess it's unreasonable to expect that kind of stability from a laptop.

Alrescha · Apr 30, 2014

mcvos said:
I grew up with unix machines that never really needed a reboot, though I guess it's unreasonable to expect that kind of stability from a laptop.

Your laptop is a Unix machine:

https://en.wikipedia.org/wiki/Single_UNIX_Specification

and OS X is very stable:

Code:

$ uptime
 9:45  up 146 days, 14:45, 2 users, load averages: 0.00 0.07 0.08

But you may have applications which are not so well-behaved.

A.

mcvos · Apr 30, 2014

Alrescha said:
Your laptop is a Unix machine:

Which is why I was blindly assuming similar stability. But of course a laptop is a very different use case than a server.

Code:
and OS X is very stable:

Code:

$ uptime 9:45 up 146 days, 14:45, 2 users, load averages: 0.00 0.07 0.08

But you may have applications which are not so well-behaved.

On a unix system, that shouldn't matter. It should be able to clean up the mess left by applications without needing a reboot. Clearly something eats disk space here and doesn't release it properly.

mfram · Apr 30, 2014

mcvos said:
On a unix system, that shouldn't matter. It should be able to clean up the mess left by applications without needing a reboot. Clearly something eats disk space here and doesn't release it properly.

It is possible for a process to keep a large file opened that has been deleted. 'du' wouldn't see the file because it's no longer in a directory. But the space can't be freed because the file is still opened. I'm not saying that's what happened in your case, but that it is possible. A reboot would obviously fix that situation.

Alrescha · Apr 30, 2014

mcvos said:
On a unix system, that shouldn't matter. It should be able to clean up the mess left by applications without needing a reboot. Clearly something eats disk space here and doesn't release it properly.

I first encountered the problem I described on a Unix system ca. 1990. No magic pill has been invented that allows the OS make poor applications suddenly good. Your desire is understandable, but I do not think it is realistic; how is the OS supposed to know that application X is using more disk space than it should? Or CPU?

If the problem is the one I described, there are tools (lsof, for instance) that will let you poke around and find open files and the applications that own them. You can then kill off those applications one by one and attempt to find the culprit. This may or may not be a more productive use of your time than rebooting your machine.

I replied mostly because you lay blame at the OS without evidence. I believe an application is the much more likely offender.

A.

addendum: mfram rightly describes another common situation (and in retrospect I think it is more likely than the one I mentioned). The solutions are the same.

Disk usage mismatch: disk full, yet I'm only using 70%

macrumors newbie

macrumors Haswell

macrumors 68040

macrumors 601

macrumors newbie

macrumors Haswell

Moderator

macrumors 601

macrumors newbie

Moderator

Moderator

macrumors newbie

Moderator

macrumors 68020

macrumors newbie

Moderator

macrumors newbie

Moderator

macrumors G5

macrumors newbie

macrumors 68020

macrumors newbie

Contributor

macrumors 68020

Our Staff