Disk usage mismatch: disk full, yet I'm only using 70%

Discussion in 'macOS' started by mcvos, Apr 27, 2014.

  1. mcvos, Apr 27, 2014
    Last edited: Apr 28, 2014

    mcvos macrumors newbie

    Joined:
    Apr 27, 2014
    #1
    My Mac regularly complains that my startup disk is full. Originally, the problem seemed to be that /var/vm/ collected a crazy amount of swapfiles, which can apparently be preventing by typing `sudo dynamic_pager -L 1073741824` every once in a while.

    Now, however, that has stopped working. According to some graphical disk usage utility (CleanGenius), I'm using 85 GB out of my 120 GB SSD, and yet Finder claims I've only got 500 MB free. What could be the problem here?

    Some relevant stuff:
    Code:
    > sudo du -hxd1 /
     75M	/.DocumentRevisions-V100
    113M	/.fseventsd
    1.0G	/.Spotlight-V100
      0B	/.Trashes
      0B	/.vol
     12G	/Applications
    4.1M	/bin
      0B	/cores
    5.0K	/dev
    1.0K	/home
    5.4G	/Library
    1.0K	/net
      0B	/Network
    9.1G	/private
    2.3M	/sbin
    3.3G	/System
     48G	/Users
    2.4G	/usr
    5.0G	/Volumes
     86G	/
    
    86 GB? Sounds like I should have plenty available. So CleanGenius wasn't wrong.

    Code:
    > ls -lkia /Volumes
    total 5242884
       10723 drwxrwxrwt@  4 root  admin         136 Apr 27 13:13 .
           2 drwxr-xr-x  30 root  wheel        1088 Mar  1 03:26 ..
    35278630 lrwxr-xr-x   1 root  admin           1 Feb 28 22:55 Mac OS X System -> /
           1 drwxrwxr--@  1 mcv   staff  5368709120 Apr 27 13:30 WualaDrive
    
    Is that 5 GB for Wuala? That's a lot (considering it's empty), but doesn't explain the mismatch.

    What else can I try?

    Edit:
    Code:
    > df -h
    Filesystem      Size   Used  Avail Capacity  Mounted on
    /dev/disk0s2   111Gi  110Gi  889Mi   100%    /
    devfs          202Ki  202Ki    0Bi   100%    /dev
    map -hosts       0Bi    0Bi    0Bi   100%    /net
    map auto_home    0Bi    0Bi    0Bi   100%    /home
    Wuala          5.0Gi    0Bi  5.0Gi     0%    /Volumes/WualaDrive
    
    So df (which I assume is similar to what Finder does) says my disk is full, while du (which is presumably what CleanGenius uses) says it's not full. Why that discrepancy?
     
  2. maflynn Moderator

    maflynn

    Staff Member

    Joined:
    May 3, 2009
    Location:
    Boston
    #2
    Download and use OmniDiskSweeper. It will provide a sorted list of what's consuming your space. I'd prefer this over the terminal commands.
     
  3. subsonix macrumors 68040

    Joined:
    Feb 2, 2008
    #3
    I would also question why you trust what one 3rd party application says over Finder.
     
  4. McGiord macrumors 601

    McGiord

    Joined:
    Oct 5, 2003
    Location:
    Dark Castle
    #4
    Check if TimeMachine is ON and you haven't done a backup recently it creates a local snapshot and takes space out of your drive.
    You can disable the local snapshots also.
    Also run the maintenance scripts in the terminal (sorry if those programs you use perform it), I Don't remember them so google them.
     
  5. mcvos thread starter macrumors newbie

    Joined:
    Apr 27, 2014
    #5
    Is OmniDiskSweeper more thorough than du? Because even du says I've got space free.

    TimeMachine says it's off. Yeah, I know, I should be backing up regularly, but in this particular case, it doesn't seem to explain the missing space.

    Where would TimeMachine store those snapshots?

    Is there anything else that can take up space in a way that's invisible to du?
     
  6. maflynn Moderator

    maflynn

    Staff Member

    Joined:
    May 3, 2009
    Location:
    Boston
    #6
    I prefer OmnidiskSweeper because of how the information is presented. Its easier to see where the space is, and drill down if needed.
     
  7. chown33 macrumors 604

    Joined:
    Aug 9, 2009
    #7
    Run Disk Utility.app (it's in Utilities).

    Select your startup disk, then click "Verify Disk".
    (Note: "Verify Disk", not "Verify Permissions".)

    When it finishes, copy and paste the output in a post here.

    If it finds any errors, you may want to do a "Repair Disk", but only do this AFTER doing a backup. And unless you're experienced in interpreting disk errors, you might want to postpone that until after confirming the findings here.

    The reason for doing a backup is that the cause of the errors may be a damaged file-system structure that leads to further file problems. Or the cause may be a slowly failing disk drive that SMART Status doesn't detect. Or the cause may have been a one-time event that won't recur. Safety first, unless you don't value your data.

    For the file-system savvy, it's possible to have unused blocks that aren't in any file, nor are they correctly represented in the free-space pool. The result is that counting blocks assigned to files ('du') adds up to a different number than the totals stored for free-space ('df'). Basically, 'df' and 'du' look at different things to arrive at their results.
     
  8. McGiord, Apr 27, 2014
    Last edited: Apr 27, 2014

    McGiord macrumors 601

    McGiord

    Joined:
    Oct 5, 2003
    Location:
    Dark Castle
    #8
    Run these: http://www.thexlab.com/faqs/maintscripts.html

    About the snapshots:
    http://pondini.org/TM/30.html
     
  9. mcvos thread starter macrumors newbie

    Joined:
    Apr 27, 2014
    #9
    I like CleanGenius for the same reason. I'll have a look at OmniDiskSweeper, but it sounds like it'll probably give the same result as du or CleanGenius.

    Verifying volume “Mac OS X System”
    Checking file systemPerforming live verification.
    Checking Journaled HFS Plus volume.
    Checking extents overflow file.
    Checking catalog file.
    Checking multi-linked files.
    Checking catalog hierarchy.
    Checking extended attributes file.
    Checking volume bitmap.
    Checking volume information.
    The volume Mac OS X System appears to be OK.

    Nothing wrong, it seems. And yet the totals from du and df don't match.

    There's nothing to repair, it seems. Fortunately? Unfortunately? No idea. Just finding a clear cause for this mysterious mismatch would be very welcome.

    Thanks, that does explain something. But should "verify disk" detect those blocks? It doesn't. Does that make sense? How would I be able to get access to those blocks?

    And if this isn't it, then what other explanation can there be for the mismatch?
     
  10. chown33 macrumors 604

    Joined:
    Aug 9, 2009
    #10
    Yes, "Verify Disk" should detect missing blocks.

    Off the top of my head, I don't know.
     
  11. chown33 macrumors 604

    Joined:
    Aug 9, 2009
    #11
    Going back to your original post, something odd is going on. I'm going to snip and hilite to point out the things that seem odd to me.

    Code:
    sudo du -hxd1 /
     ..snip..
    [B]5.0G	/Volumes[/B]
     86G	/
    ..snip..
    
    > df -h
    Filesystem      Size   Used  Avail Capacity  Mounted on
    /dev/disk0s2   111Gi  110Gi  889Mi   100%    /
    devfs          202Ki  202Ki    0Bi   100%    /dev
    map -hosts       0Bi    0Bi    0Bi   100%    /net
    map auto_home    0Bi    0Bi    0Bi   100%    /home
    [B]Wuala          5.0Gi    0Bi  5.0Gi     0%    /Volumes/WualaDrive[/B]
    
    First, the posted 'du' command has the -x option. This should prevent 'du' from crossing into any other mounted file-systems. In particular, it should prevent calculating or reporting disk-usage for any external drives, secondary partitions, mounted disk-images, etc.

    Second, the report from 'df' shows there's a file-system named "Wuala" mounted on "/Volumes/Wualadrive". Its reported size is identical to du's calculated size for "/Volumes". Furthermore, the 'ls' command shows nothing unexplained in "/Volumes": both the symlink to "/" and a directory named "WualaDrive" are expected.

    The odd thing is that -x is supposed to prevent 'du' from crossing into another mounted file-system. So why is "/Volumes/WualaDrive" being counted by du? For now, consider that a question for further investigation.

    When I google Wuala, I learn that it's a cloud-storage file-system. This may provide a basis for an explanation, because any cloud-storage file-system also maintains a locally-stored component that consumes local disk space.

    I suggest unmounting (ejecting) WualaDrive, and running the same 'du' command as before, along with the same 'df' command, and posting the results.

    It's impossible for me to say exactly what WualaDrive is doing, and I'd rather not guess. So the simplest thing is to take it out of the picture, rerun the tests, and see what the results are.


    Some of your overall SSD space will be consumed by partitioning information, a recovery partition, and an EFI partition. None of those will normally appear in Disk Utility, nor do they appear as mounted file-systems, because they're not mounted. If a file-system isn't mounted, 'df' has no way to show it, and neither will 'du'.

    To learn the sizes of all the partitions, use this Terminal command:
    Code:
    diskutil list
    
    Post the complete output. Please use CODE tags, so the blank-padding works.
     
  12. mcvos, Apr 27, 2014
    Last edited: Apr 28, 2014

    mcvos thread starter macrumors newbie

    Joined:
    Apr 27, 2014
    #12
    Yeah, Wuala is cloud storage, similar to Dropbox, except that it encrypts everything before it's transmitted. Not sure why it needs its own filesystem, but I guess there's a good reason for that.

    With Wuala turned off, I get:
    Code:
    > df -h
    Filesystem      Size   Used  Avail Capacity  Mounted on
    /dev/disk0s2   111Gi  110Gi  923Mi   100%    /
    devfs          202Ki  202Ki    0Bi   100%    /dev
    map -hosts       0Bi    0Bi    0Bi   100%    /net
    map auto_home    0Bi    0Bi    0Bi   100%    /home
    
    Code:
    > ls -lkia /Volumes
    total 4
       10723 drwxrwxrwt@  3 root  admin   102 Apr 28 08:20 .
           2 drwxr-xr-x  30 root  wheel  1088 Mar  1 03:26 ..
    35278630 lrwxr-xr-x   1 root  admin     1 Feb 28 22:55 Mac OS X System -> /
    
    Code:
    > sudo du -hxd1 /
    Password:
     75M	/.DocumentRevisions-V100
    113M	/.fseventsd
    1011M	/.Spotlight-V100
      0B	/.Trashes
      0B	/.vol
     12G	/Applications
    4.1M	/bin
      0B	/cores
    5.0K	/dev
    1.0K	/home
    5.4G	/Library
    1.0K	/net
      0B	/Network
    9.1G	/private
    2.3M	/sbin
    3.3G	/System
     48G	/Users
    2.4G	/usr
    4.0K	/Volumes
     81G	/
    
    Still the same mismatch between df and du, although du now reports 5 GB less now that Wuala is turned off, while df doesn't report 5 GB freed. That is certainly somewhat mysterious.
     
  13. chown33 macrumors 604

    Joined:
    Aug 9, 2009
    #13
    Maybe it's so it can detect changes on the local storage, so it knows what to sync without having to scan everything. If you're a programmer, look up the FSEvents API:
    http://en.wikipedia.org/wiki/FSEvents
    https://developer.apple.com/library...ference/FSEvents_Ref/Reference/reference.html


    Suppose that Wuala has a locally stored disk-image. That will imply a disk-image file stored somewhere on the startup disk. That further implies it will show up as an allocated file of a particular size in 'du', and will also appear as a file-system of that size in 'df' when it's mounted. Unmounting the disk-image will remove the storage from 'df' (file-system disappears), but not from 'du' (disk-image file remains intact).

    I'm not saying this is definitely happening, but it's plausible, it fits with how cloud storage is usually done, and it precisely accounts for the observed results. You could use one of the graphical utilities for examining disk-space consumption, and look for large files about 5Gi in size. However, local storage might be a sparse bundle, so it might not show up as a single large file, but as a directory with a bunch of 8MB files:
    http://en.wikipedia.org/wiki/Sparse_image


    However, I still don't have a plausible explanation for disparity in the other sizes. I have to run some tests, so I won't be able to post any results for several hours (let's say 8, just to cover things).

    Meanwhile, please post the output of this command:
    Code:
    diskutil list
    
     
  14. Alrescha, Apr 28, 2014
    Last edited: Apr 30, 2014

    Alrescha macrumors 68020

    Joined:
    Jan 1, 2008
    #14
    For what it is worth, there are times when some disk tools may not see (and therefore not count) an open file. If you have not restarted your Mac as part of debugging, it is worth a try. If this is happening you will either get your space back or the file should become visible to tools after a restart.

    A.
     
  15. mcvos thread starter macrumors newbie

    Joined:
    Apr 27, 2014
    #15
    But the exact opposite happened in my case: After quitting Wuala, du didn't show it anymore, but df was unchanged.

    I don't think it matters much though; Wuala (5GB) is much smaller than the space I'm missing (~35GB).

    Code:
    > diskutil list
    /dev/disk0
       #:                       TYPE NAME                    SIZE       IDENTIFIER
       0:      GUID_partition_scheme                        *120.0 GB   disk0
       1:                        EFI                         209.7 MB   disk0s1
       2:                  Apple_HFS Mac OS X System         119.2 GB   disk0s2
       3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
    /dev/disk1
       #:                       TYPE NAME                    SIZE       IDENTIFIER
       0:     Apple_partition_scheme                        *18.8 MB    disk1
       1:        Apple_partition_map                         32.3 KB    disk1s1
       2:                  Apple_HFS Flash Player            18.8 MB    disk1s2
    /dev/disk2
       #:                       TYPE NAME                    SIZE       IDENTIFIER
       0:     Apple_partition_scheme                        *16.1 MB    disk2
       1:        Apple_partition_map                         32.3 KB    disk2s1
       2:                  Apple_HFS Flash Player            16.1 MB    disk2s2
    /dev/disk3
       #:                       TYPE NAME                    SIZE       IDENTIFIER
       0:     Apple_partition_scheme                        *16.1 MB    disk3
       1:        Apple_partition_map                         32.3 KB    disk3s1
       2:                  Apple_HFS Flash Player            16.1 MB    disk3s2
    /dev/disk0 looks like my SSD. Not sure what the others are, but again they're too small to add up to 35GB.

    Reboot? Now there's an idea. I hardly ever reboot. My Macbook is my mobile workstation, and I've always got way too much stuff open that I don't want to close just yet. And so far, it has always been stable enough to handle it. But if a lack of booting could mean that 35GB of free disk space is not recognized as such by `df`, then I guess I'd better give that a try. I'll report back when I've found the time and opportunity to do that.
     
  16. chown33 macrumors 604

    Joined:
    Aug 9, 2009
    #16
    Sorry about that. My use of df and du got confused.


    I did some more testing, and couldn't come up with anywhere the space could have gone.

    My first thought was resource-forks. But in my test, those were correctly counted by 'du'.

    My other thought was xattr's. It turns out that 'du' DOESN'T count those, but I don't think it's likely that xattr's could have consumed that much space, unless there's something completely out of control.
     
  17. mcvos thread starter macrumors newbie

    Joined:
    Apr 27, 2014
    #17
    What are xattrs? Is there any way I can check if they're the problem?
     
  18. chown33 macrumors 604

    Joined:
    Aug 9, 2009
    #18
    http://en.wikipedia.org/wiki/Extended_file_attributes

    If you haven't rebooted yet, do that first. Checking all the xattr's won't be simple. So do the easy thing first and see if it fixes the problem.
     
  19. chrfr macrumors 603

    Joined:
    Jul 11, 2009
    #19
    The fact that you have 3 Flash Player update images mounted tells me that the computer hasn't been restarted in a long time. Rebooting is the first logical place to start with any troubleshooting.
     
  20. mcvos thread starter macrumors newbie

    Joined:
    Apr 27, 2014
    #20
    What do you know. Rebooting fixed the problem. I guess from now on I need to reboot every time I run out of disk space.

    It would be nice if there was a way to reset whatever rebooting resets, without having to reboot. I grew up with unix machines that never really needed a reboot, though I guess it's unreasonable to expect that kind of stability from a laptop.
     
  21. Alrescha macrumors 68020

    Joined:
    Jan 1, 2008
    #21
    Your laptop is a Unix machine:

    https://en.wikipedia.org/wiki/Single_UNIX_Specification

    and OS X is very stable:

    Code:
    $ uptime
     9:45  up 146 days, 14:45, 2 users, load averages: 0.00 0.07 0.08
    
    But you may have applications which are not so well-behaved.

    A.
     
  22. mcvos thread starter macrumors newbie

    Joined:
    Apr 27, 2014
    #22
    Which is why I was blindly assuming similar stability. But of course a laptop is a very different use case than a server.

    On a unix system, that shouldn't matter. It should be able to clean up the mess left by applications without needing a reboot. Clearly something eats disk space here and doesn't release it properly.
     
  23. mfram macrumors 65816

    Joined:
    Jan 23, 2010
    Location:
    San Diego, CA USA
    #23
    It is possible for a process to keep a large file opened that has been deleted. 'du' wouldn't see the file because it's no longer in a directory. But the space can't be freed because the file is still opened. I'm not saying that's what happened in your case, but that it is possible. A reboot would obviously fix that situation.
     
  24. Alrescha macrumors 68020

    Joined:
    Jan 1, 2008
    #24
    I first encountered the problem I described on a Unix system ca. 1990. No magic pill has been invented that allows the OS make poor applications suddenly good. Your desire is understandable, but I do not think it is realistic; how is the OS supposed to know that application X is using more disk space than it should? Or CPU?

    If the problem is the one I described, there are tools (lsof, for instance) that will let you poke around and find open files and the applications that own them. You can then kill off those applications one by one and attempt to find the culprit. This may or may not be a more productive use of your time than rebooting your machine.

    I replied mostly because you lay blame at the OS without evidence. I believe an application is the much more likely offender.

    A.

    addendum: mfram rightly describes another common situation (and in retrospect I think it is more likely than the one I mentioned). The solutions are the same.
     

Share This Page