Bloated Folder Issues

Discussion in 'macOS' started by Apocrathia, Jun 24, 2010.

  1. Apocrathia macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #1
    Okay, so this is really weird. I back my iMac up to my Drobo via AFP to my Mac Mini server which is directly attached to the Drobo. Whatever. I leave my music folder out of it, because I'm constantly getting new music and I'd like to be able to have a backup that I can actually access and listen to my music. So here's what I have setup. Here is my crontab:
    Code:
    @daily rsync -a /Users/ianyoung/Music/iTunes/ /Volumes/Network\ Storage/Audio/ &> /dev/null
    This rsync tab just backs up my itunes folder to my drobo every night. Okay, so here's a ls -l of the itunes folder on my imac:
    Code:
    drwxr-xr-x  11 ianyoung  staff  374 Jun 21 18:05 iTunes
    Now that's the folder whos contents are getting rsynced every night. well here's the folder that it's going to...
    Code:
    drwxr-xr-x+ 11367 ianyoung  staff  60548526 Jun 21 18:05 Audio
    I don't know wtf happens, but the filesystem thinks that that folder is so huge. I can't even get into it in the terminal to run an ls. Any suggestions?
     
  2. Hal Itosis macrumors 6502a

    Hal Itosis

    Joined:
    Feb 20, 2010
    #2
    Just a quick reply here, but at first glance it appears that since you have omitted the --delete option, rsync is never removing stuff you delete (or rename, or move/rearrange) in the source folder from the backup folder. Try a few test runs (with -nv options too) to see what it will do.
     
  3. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #3
    Okay, so here's how bad that folder is. This is the command I started an hour ago...
    Code:
    rsync -aznvv --delete /Users/ianyoung/Music/iTunes/ /Volumes/Network\ Storage/Audio/
    An hour later, here's where I'm at...
    Code:
    building file list ... 
    done
    deleting in .
    
    yeah, i might as well wipe and reload that folder. Using -az --delete should work and keep a 1:1 ratio backup I assume? I really suck at rsync's syntax.

    Edit:
    I'm also considering running the whole script over compressed ssh. In addition to the -z argument, it'd be compression^2. I'm a bit skeptical in terms of data loss. I know that ssh -C uses the gzip compression algorithm, but I'm not sure what rsync uses.
     
  4. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #4
    Still unable to delete original folder

    Yeah, I let the rm run over the weekend and the folder is still not gone. I can move it to the trash in finder, but then I can't empty the trash. I'm unable to just move the file to /dev/null. any help on a quick way to just delete the damn folder?
     
  5. Hal Itosis macrumors 6502a

    Hal Itosis

    Joined:
    Feb 20, 2010
    #5
    What does "not gone" mean?... one or two stubborn files remain inside, or everything that was there to begin with is still there? What was the (exact) syntax of the rm command? Did it spit out any errors along the way? Any messages in system.log (Console.app)?


    What (exactly) does happen? Some sort of dialog window i imagine, but what words does it contain? Did you try holding down the option key (which helps deal with locked items) to force empty?

    Have you checked this backup disk (or the source disk for that matter) with Disk Utility yet? (could be a damaged directory/catalog or something)

    EDIT: Btw, it may be time we knew which OS version *and* which rsync version is under discussion.
     
  6. Jason Beck macrumors 68000

    Jason Beck

    Joined:
    Oct 19, 2009
    Location:
    Cedar City, Utah
    #6

    I agree when I have a stubborn folder that won't delete from the trash, usually i can force/secure erase it :/
     
  7. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #7
    To be quite honest, the folder had become so bloated and corrupted that I couldn't even get in to run an ls. What's actually still in there is anybody's guess. I still can't get into the folder and run an ls.

    I ran 'rm -rfv folder' in a detached screen over the weekend. started it friday night, came back to it tonight, still going with no output yet.

    I'm trying to empty the trash right now, the folder is in the .Trashes folder right now. The dialog is still counting files to delete, but it's at least going up. I'm just going to let that run over the week and hopefully it'll do the job.

    is secure empty trash the same as force empty? because there is no force empty option when i hold the option key.
     
  8. Hal Itosis macrumors 6502a

    Hal Itosis

    Joined:
    Feb 20, 2010
    #8
    Sounds like that volume needs repairing with Disk Utility.


    What... -v with no output? So it hasn't deleted anything then. You should see **something** almost immediately with -v... else, you have other problems.


    Sounds like a secure delete... a guaranteed time waster (in this case).


    No they're not the same. Force empty doesn't add any (significant) text to the menu... it just works more powerfully and without any "Are you sure..." -type of intermediary dialog.


    Versions?????
     
  9. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #9
    Sorry, I had already replied before you posted the version request. The machine is 10.6.4 and here's the rsync --version print out
    Code:
    rsync --version
    rsync  version 2.6.9  protocol version 29
    Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
    <http://rsync.samba.org/>
    Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
                  inplace, IPv6, 64-bit system inums, 64-bit internal inums
    
    rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
    are welcome to redistribute it under certain conditions.  See the GNU
    General Public Licence for details.
     
  10. Hal Itosis macrumors 6502a

    Hal Itosis

    Joined:
    Feb 20, 2010
    #10
    Okay, thanks.

    As i reread all this, it almost seems as if rm and ls and empty trash were all going on concurrently... or at least two of them? Was the folder moved to the trash while rm seemed stalled? idunno. I really think a fresh restart immediately followed by Disk Utility repairs all around would be good to clear the air.

    Btw, do ls and rm work perfectly fine everywhere else (other than the weird folder)?
     
  11. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #11
    ls, rm, cp, touch, any sort of file operation works fine anywhere else.. it's one folder that's problematic. None of the operations were run concurrently. I'd try to ls, nothing would happen, cancel, try something else, cancel, rinse and repeat. It's still running the empty trash command right now. I've already run disk utility and the drive checks out fine. So I literally have no clue what is going wrong here. my rsync conjob is setup correctly now and seems to be working like a champ.

    here's the rm command that i'm using.
    Code:
    root# rm -rfv Audio2/
    
    ^C
    It seriously just stalls and goes nowhere. It's pretty bizarre. The empty trash is running right now and seems to still be picking up files. maybe it'll be done by the end of the week, lol.
     
  12. Hal Itosis macrumors 6502a

    Hal Itosis

    Joined:
    Feb 20, 2010
    #12
    Might could be some networking-related voodoo then. ::shrug::
    Can that Dobro be attached/mounted directly via FW or USB?

    [in fact, how did Disk Utility even verify a volume which is networked via afp?]
     
  13. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #13
    Oh I'm not doing any of this over AFP. I'm ssh'd into the host system that's directly connected via Firewire. I'm running all of the shell commands over ssh and running disk utility through remote desktop. The system that the Drobo is hooked into is a headless mac mini that I just remote into. It's my home server/plex machine.
     
  14. Hal Itosis macrumors 6502a

    Hal Itosis

    Joined:
    Feb 20, 2010
    #14
    >> directly connected via Firewire

    Sorry, i got the "AFP" bit from post #1 (and misunderstood the directly attached part somehow).

    But okay... so let's bypass all the fancy ssh stuff and just run shell commands as the current console user then. [who uses ssh to access a volume which is already mounted directly... and why? If you want a root shell just type sudo -s]


    EDIT: or wait a minute: "headless Mini" <sigh> okay.

    Can we not connect the Dobro directly to the iMac for troubleshooting purposes... or put a head+kybd on the Mini? I have a hard time believing that Dobro is trouble-free... or if it is, then there is something funky with the ssh part of your connection (and i'm not all that familiar with the ins and outs of that aspect).
     
  15. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #15
    The system that the Drobo is connected to has no keyboard on it, so I just ssh in from my iMac. The volume is mounted via AFP on my iMac, and I could run the same commands on my iMac, but then it actually would be running over AFP. sshing into the server is just the most direct way to get to that box. I could pull up a terminal on the server through remote desktop, but that'd be a huge waste.
     
  16. Hal Itosis macrumors 6502a

    Hal Itosis

    Joined:
    Feb 20, 2010
    #16
    Yeah... because a rm -vfr command that just sits there for 24 hours isn't a waste. :rolleyes:
     
  17. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #17
    hah, yeah. the real problem is that i honestly have no clue what it's doing. there are no anomalies in the syslogs or anything. the finder trash emptying is up to 600k files, and I know there are only 20k songs in my music folder. However, since it's been writing over and over and over, who knows how big it really is. The hardware checks out fine. It's sad to think that that folder has just become THAT bad.
     
  18. Hal Itosis macrumors 6502a

    Hal Itosis

    Joined:
    Feb 20, 2010
    #18
    I'm out of guesses myself. If the Dobro is fine filesystem-wise (and physically), then the borderline behavior would appear to be somewhere along the ssh/afp connection.

    I usually use man3 as a benchmark for hefty folders (it contains between 5000 - 7000 items at the top level typically).
    Code:
    [SIZE="1"]
    $ [COLOR="Blue"]cd /Volumes/iMacHD/usr/share/man/man3[/COLOR]
    
    $ [COLOR="Blue"]df -h .[/COLOR]
    Filesystem                                Size   Used  Avail Capacity  Mounted on
    afp_17q2D6000d4Q0000oM0000VU-1.2d00000b   64Gi   25Gi   39Gi    39%    /Volumes/iMacHD
    
    $ [COLOR="Blue"]time ls -l |wc -l[/COLOR]
        5690
    [B]
    real	0m17.485s[/B]
    user	0m0.239s
    sys	0m1.026s[/SIZE]
    
    And that is afp-only, not ssh (which i'd expect would yield quicker results).

    Is there an OS backup on the Dobro (i.e., one with a man3 folder)?
    If so, try that and see what sort of time values your setup produces.

    idunno... i have to give up at this point (sorry, ssh is not my forte).
    EDIT: i still say direct mounting the Dobro to the iMac would answer a lot of questions.
     
  19. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #19
    I started running a force empty trash on Sunday night. I checked on the server this morning and this is where I'm at:
    [​IMG]
    It's counted 5 million files in that folder. Now I'm wondering how long it's going to take to delete everything. I can see why I was getting no output from my 'rm -rfv' command in the terminal, there was just that much crap that it had to index before it could delete it.
    In other news, the new cron job is working like a champ. I'm glad I was at least able to get that issue solved.
     
  20. Apocrathia thread starter macrumors 6502

    Joined:
    Jan 6, 2009
    Location:
    University of South Alabama
    #20
    Well I logged into my server today, which was running Finder force empty trash and was greeted with this:
    [​IMG]
    Any ideas? I think I'm just going to set a detached screen running rm -rf and leave it for like a month to see what happens. fml.
     

Share This Page