Bloated Folder Issues

[Apocrathia]

Okay, so this is really weird. I back my iMac up to my Drobo via AFP to my Mac Mini server which is directly attached to the Drobo. Whatever. I leave my music folder out of it, because I'm constantly getting new music and I'd like to be able to have a backup that I can actually access and listen to my music. So here's what I have setup. Here is my crontab:

@daily rsync -a /Users/ianyoung/Music/iTunes/ /Volumes/Network\ Storage/Audio/ &> /dev/null

This rsync tab just backs up my itunes folder to my drobo every night. Okay, so here's a ls -l of the itunes folder on my imac:

drwxr-xr-x  11 ianyoung  staff  374 Jun 21 18:05 iTunes

Now that's the folder whos contents are getting rsynced every night. well here's the folder that it's going to...

drwxr-xr-x+ 11367 ianyoung  staff  60548526 Jun 21 18:05 Audio

I don't know wtf happens, but the filesystem thinks that that folder is so huge. I can't even get into it in the terminal to run an ls. Any suggestions?

 

[Hal Itosis]

Just a quick reply here, but at first glance it appears that since you have omitted the --delete option, rsync is never removing stuff you delete (or rename, or move/rearrange) in the source folder from the backup folder. Try a few test runs (with -nv options too) to see what it will do.

 

[Apocrathia]

Just a quick reply here, but at first glance it appears that since you have omitted the --delete option, rsync is never removing stuff you delete (or rename, or move/rearrange) in the source folder from the backup folder. Try a few test runs (with -nv options too) to see what it will do.

Okay, so here's how bad that folder is. This is the command I started an hour ago...

rsync -aznvv --delete /Users/ianyoung/Music/iTunes/ /Volumes/Network\ Storage/Audio/

An hour later, here's where I'm at...

building file list ... 
done
deleting in .

yeah, i might as well wipe and reload that folder. Using -az --delete should work and keep a 1:1 ratio backup I assume? I really suck at rsync's syntax.

Edit:
I'm also considering running the whole script over compressed ssh. In addition to the -z argument, it'd be compression^2. I'm a bit skeptical in terms of data loss. I know that ssh -C uses the gzip compression algorithm, but I'm not sure what rsync uses.

 

[Apocrathia]

Still unable to delete original folder

Yeah, I let the rm run over the weekend and the folder is still not gone. I can move it to the trash in finder, but then I can't empty the trash. I'm unable to just move the file to /dev/null. any help on a quick way to just delete the damn folder?

 

[Hal Itosis]

Yeah, I let the rm run over the weekend and the folder is still not gone.

What does "not gone" mean?... one or two stubborn files remain inside, or everything that was there to begin with is still there? What was the (exact) syntax of the rm command? Did it spit out any errors along the way? Any messages in system.log (Console.app)?

I can move it to the trash in finder, but then I can't empty the trash.

What (exactly) does happen? Some sort of dialog window i imagine, but what words does it contain? Did you try holding down the option key (which helps deal with locked items) to force empty?

Have you checked this backup disk (or the source disk for that matter) with Disk Utility yet? (could be a damaged directory/catalog or something)

EDIT: Btw, it may be time we knew which OS version *and* which rsync version is under discussion.

 

[Jason Beck]

What does "not gone" mean?... one or two stubborn files remain inside, or everything that was there to begin with is still there? What was the (exact) syntax of the rm command? Did it spit out any errors along the way? Any messages in system.log (Console.app)?



What (exactly) does happen? Some sort of dialog window i imagine, but what words does it contain? Did you try holding down the option key (which helps deal with locked items) to force empty?

Have you checked this backup disk (or the source disk for that matter) with Disk Utility yet? (could be a damaged directory/catalog or something)

I agree when I have a stubborn folder that won't delete from the trash, usually i can force/secure erase it :/

 

[Apocrathia]

To be quite honest, the folder had become so bloated and corrupted that I couldn't even get in to run an ls. What's actually still in there is anybody's guess. I still can't get into the folder and run an ls.

I ran 'rm -rfv folder' in a detached screen over the weekend. started it friday night, came back to it tonight, still going with no output yet.

I'm trying to empty the trash right now, the folder is in the .Trashes folder right now. The dialog is still counting files to delete, but it's at least going up. I'm just going to let that run over the week and hopefully it'll do the job.

is secure empty trash the same as force empty? because there is no force empty option when i hold the option key.

 

[Hal Itosis]

To be quite honest, the folder had become so bloated and corrupted that I couldn't even get in to run an ls. What's actually still in there is anybody's guess. I still can't get into the folder and run an ls.

Sounds like that volume needs repairing with Disk Utility.

I ran 'rm -rfv folder' in a detached screen over the weekend. started it friday night, came back to it tonight, still going with no output yet.

What... -v with no output? So it hasn't deleted anything then. You should see **something** almost immediately with -v... else, you have other problems.

I'm trying to empty the trash right now, the folder is in the .Trashes folder right now. The dialog is still counting files to delete, but it's at least going up.

Sounds like a secure delete... a guaranteed time waster (in this case).

I'm just going to let that run over the week and hopefully it'll do the job.

is secure empty trash the same as force empty? because there is no force empty option when i hold the option key.

No they're not the same. Force empty doesn't add any (significant) text to the menu... it just works more powerfully and without any "Are you sure..." -type of intermediary dialog.


Versions?????

 

[Apocrathia]

Sorry, I had already replied before you posted the version request. The machine is 10.6.4 and here's the rsync --version print out

rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

 

[Hal Itosis]

Okay, thanks.

As i reread all this, it almost seems as if rm and ls and empty trash were all going on concurrently... or at least two of them? Was the folder moved to the trash while rm seemed stalled? idunno. I really think a fresh restart immediately followed by Disk Utility repairs all around would be good to clear the air.

Btw, do ls and rm work perfectly fine everywhere else (other than the weird folder)?

 

[Apocrathia]

Okay, thanks.

As i reread all this, it almost seems as if rm and ls and empty trash were all going on concurrently... or at least two of them? Was the folder moved to the trash while rm seemed stalled? idunno. I really think a fresh restart immediately followed by Disk Utility repairs all around would be good to clear the air.

Btw, do ls and rm work perfectly fine everywhere else (other than the weird folder)?

ls, rm, cp, touch, any sort of file operation works fine anywhere else.. it's one folder that's problematic. None of the operations were run concurrently. I'd try to ls, nothing would happen, cancel, try something else, cancel, rinse and repeat. It's still running the empty trash command right now. I've already run disk utility and the drive checks out fine. So I literally have no clue what is going wrong here. my rsync conjob is setup correctly now and seems to be working like a champ.

here's the rm command that i'm using.

root# rm -rfv Audio2/

^C

It seriously just stalls and goes nowhere. It's pretty bizarre. The empty trash is running right now and seems to still be picking up files. maybe it'll be done by the end of the week, lol.

 

[Hal Itosis]

Might could be some networking-related voodoo then. ::shrug::
Can that Dobro be attached/mounted directly via FW or USB?

[in fact, how did Disk Utility even verify a volume which is networked via afp?]

 

[Apocrathia]

Might could be some networking-related voodoo then. ::shrug::
Can that Dobro be attached/mounted directly via FW or USB?

[in fact, how did Disk Utility even verify a volume which is networked via afp?]

Oh I'm not doing any of this over AFP. I'm ssh'd into the host system that's directly connected via Firewire. I'm running all of the shell commands over ssh and running disk utility through remote desktop. The system that the Drobo is hooked into is a headless mac mini that I just remote into. It's my home server/plex machine.

 

[Hal Itosis]

Oh I'm not doing any of this over AFP. I'm ssh'd into the host system that's directly connected via Firewire. I'm running all of the shell commands over ssh and running disk utility through remote desktop. The system that the Drobo is hooked into is a headless mac mini that I just remote into. It's my home server/plex machine.

>> directly connected via Firewire

Sorry, i got the "AFP" bit from post #1 (and misunderstood the directly attached part somehow).

But okay... so let's bypass all the fancy ssh stuff and just run shell commands as the current console user then. [who uses ssh to access a volume which is already mounted directly... and why? If you want a root shell just type sudo -s]


EDIT: or wait a minute: "headless Mini" <sigh> okay.

Can we not connect the Dobro directly to the iMac for troubleshooting purposes... or put a head+kybd on the Mini? I have a hard time believing that Dobro is trouble-free... or if it is, then there is something funky with the ssh part of your connection (and i'm not all that familiar with the ins and outs of that aspect).

 

[Apocrathia]

>> directly connected via Firewire

Sorry, i got the "AFP" bit from post #1 (and missed the directly mounted part somehow).

But okay... so let's bypass all the fancy ssh stuff and just run shell commands as the current console user then. [who uses ssh to access a volume which is already mounted directly... and why? If you want a root shell just type sudo -s]

The system that the Drobo is connected to has no keyboard on it, so I just ssh in from my iMac. The volume is mounted via AFP on my iMac, and I could run the same commands on my iMac, but then it actually would be running over AFP. sshing into the server is just the most direct way to get to that box. I could pull up a terminal on the server through remote desktop, but that'd be a huge waste.

 

[Hal Itosis]

I could pull up a terminal on the server through remote desktop, but that'd be a huge waste.

Yeah... because a rm -vfr command that just sits there for 24 hours isn't a waste.

 

[Apocrathia]

Yeah... because a rm -vfr command that just sits there for 24 hours isn't a waste.

hah, yeah. the real problem is that i honestly have no clue what it's doing. there are no anomalies in the syslogs or anything. the finder trash emptying is up to 600k files, and I know there are only 20k songs in my music folder. However, since it's been writing over and over and over, who knows how big it really is. The hardware checks out fine. It's sad to think that that folder has just become THAT bad.

 

[Hal Itosis]

However, since it's been writing over and over and over, who knows how big it really is. The hardware checks out fine. It's sad to think that that folder has just become THAT bad.

I'm out of guesses myself. If the Dobro is fine filesystem-wise (and physically), then the borderline behavior would appear to be somewhere along the ssh/afp connection.

I usually use man3 as a benchmark for hefty folders (it contains between 5000 - 7000 items at the top level typically).

[SIZE="1"]
$ [COLOR="Blue"]cd /Volumes/iMacHD/usr/share/man/man3[/COLOR]

$ [COLOR="Blue"]df -h .[/COLOR]
Filesystem                                Size   Used  Avail Capacity  Mounted on
afp_17q2D6000d4Q0000oM0000VU-1.2d00000b   64Gi   25Gi   39Gi    39%    /Volumes/iMacHD

$ [COLOR="Blue"]time ls -l |wc -l[/COLOR]
    5690
[B]
real	0m17.485s[/B]
user	0m0.239s
sys	0m1.026s[/SIZE]

And that is afp-only, not ssh (which i'd expect would yield quicker results).

Is there an OS backup on the Dobro (i.e., one with a man3 folder)?
If so, try that and see what sort of time values your setup produces.

idunno... i have to give up at this point (sorry, ssh is not my forte).
EDIT: i still say direct mounting the Dobro to the iMac would answer a lot of questions.

 

[Apocrathia]

I started running a force empty trash on Sunday night. I checked on the server this morning and this is where I'm at:

It's counted 5 million files in that folder. Now I'm wondering how long it's going to take to delete everything. I can see why I was getting no output from my 'rm -rfv' command in the terminal, there was just that much crap that it had to index before it could delete it.
In other news, the new cron job is working like a champ. I'm glad I was at least able to get that issue solved.

 

[Apocrathia]

Well I logged into my server today, which was running Finder force empty trash and was greeted with this:

Any ideas? I think I'm just going to set a detached screen running rm -rf and leave it for like a month to see what happens. fml.