Finder extremely slow at replacing files

[LosLoco]

Hola peeps!

I'm having this very annoying problem with the Mavericks Finder.

I'm trying to move about 3000 files (of a total size of about 176mb) from one folder to another.
The destination-folder already contain about 150.000 files.
Some of the files, I'm trying to move, is updated version of some of the files in the destination folder. Therefore, the new files should replace the old ones.
Problem: After Finder asks what to do with the duplicate files, I choose "replace". But this process (of replacing old files) sometimes takes several hours.

Moving files without duplicates in the destination-folder takes a couple of seconds (even when it's several thousand small files), so I'm quite sure the "replacing" is the culprit. But surely, it shouldn't take so long?

I've tried a couple of test. If I try to move just 10 files, that have duplicates in the destination-folder, it takes up to 6 minutes to complete... It's faster to locate the 10 old files, delete them and then move the 10 new files in. But this isn't really a viable solution when trying to move 3000 files... :/

Anyone with a solution to this problem?

Thanks in advance !

 

[w0lf]

This doesn't sound normal.

Anything show up in console while moving and replacing?
Are you maybe copying between different drives and doing a copy and replace rather than move and replace?

 

[chown33]

What format is the destination disk? Is it one of the HFS formats (HFS+, HFSX)? One of the FAT formats (FAT-32, ExFAT)? Is it NTFS and you're using a third-party NTFS extension? Is it some other format?


Launch "Disk Utility.app" (in your /Applications/Utilities folder) and perform a "Verify Disk" on the destination disk. Do any errors show up? Post the complete output.


And just to clarify, you're trying to replace the files in a folder that contains about 150 thousand files directly in that folder, as opposed to that number of files in various nested sub-folders.

 

[LosLoco]

This doesn't sound normal.

Anything show up in console while moving and replacing?
Are you maybe copying between different drives and doing a copy and replace rather than move and replace?

I have no idea, haven't checked console Will check next time I get a chance (hopefully soon )

- and no, it's the same drive, and it is indeed moving and replacing..

What's more - this is happening on both my rMPB and my iMac ... :/

What format is the destination disk? Is it one of the HFS formats (HFS+, HFSX)? One of the FAT formats (FAT-32, ExFAT)? Is it NTFS and you're using a third-party NTFS extension? Is it some other format?


Launch "Disk Utility.app" (in your /Applications/Utilities folder) and perform a "Verify Disk" on the destination disk. Do any errors show up? Post the complete output.


And just to clarify, you're trying to replace the files in a folder that contains about 150 thousand files directly in that folder, as opposed to that number of files in various nested sub-folders.

Ehm, the format is standard format mac/osX (I have no clue what it's called )

Did a "Verify Disk" just now, and it says everything is ok (on my rMBP, haven't done it on my iMac, where the probem also exists)

And yes - it's (approximately) 150.000 files in one folder, where I'm trying to replace between 1300-1500 ...

Oh, it's .png-files, by the way. Don't know if that has anything to say (if it's a problem with thumbnails or something...)

 

[chown33]

Ehm, the format is standard format mac/osX (I have no clue what it's called )

Please confirm exactly what format the disk is. It should be displayed in the lower part of Disk Utility's main window, when the volume is selected. Please take a screenshot of that part of the window and post it.

Oh, it's .png-files, by the way. Don't know if that has anything to say (if it's a problem with thumbnails or something...)

That may be significant.


What is the purpose of having so many files at the same level? Is there a specific reason you can't use sub-folders?

A huge number of files can trigger pathological effects, for various possible causes. I would consider anything more than 10 thousand to be "huge". You've exceeded that by 15 times, so I'm not really surprised you're seeing pathological results. The exact cause has yet to be determined, so there may or may not be a work-around. If there isn't an acceptable work-around, then the only solution would be to redistribute the files into sub-folders, thereby avoiding the pathological number of files in a single folder.

 

[w0lf]

I agree with chpwn33, probably has to do with the ridiculous size of your folder. That being said you could probably write a bash script to do the move which could end up working out better than Finder as I'm not sure of the process Finder uses when doing a move and replace.

 

[LosLoco]

Please confirm exactly what format the disk is. It should be displayed in the lower part of Disk Utility's main window, when the volume is selected. Please take a screenshot of that part of the window and post it.



That may be significant.


What is the purpose of having so many files at the same level? Is there a specific reason you can't use sub-folders?

A huge number of files can trigger pathological effects, for various possible causes. I would consider anything more than 10 thousand to be "huge". You've exceeded that by 15 times, so I'm not really surprised you're seeing pathological results. The exact cause has yet to be determined, so there may or may not be a work-around. If there isn't an acceptable work-around, then the only solution would be to redistribute the files into sub-folders, thereby avoiding the pathological number of files in a single folder.

It seems to be HFS+

Making subfolders isn't really an option, as that would make it impossible for me to "easily" replace files, when updated versions is released

The folder contains a facepack for Football Manager

And I know its a crazy lot of files - but before updating to Mavericks, it wasn't a problem at all. The replacing would maybe take a minute or two.

I agree with chpwn33, probably has to do with the ridiculous size of your folder. That being said you could probably write a bash script to do the move which could end up working out better than Finder as I'm not sure of the process Finder uses when doing a move and replace.

Oh, and I have absolutely no idea how to write such a bash script (I've heard so many great things about scripts in osX, but I've never touched it myself )

 

[chown33]

It seems to be HFS+

Good. That suggests it's not JUST due to the huge number of files, but is related to something Finder-specific.

Making subfolders isn't really an option, as that would make it impossible for me to "easily" replace files, when updated versions is released

The folder contains a facepack for Football Manager

I didn't know there were that many players in the NFL. Or even in the NFL and NCAA combined.

And I know its a crazy lot of files - but before updating to Mavericks, it wasn't a problem at all. The replacing would maybe take a minute or two.

Well, that suggests it's something the Finder is doing. I don't know what that might be, and I don't care to guess. You should file a bug-report, though, giving all the details, such as 150k PNG files:
https://bugreport.apple.com/‎


As a work-around, would you be willing to change your workflow?

For example, instead of dropping new files directly into the Big Giant Folder (hereafter, BGF), what if you dropped new files into a Staging-Area Folder (SAF), and then you dropped the SAF onto an Automator workflow app, and that app then moved everything in SAF into BGF. Would that be acceptable?

I'm pretty sure an Automator workflow using a single shell script could do the actual moving and replacing. It won't be a complex shell script, and I'd be willing to write that, and tell you how to make an Automator workflow, if you're willing to test it and report back on how it works (or doesn't).

For the sake of your data, you should probably have a backup, just in case.

 

[LosLoco]

Good. That suggests it's not JUST due to the huge number of files, but is related to something Finder-specific.


I didn't know there were that many players in the NFL. Or even in the NFL and NCAA combined.



Well, that suggests it's something the Finder is doing. I don't know what that might be, and I don't care to guess. You should file a bug-report, though, giving all the details, such as 150k PNG files:
https://bugreport.apple.com/‎


As a work-around, would you be willing to change your workflow?

For example, instead of dropping new files directly into the Big Giant Folder (hereafter, BGF), what if you dropped new files into a Staging-Area Folder (SAF), and then you dropped the SAF onto an Automator workflow app, and that app then moved everything in SAF into BGF. Would that be acceptable?

I'm pretty sure an Automator workflow using a single shell script could do the actual moving and replacing. It won't be a complex shell script, and I'd be willing to write that, and tell you how to make an Automator workflow, if you're willing to test it and report back on how it works (or doesn't).

For the sake of your data, you should probably have a backup, just in case.

It's actually soccer, and it's world wide, so yeah... There are a lot

But I'd be willing to try (almost) anything, if it can help make it any faster

So it's no trouble moving it to another folder first (SAF) before the "BGF"

 

[chown33]

It's actually soccer, and it's world wide, so yeah... There are a lot

But I'd be willing to try (almost) anything, if it can help make it any faster

So it's no trouble moving it to another folder first (SAF) before the "BGF"

OK, I'll post something later today, or it might be Monday.

 

[LosLoco]

OK, I'll post something later today, or it might be Monday.

Lovely Looking forward to trying it out

 

[chown33]

Lovely Looking forward to trying it out

Please post the complete path of your Big Giant Folder. I'll need to incorporate it into the script.

 

[LosLoco]

Please post the complete path of your Big Giant Folder. I'll need to incorporate it into the script.

Hey!

Sorry for the late reply! I've been away for work since monday, and haven't had the slightest thought of checking in here... Bad behavior, sorry :/

The path is

/Users/LosLoco/Documents/Sports Interactive/Football Manager 2014/graphics/faces



Thanks again for helping As it turns out, a new update has just been released, so I can try it out right away, when you've got it ready (if you're still up for it, considering my late reply here )

Best regards

 

[chown33]

Here's the script:

#!/bin/bash
# TEST-ONLY : Ignores any inputs.

## Inputs come only from STAGING dir.
## Only the PNG files in STAGING dir are moved to TARGET dir.

STAGING="$HOME/Documents/Staging"
TARGET="/Users/LosLoco/Documents/Sports Interactive/Football Manager 2014/graphics/faces"


function barf ()
{
  osascript -e 'tell app "System Events"' \
    -e 'activate' \
    -e "display alert \"Stager Failed\" message \"$*\" " \
    -e 'end tell'
  echo "Failed: $*"
  exit 0
}

function testItemIsPNG ()
{
  file -bkI "$1" | grep -Fhi 'image/png' &>/dev/null
}


# Confirm that TARGET is a directory.
# Does not proceed unless dir exists.
[ -d "$TARGET" ] || barf "Must be a directory:\n  $TARGET"

for ITEM in "$STAGING/"* 
do
#  echo item: "$ITEM"
  [ -f "$ITEM" ] || { echo "skip:" "$ITEM"; continue; }

  testItemIsPNG "$ITEM" && mv -fv "$ITEM" "$TARGET"
done

exit 0

Here's how to use it.

1. Launch Automator.
2. When it asks you to choose a template, choose Workflow.
3. At the left, type or paste shell into the Search box.
4. The Run Shell Script action should be the only action remaining in the list.
5. Drag a Run Shell Script action into the big gray area at right (says "Drag actions or files here to build your workflow.")
6. The template will show cat in its command area.
7. Click in the command area, Select All (cmd-A), then paste the entire above script into the command area. IMPORTANT: There must be no remaining cat command.
8. Leave the Shell: popup at /bin/bash
9. Leave the Pass input: popup at its default (doesn't matter yet).
10. Click the Results button below the command area.
11. Save.
12. Run.

Some results should appear, although it will probably indicate everything was skipped.

Now run a test. This uses only files you already have. This is to test two things:
1. That the workflow behaves as expected.
2. That it's faster than using Finder.

Make a folder named Staging in your Documents folder. Duplicate about 20 files from your current Big Giant faces folder, and put them in the Staging folder. Then run the workflow. If it's working, then it should take only a few seconds to complete, and the Results showing in the Automator window should list every item that was skipped or moved. Copy and paste the Results here (it will be a selectable list of lines).

After a successful run, every PNG file in Staging should be in the Big Giant faces folder, and any existing file of the same name should be replaced.

If the test works with 20 files, repeat it with 100 and 500 files. You could use your new face-files, but if something goes wrong, it can end up damaging your current Big Giant faces folder. That's one reason I gave the test using duplicates of files already in faces. If something goes wrong, or it stops part way through, it shouldn't break the existing faces.

Nevertheless, you should make a backup of your Big Giant faces folder, before doing anything else, just to be safe.


This workflow has been written to use only a Staging folder in your Documents folder, and to write only to a specific folder. It can be extended so that it accepts files or folders dropped on it. I chose not to do that yet, because we first need to discover if this will be faster than Finder. If it's not, there's no point in a more complex script that won't solve the problem.