macOS Bash Script Not Executing Correctly

[MikeVine]

Besides using my original DiscontinuedSince2007.txt file, do I need to save that as a UTF-8 format text file (DiscontinuedSince2007UTF-8.txt), for the script to do its conversion?

 

[MikeVine]

Attached are the list text files, and the Terminal results. I also included a screen capture of the UTF-8 format used in TextEdit. I'm still not getting the files to move to the destination folder. Here is the script used:

#!/bin/bash -x
SAVEIFS=$IFS
IFS=$(echo -en "\n\b\r")
target="/Users/mvineski/Desktop/All"
destination="/Users/mvineski/Desktop/Discontinued"
filelistUTF16="/Users/mvineski/Desktop/DiscontinuedSince2007.txt"
filelistUTF8="/Users/mvineski/Desktop/DiscontinuedSince2007UTF-8.txt"
textutil -convert txt -encoding UTF-8 -output "${filelistUTF8}" "${filelistUTF16}"
fnames=$(cat "$filelistUTF8")

for i in $fnames; do
  mv "${target}/${i}"* "${destination}/"
#  exit 1
  echo "copying $i"
done
IFS=$SAVEIFS
echo "done"

Here is what I type in Terminal:
chmod 700 /Users/mvineski/Desktop/MoveFiles_WildcardSearch_FromList.sh
/Users/mvineski/Desktop/MoveFiles_WildcardSearch_FromList.sh

Sorry I'm so ignorant about this programming stuff. Mike

 

[chown33]

Looking the Terminal output, there are many error messages like this:

+ mv '/Users/mvineski/Desktop/All/T1392*' /Users/mvineski/Desktop/Discontinued/
mv: rename /Users/mvineski/Desktop/All/T1392* to /Users/mvineski/Desktop/Discontinued/T1392*: No such file or directory

What this is saying is that there's no such file as "/Users/mvineski/Desktop/All/T1392*". In short, it didn't expand the wild-card * because there's no file located in that directory whose name starts with T1392. The same error is repeated for approximately every file in the list.

So look in /Users/mvineski/Desktop/All/ and see if there's a file whose name starts with T1392. If not, then that's the problem.

If there's no folder named /Users/mvineski/Desktop/All/, then that's the problem, and you should change the target in your script.

If the folder on your Desktop whos name is "All" is really an alias, then that's the problem. If that's really the case, i.e. an alias instead of the actual folder, then please state that clearly in a post.

If there is a T1392 file in the Desktop/All folder, then copy and paste this command into Terminal, then copy and paste the complete output into a post here:

ls -ld ~/Desktop/All/T139*

For testing, I recommend making a much shorter list in "DiscontinuedSince2007.txt". Start with something testable and easy to manage visually, like 10 files. It doesn't matter which 10 files they are, as long as the list in the text file is consistent with the actual files in the folder. After you get the script working with 10 files, try it with the full list.

 

[kryten2]

Besides using my original DiscontinuedSince2007.txt file, do I need to save that as a UTF-8 format text file (DiscontinuedSince2007UTF-8.txt), for the script to do its conversion?

No need to save. The textutil command takes care of the conversion from UTF-16 to UTF-8. Look at the tumbnail from terminal, the converted file still has the line terminators : ASCII text, with CRLF line terminators. I tested the script with 1429 files. I attached the output of the script.

 

[MikeVine]

I ran a small test with 10 image files in the target folder and the list contained 5 discontinued files to move. The script moved these image files perfectly to the Discontinued folder.

I then ran the script on the live target folder that contains 3,145 image files and the list contained 1,025 discontinued files. The script moved 124 files to the discontinued folder. The problem is, the script moved files that were not in the list file to the discontinued folder. I've attached the Terminal results, list file, and the files that were moved to the discontinued folder. Here is the script used:

#!/bin/bash -x
SAVEIFS=$IFS
IFS=$(echo -en "\n\b\r")
target="/Users/mvineski/Desktop/All"
destination="/Users/mvineski/Desktop/Discontinued"
filelistUTF16="/Users/mvineski/Desktop/DDsince2007.txt"
filelistUTF8="/Users/mvineski/Desktop/DDsince2007UTF-8.txt"
textutil -convert txt -encoding UTF-8 -output "${filelistUTF8}" "${filelistUTF16}"
fnames=$(cat "$filelistUTF8")

for i in $fnames; do
mv "${target}/${i}"* "${destination}/"
# exit 1
echo "copying $i"
done
IFS=$SAVEIFS
echo "done"

Any ideas? Thanks, Mike

 

[chown33]

There are at least two separate problems here.


The first problem is apparently missing files.

These apparently missing files are ones whose names are listed in the "DDsince2007" file, but which don't appear in the Desktop/All folder. The script as given is not capable of handling these.

If you actually have files like this, i.e. listed in "DDsince2007" but not present in Desktop/All, then the script will need to be modified to handle this.

First, however, I want to absolutely confirm that the files not being moved are really missing from Desktop/All. Please copy and paste this command into Terminal, then copy and paste the complete output into a post here:

ls -ld ~/Desktop/All/T1400*

There are at least 4 files named in "DDsince2007" that should match this wildcard pattern.


The second problem is a programming logic bug in the script.

The * wildcard is matching multiple files (this is correct behavior for *, so it's a programmer bug). All matching files are then moved. Here's an example from the Terminal output:

+ for i in '$fnames'
+ mv '/Users/mvineski/Desktop/All/T18001 RedCal16p.jpg' '/Users/mvineski/Desktop/All/T18002 YellowCal16p.jpg' '/Users/mvineski/Desktop/All/T18003 WhiteCal16p.jpg' '/Users/mvineski/Desktop/All/T18004 BlueCal16p.jpg' '/Users/mvineski/Desktop/All/T18005 GreenCal16p.jpg' '/Users/mvineski/Desktop/All/T18006 PurpleCal16p.jpg' '/Users/mvineski/Desktop/All/T18007 OrangeCal16p.jpg' '/Users/mvineski/Desktop/All/T18008 DeepPinkCal16p.jpg' '/Users/mvineski/Desktop/All/T18009 RainbowPlaid16p.jpg' /Users/mvineski/Desktop/Discontinued/
+ echo 'copying T1800'

We can see from this that the token read from the list-file (DDsince2007) is "T1800" (last line above). We can also see that the expansion of T1800* matches multiple actual files, all of which are moved.

As it turns out, T18001 is the next token from the list-file, but its corresponding real file has already been moved, so that attempt to move fails.

A correct wildcard pattern must match only ONE file. In order to come up with that pattern, I need the answer to this question:

Do all the filenames have a space after the Tnumber, and before the remainder of the filename?
​

 

[MikeVine]

Yes, all the filenames have a space after the Tnumber, and before the remainder of the filename.
The DDsince2007.txt list does have extra Tnumbers listed within it. It is possible that not all the image files in the "All" folder will be contained in the list text file.

 

[chown33]

Thanks for the info.

Also, please copy and paste this command into Terminal, then copy and paste the complete output into a post here:

ls -ld ~/Desktop/All/T1400*

This wasn't an idle request. I really did want to see the actual output from the actual command.

 

[kryten2]

Can you also post the output of the entire /Users/mvineski/Desktop/All directory for testing? Thanks.

ls ~/Desktop/All/ > lsoutputalldir.txt

 

[MikeVine]

kryten2, chown33,
Attached are your command line results from Terminal.
Mike

 

[kryten2]

It's the lsoutputalldir.txt file that I want. It should be in your /Users/mvineski folder. In Finder-->places-->mvineski. Is that the live target folder that contains the 3,145 image files? Thanks

 

[MikeVine]

Sorry about that. Please see attached.

 

[chown33]

First, there are T-numbers listed in the Discontinued list whose actual filenames do not adhere to the "space after number" pattern. One example is T92918, whose actual filename is simply "T92918.jpg". Another example is "T92620_1_H20p.jpg", which contains no spaces.

These deviations made it harder to get the wildcard pattern correct, but the following should work.

#!/bin/bash -x
SAVEIFS=$IFS
IFS=$(echo -en "\n\b\r")
target="/Users/mvineski/Desktop/All"
destination="/Users/mvineski/Desktop/Discontinued"

list="/Users/mvineski/Desktop/DDsince2007.txt"

fnames=$(textutil -convert txt -stdout "$list")

pat="[\ _.]*jpg"

for i in $fnames; do
  echo "trying $i"
  [ -f "$target/$i"$pat ] && mv "$target/$i"$pat "$destination/" && echo "$i moved"
done

IFS=$SAVEIFS
echo "done"

 

[MikeVine]

You programmers rock! The script executed perfectly. I want to thank all of you who contributed to this script and being so patient with me. This will save me loads of time in the future. Thanks again!