macOS Bash Script Not Executing Correctly

[MikeVine]

I found this script in an older thread and it's not functioning how I need it to. I'm not a programmer and any help on this would be great. I need to copy files from one location to another based on a list. Here is the script I'm using:

#!/bin/bash -x

target="/Users/mvineski/Desktop/Photoproject/Images"
destination="/Users/mvineski/Desktop/Photoproject/Chapter\ 1"

fnames=$(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt)

for i in $fnames; do
  cp ${target}/${i} ${destination}/
  exit 1
  echo "copying $i"
done
echo "done"

When I run it, here are the results in terminal:

Last login: Wed Apr 25 12:36:26 on console
mvineskis-Mac-Pro:~ mvineski$ #!/bin/bash -x
mvineskis-Mac-Pro:~ mvineski$ 
mvineskis-Mac-Pro:~ mvineski$ target="/Users/mvineski/Desktop/Photoproject/Images"
mvineskis-Mac-Pro:~ mvineski$ destination="/Users/mvineski/Desktop/Photoproject/Chapter\ 1"
mvineskis-Mac-Pro:~ mvineski$ 
mvineskis-Mac-Pro:~ mvineski$ fnames=$(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt)
mvineskis-Mac-Pro:~ mvineski$ 
mvineskis-Mac-Pro:~ mvineski$ for i in $fnames; do
>   cp ${target}/${i} ${destination}/
>   exit 1
>   echo "copying $i"
> done
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
logout

[Process completed]

The problem is the files do not get copied to the destination folder. Help please. Thanks.

 

[robvas]

Are you cutting + pasting to the console, or are you saving it to a file, changing the permissions to executable, and then running it?

nano myscript.sh
edit then save your file
chmod 750 myscript.sh
./myscript.sh

?

 

[MikeVine]

First, I tried cutting and pasting the script from my text editor into a new Terminal window, which didn't work. Then I typed this in Terminal to run the script:

Last login: Wed Apr 25 13:53:32 on ttys000
mvineskis-Mac-Pro:~ mvineski$ chmod 777 /Users/mvineski/Desktop/bash_script.sh
mvineskis-Mac-Pro:~ mvineski$ chmod +x /Users/mvineski/Desktop/bash_script.sh
mvineskis-Mac-Pro:~ mvineski$ /Users/mvineski/Desktop/bash_script.sh
+ target=/Users/mvineski/Desktop/Photoproject/Images
+ destination='/Users/mvineski/Desktop/Photoproject/Chapter\ 1'
++ cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt
+ fnames=$'T089\rT341\rT343'
+ for i in '$fnames'
+ cp $'/Users/mvineski/Desktop/Photoproject/Images/T089\rT341\rT343' '/Users/mvineski/Desktop/Photoproject/Chapter\' 1/
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
+ exit 1
mvineskis-Mac-Pro:~ mvineski$

Still, no files were copied from the target folder to the destination folder.

 

[sero]

you could paste it into terminal like this, but first you need to fix the paths and filenames so that it makes sense

for i in $(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt); do cp "/Users/mvineski/Desktop/Photoproject/Images/${i}" /Users/mvineski/Desktop/Photoproject/Chapter\ 1/; echo "copying $i"; done; echo "done"

 

[balamw]

Do the file names in the file have spaces in them? You may just be missing some double quotes.

B

 

[kryten2]

The cp command didn't succeed as you can see.
Take a look at this :

fnames=$'T089\rT341\rT343'
cp $'/Users/mvineski/Desktop/Photoproject/Images/T089\rT341\rT343'

Those line breaks/endings shouldn't be there. Open Terminal and run this to change the line ending ( a '\r' (Mac) to a '\n' (UNIX) ):

perl -pi -e 's/\r/\n/g' /Users/mvineski/Desktop/Photoproject/Chapter1.txt

What exactly is in that Chapter1.txt file? Can you post a few lines of that or as a zip file?

 

[MikeVine]

Attached is the Chapter1.txt file list which contains this copy:
T089
T341
T343

The actual image file names that need to be copied to the destination folder are as follows:
T089 Kids Choice Sh 50p.jpg
T341 Look Preschool 35p.jpg
T343 Kindergarten 35p.jpg
Thanks.

 

[kryten2]

Your Chapter1.txt file has the wrong line breaks and needs to contain the full file names : T089 Kids Choice Sh 50p.jpg etc. In Terminal :

ls /Users/mvineski/Desktop/Photoproject/Images > /Users/mvineski/Desktop/Photoproject/Chapter1.txt

The script :

#!/bin/bash -x
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
target="/Users/mvineski/Desktop/Photoproject/Images"
destination="/Users/mvineski/Desktop/Photoproject/Chapter 1"

fnames=$(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt)

for i in $fnames; do
  cp ${target}/${i} ${destination}/
  echo "copying $i"
done
IFS=$SAVEIFS
echo "done"

 

[MikeVine]

Is there any way the script could do a wildcard search for just the Tnumber (T089) in the list and copy the image file with the full name (T089 Kids Choice Sh 50p.jpg) to the destination folder? The reason I ask is I only get a list of just the Tnumber and not the full image file name. Thanks everyone for all your help with this script!!

 

[kryten2]

cp ${target}/${i}* ${destination}/

 

[MikeVine]

kryten2,
Your script updates worked perfectly for me. Thank you!
I have one more coding question: instead of copying the files, what would I need to change in the script for moving the files from the target folder to the destination folder?
Again, thanks to everyone for helping me with this script!
Mike

#!/bin/bash -x
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
target="/Users/mvineski/Desktop/Photoproject/Images"
destination="/Users/mvineski/Desktop/Photoproject/Chapter"

fnames=$(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt)

for i in $fnames; do
  cp ${target}/${i}* ${destination}/
  echo "copying $i"
done
IFS=$SAVEIFS
echo "done"

 

[kryten2]

Change line :

cp ${target}/${i}* ${destination}/

to

[B]mv[/B] ${target}/${i}* ${destination}/

There are a lot of BASH shell scripting tutorials to be found on the net. Check these out if you want to learn some basic things. Copying files, moving files etc.

http://tldp.org/LDP/Bash-Beginners-Guide/html/index.html

 

[MikeVine]

Well, I jumped the gun. When I tried the script with the actual list and image files, no files were moved to the destination folder. There are at least 14 image files that should have been moved from the target folder to the destination folder, that are contained in the list.

I've attached the following files in Archive.zip:
TerminalSavedOutput.txt (Results from Terminal)
MoveFiles_WildcardSearch_FromList.sh (script used)
DiscontinuedSince2007.txt (list used)

Any ideas? Thanks, Mike

 

[chown33]

1. Exactly how and where are you obtaining "DiscontinuedSince2007.txt"?
Exactly what program and OS produced the file?

When inspected using a hex editor, I see two non-text bytes 0xFFFE at the start. Also, the entire file appears to be in 16-bit Unicode. Knowing how and where the file came from would clarify that.

In any case, the text will need more filtering than is currently being applied.

Example hex dump of start of file:

hexdump -C DiscontinuedSince2007.txt  | head 

00000000  ff fe 54 00 30 00 39 00  30 00 37 00 37 00 0d 00  |..T.0.9.0.7.7...|
00000010  0a 00 54 00 30 00 39 00  31 00 39 00 34 00 0d 00  |..T.0.9.1.9.4...|
00000020  0a 00 54 00 30 00 39 00  32 00 30 00 32 00 0d 00  |..T.0.9.2.0.2...|
00000030  0a 00 54 00 30 00 39 00  32 00 30 00 34 00 0d 00  |..T.0.9.2.0.4...|
00000040  0a 00 54 00 31 00 30 00  30 00 34 00 31 00 0d 00  |..T.1.0.0.4.1...|
00000050  0a 00 54 00 33 00 31 00  33 00 36 00 0d 00 0a 00  |..T.3.1.3.6.....|
00000060  54 00 33 00 31 00 39 00  30 00 31 00 0d 00 0a 00  |T.3.1.9.0.1.....|
00000070  54 00 33 00 37 00 30 00  30 00 35 00 0d 00 0a 00  |T.3.7.0.0.5.....|
00000080  54 00 33 00 37 00 30 00  30 00 36 00 0d 00 0a 00  |T.3.7.0.0.6.....|
00000090  54 00 33 00 37 00 30 00  31 00 31 00 0d 00 0a 00  |T.3.7.0.1.1.....|

2. In addition to 0xFFFE, there are carriage-returns in the filenames, which appear as \r characters in the Terminal output.

These also need to be stripped out.


3. Please provide a list of the files that should have been moved but weren't, and ones that were actually moved.

In the Terminal output, the first file is clearly not moved, because its attempted name contains the 0xFFFE (which appear as ?? in the output).

On further inspection, it looks to me like every file failed, due to the \r. So according to the Terminal output, NO FILES were moved. There are 1645 instances of "No such file or directory" and 1645 files in the input list.

 

[balamw]

Also, the entire file appears to be in 16-bit Unicode. Knowing how and where the file came from would clarify that.

You beat me to it.

I didn't see this when I opened the file in TextEdit and was able to fix it just by saving it as UTF-8 instead of UTF-16 in TextEdit.

The last one seemed to work OK as there was no \r.

+ mv '/Users/mvineski/Desktop/DVD2012/T72001_T98004/TA67923*' /Users/mvineski/Desktop/DVD2012/zDD/
mv: rename /Users/mvineski/Desktop/DVD2012/T72001_T98004/TA67923* to /Users/mvineski/Desktop/DVD2012/zDD/TA67923*: No such file or directory
+ echo 'copying TA67923'
copying TA67923

B

 

[chown33]

I didn't see this when I opened the file in TextEdit and was able to fix it just by saving it as UTF-8 instead of UTF-16 in TextEdit.

The textutil command should be able to change the encoding in a shell script. I don't have time right now.

Also, adding \r to the value assigned to IFS should let the shell strip \r's as whitespace.

And I recommend this:

mv "${target}/${i}"* "${destination}/" || echo 2>&1 "Didn't copy $i"

I typed that hastily, so it could be wrong. But it also gives the drift: use conditional execution to echo failures.

And note the quoting.

 

[sero]

a bit of a hack but I think it'll work for this instance. see the posts by kryten2, chown33 and balamw if this isn't just a one time thing.

#!/bin/bash
target="/Users/mvineski/Desktop/DVD2012/T72001_T98004"
destination="/Users/mvineski/Desktop/DVD2012/zDD"
list="/Users/mvineski/Desktop/BashScript_MoveFiles/DiscontinuedSince2007.txt"

#fix_list
tr -d '\r' < "$list" >fixed_list && cat fixed_list|sed 's/^[^a-zA-Z0-9][^a-zA-Z0-9]//g' >tmp && mv tmp fixed_list
fnames=$(cat fixed_list)
for i in $fnames; do
  mv "${target}/${i}*" "${destination}/"
  echo "copying $i"
done
rm fixed_list
echo "done"

 

[MikeVine]

Windows users send me the list as a Excel document (.xlsx) containing one column and multiple rows of Tnumbers. I'm on a Mac, 2x2.4GHz Quad-Core Intel Xeon running MacOS10.6.8. I open that .xlsx file in my Mac 2004 version of Excel and Save As a UTF-16 format text file (no UTF-8 format available). I also tried saving the UTF-16 text file to UTF-8 in TextEdit, but that didn't work either.

Here is a partial list of items that need to be moved:
T72309
T72312
T72316
T23322
T72325
T72327
T72329
T72330
T72334
T72335
T72339
T73017
T73012
T1151
No files were moved when I executed the script.

This process is not a one time thing. I either need to get a script to work, or move files manually (what a drag!).

Can you please let me know of a procedure or correct format I should select in Excel for saving out a workable text file list that doesn't include any unnecessary hidden characters that make the script fail? Thanks, Mike

 

[kryten2]

@chown33, @balamw, @sero : Thank you for your feedback.
I saw MikeVine's post, took a quick look at the provided files and yelled it's those carriage-returns again plus an added bonus of two ?? at the start of the txt file. Had to do something else so I couldn't respond immediately. I've attached a clean txt file.

 

[chown33]

I also tried saving the UTF-16 text file to UTF-8 in TextEdit, but that didn't work either.

Please describe exactly what you did, and exactly what happened.

There are many ways to interpret "didn't work". Examples:

1. I tried to Save As in TextEdit, but I see no option to save as UTF-8. Here's a screen shot of the dialog <post screen shot here>.

2. I was able to Save As UTF-8 in TextEdit, and I changed the name of the input file in the script, but even as UTF-8, the shell script still failed (new UTF-8 file, shell script, and output are included as a zip file here).
​

The solution in each case will be different, because the exact problem is different. "Didn't work" is not a useful description of what you did or what happened.


I'm confident we can come up with a workable solution, but we need to know exactly what's happening on your computer. Describing what happens is crucial, otherwise we have to invent test data and run script tests on that, assuming we do any testing at all.

 

[MikeVine]

I apologize for not typing a clear explanation of my problem. I downloaded kryten2's "clean" text file and the script executed correctly! All discontinued image files were moved to the destination folder based on the text file.

I now need an explanation on how to make a "clean" text file from an Excel document, using Mac software. Thanks everyone!

 

[chown33]

1. Open the UTF-16 file in TextEdit.
2. File > Save As...
3. A dialog appears.
4. Near bottom of dialog, find "Plain Text Encoding:" and a popup.
5. Choose "Unicode (UTF-8)" from the popup.
6. Click the Save button.

If you don't see what's described in step 4, post a screenshot of your dialog.

If you don't see a "Unicode (UTF-8)" item in step 5, again, post a screenshot with the popup expanded. Or try choosing "Customize Encodings List..." and add UTF-8 to the list


I don't have that version of Excel, but I find it hard to believe there isn't a similar option in the Excel dialog. It's pretty much the standard dialog for saving text files.

 

[sero]

I would save the file as csv rather than text. Also, if you can get your clients to send you files in .xls (rather than .xlsx), you could script the conversion to csv. Check out this perl script that converts xls to csv.

 

[MikeVine]

I tried saving the Excel .xls doc to .csv and the Excel unicode UTF-16 to UTF-8 in TextEdit, and neither one worked. Both contained those returns. Whatever kryten2 did to my text file to make it "clean" worked great.

 

[kryten2]

As chown33 mentioned in post #16 I looked at the textutil command and added \r to the value assigned to IFS to let the shell strip \r's as whitespace. I tested with reading from the new file converted from your original "DiscontinuedSince2007.txt" file. It worked. The ?? are gone and it doesn't trip on the carriage-returns. Change target,destination, filelistUTF16 and filelistUTF8 to your paths.

#!/bin/bash -x
SAVEIFS=$IFS
IFS=$(echo -en "\n\b\r")
target="/Users/test/Desktop/Images"
destination="/Users/test/Desktop/Chapter 1"
filelistUTF16="/Users/test/Downloads/ZipFiles/Archive/DiscontinuedSince2007.txt"
filelistUTF8="/Users/test/Downloads/ZipFiles/Archive/DiscontinuedSince2007UTF-8.txt"
textutil -convert txt -encoding UTF-8 -output "${filelistUTF8}" "${filelistUTF16}"
fnames=$(cat "$filelistUTF8")

for i in $fnames; do
  mv "${target}/${i}"* "${destination}/"
#  exit 1
  echo "copying $i"
done
IFS=$SAVEIFS
echo "done"