Bash Script Not Executing Correctly

Discussion in 'Mac Programming' started by MikeVine, Apr 25, 2012.

  1. MikeVine, Apr 25, 2012
    Last edited by a moderator: Apr 25, 2012

    macrumors newbie

    Joined:
    Apr 25, 2012
    #1
    I found this script in an older thread and it's not functioning how I need it to. I'm not a programmer and any help on this would be great. I need to copy files from one location to another based on a list. Here is the script I'm using:
    Code:
    #!/bin/bash -x
    
    target="/Users/mvineski/Desktop/Photoproject/Images"
    destination="/Users/mvineski/Desktop/Photoproject/Chapter\ 1"
    
    fnames=$(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt)
    
    for i in $fnames; do
      cp ${target}/${i} ${destination}/
      exit 1
      echo "copying $i"
    done
    echo "done"
    
    When I run it, here are the results in terminal:
    Code:
    Last login: Wed Apr 25 12:36:26 on console
    mvineskis-Mac-Pro:~ mvineski$ #!/bin/bash -x
    mvineskis-Mac-Pro:~ mvineski$ 
    mvineskis-Mac-Pro:~ mvineski$ target="/Users/mvineski/Desktop/Photoproject/Images"
    mvineskis-Mac-Pro:~ mvineski$ destination="/Users/mvineski/Desktop/Photoproject/Chapter\ 1"
    mvineskis-Mac-Pro:~ mvineski$ 
    mvineskis-Mac-Pro:~ mvineski$ fnames=$(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt)
    mvineskis-Mac-Pro:~ mvineski$ 
    mvineskis-Mac-Pro:~ mvineski$ for i in $fnames; do
    >   cp ${target}/${i} ${destination}/
    >   exit 1
    >   echo "copying $i"
    > done
    usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
           cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
    logout
    
    [Process completed]
    
    The problem is the files do not get copied to the destination folder. Help please. Thanks.
     
  2. macrumors 68000

    Joined:
    Mar 29, 2009
    Location:
    USA
    #2
    Are you cutting + pasting to the console, or are you saving it to a file, changing the permissions to executable, and then running it?

    nano myscript.sh
    edit then save your file
    chmod 750 myscript.sh
    ./myscript.sh

    ?
     
  3. MikeVine, Apr 25, 2012
    Last edited by a moderator: Apr 25, 2012

    thread starter macrumors newbie

    Joined:
    Apr 25, 2012
    #3
    First, I tried cutting and pasting the script from my text editor into a new Terminal window, which didn't work. Then I typed this in Terminal to run the script:
    Code:
    Last login: Wed Apr 25 13:53:32 on ttys000
    mvineskis-Mac-Pro:~ mvineski$ chmod 777 /Users/mvineski/Desktop/bash_script.sh
    mvineskis-Mac-Pro:~ mvineski$ chmod +x /Users/mvineski/Desktop/bash_script.sh
    mvineskis-Mac-Pro:~ mvineski$ /Users/mvineski/Desktop/bash_script.sh
    + target=/Users/mvineski/Desktop/Photoproject/Images
    + destination='/Users/mvineski/Desktop/Photoproject/Chapter\ 1'
    ++ cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt
    + fnames=$'T089\rT341\rT343'
    + for i in '$fnames'
    + cp $'/Users/mvineski/Desktop/Photoproject/Images/T089\rT341\rT343' '/Users/mvineski/Desktop/Photoproject/Chapter\' 1/
    usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
           cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
    + exit 1
    mvineskis-Mac-Pro:~ mvineski$ 
    
    Still, no files were copied from the target folder to the destination folder.
     
  4. macrumors member

    Joined:
    Aug 28, 2008
    #4
    you could paste it into terminal like this, but first you need to fix the paths and filenames so that it makes sense

    Code:
    for i in $(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt); do cp "/Users/mvineski/Desktop/Photoproject/Images/${i}" /Users/mvineski/Desktop/Photoproject/Chapter\ 1/; echo "copying $i"; done; echo "done"
    
     
  5. Moderator

    balamw

    Staff Member

    Joined:
    Aug 16, 2005
    Location:
    New England
    #5
    Do the file names in the file have spaces in them? You may just be missing some double quotes.

    B
     
  6. kryten2, Apr 25, 2012
    Last edited: Apr 25, 2012

    macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
    #6
    The cp command didn't succeed as you can see.
    Take a look at this :

    Code:
    fnames=$'T089\rT341\rT343'
    cp $'/Users/mvineski/Desktop/Photoproject/Images/T089\rT341\rT343'
    Those line breaks/endings shouldn't be there. Open Terminal and run this to change the line ending ( a '\r' (Mac) to a '\n' (UNIX) ):

    Code:
    perl -pi -e 's/\r/\n/g' /Users/mvineski/Desktop/Photoproject/Chapter1.txt
    What exactly is in that Chapter1.txt file? Can you post a few lines of that or as a zip file?
     
  7. thread starter macrumors newbie

    Joined:
    Apr 25, 2012
    #7
    Attached is the Chapter1.txt file list which contains this copy:
    T089
    T341
    T343

    The actual image file names that need to be copied to the destination folder are as follows:
    T089 Kids Choice Sh 50p.jpg
    T341 Look Preschool 35p.jpg
    T343 Kindergarten 35p.jpg
    Thanks.
     

    Attached Files:

  8. macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
    #8
    Your Chapter1.txt file has the wrong line breaks and needs to contain the full file names : T089 Kids Choice Sh 50p.jpg etc. In Terminal :

    Code:
    ls /Users/mvineski/Desktop/Photoproject/Images > /Users/mvineski/Desktop/Photoproject/Chapter1.txt
    The script :

    Code:
    #!/bin/bash -x
    SAVEIFS=$IFS
    IFS=$(echo -en "\n\b")
    target="/Users/mvineski/Desktop/Photoproject/Images"
    destination="/Users/mvineski/Desktop/Photoproject/Chapter 1"
    
    fnames=$(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt)
    
    for i in $fnames; do
      cp ${target}/${i} ${destination}/
      echo "copying $i"
    done
    IFS=$SAVEIFS
    echo "done"
     

    Attached Files:

  9. thread starter macrumors newbie

    Joined:
    Apr 25, 2012
    #9
    Is there any way the script could do a wildcard search for just the Tnumber (T089) in the list and copy the image file with the full name (T089 Kids Choice Sh 50p.jpg) to the destination folder? The reason I ask is I only get a list of just the Tnumber and not the full image file name. Thanks everyone for all your help with this script!!
     
  10. macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
  11. MikeVine, Apr 27, 2012
    Last edited by a moderator: Apr 27, 2012

    thread starter macrumors newbie

    Joined:
    Apr 25, 2012
    #11
    kryten2,
    Your script updates worked perfectly for me. Thank you!
    I have one more coding question: instead of copying the files, what would I need to change in the script for moving the files from the target folder to the destination folder?
    Again, thanks to everyone for helping me with this script!
    Mike
    Code:
    #!/bin/bash -x
    SAVEIFS=$IFS
    IFS=$(echo -en "\n\b")
    target="/Users/mvineski/Desktop/Photoproject/Images"
    destination="/Users/mvineski/Desktop/Photoproject/Chapter"
    
    fnames=$(cat /Users/mvineski/Desktop/Photoproject/Chapter1.txt)
    
    for i in $fnames; do
      cp ${target}/${i}* ${destination}/
      echo "copying $i"
    done
    IFS=$SAVEIFS
    echo "done"
     
  12. kryten2, Apr 27, 2012
    Last edited: Apr 27, 2012

    macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
    #12
    Change line :

    Code:
    cp ${target}/${i}* ${destination}/
    to

    Code:
    [B]mv[/B] ${target}/${i}* ${destination}/
    There are a lot of BASH shell scripting tutorials to be found on the net. Check these out if you want to learn some basic things. Copying files, moving files etc.

    http://tldp.org/LDP/Bash-Beginners-Guide/html/index.html
     
  13. thread starter macrumors newbie

    Joined:
    Apr 25, 2012
    #13
    Well, I jumped the gun. When I tried the script with the actual list and image files, no files were moved to the destination folder. There are at least 14 image files that should have been moved from the target folder to the destination folder, that are contained in the list.

    I've attached the following files in Archive.zip:
    TerminalSavedOutput.txt (Results from Terminal)
    MoveFiles_WildcardSearch_FromList.sh (script used)
    DiscontinuedSince2007.txt (list used)

    Any ideas? Thanks, Mike
     

    Attached Files:

  14. chown33, Apr 30, 2012
    Last edited: Apr 30, 2012

    macrumors 603

    Joined:
    Aug 9, 2009
    #14
    1. Exactly how and where are you obtaining "DiscontinuedSince2007.txt"?
    Exactly what program and OS produced the file?

    When inspected using a hex editor, I see two non-text bytes 0xFFFE at the start. Also, the entire file appears to be in 16-bit Unicode. Knowing how and where the file came from would clarify that.

    In any case, the text will need more filtering than is currently being applied.

    Example hex dump of start of file:
    Code:
    hexdump -C DiscontinuedSince2007.txt  | head 
    
    00000000  ff fe 54 00 30 00 39 00  30 00 37 00 37 00 0d 00  |..T.0.9.0.7.7...|
    00000010  0a 00 54 00 30 00 39 00  31 00 39 00 34 00 0d 00  |..T.0.9.1.9.4...|
    00000020  0a 00 54 00 30 00 39 00  32 00 30 00 32 00 0d 00  |..T.0.9.2.0.2...|
    00000030  0a 00 54 00 30 00 39 00  32 00 30 00 34 00 0d 00  |..T.0.9.2.0.4...|
    00000040  0a 00 54 00 31 00 30 00  30 00 34 00 31 00 0d 00  |..T.1.0.0.4.1...|
    00000050  0a 00 54 00 33 00 31 00  33 00 36 00 0d 00 0a 00  |..T.3.1.3.6.....|
    00000060  54 00 33 00 31 00 39 00  30 00 31 00 0d 00 0a 00  |T.3.1.9.0.1.....|
    00000070  54 00 33 00 37 00 30 00  30 00 35 00 0d 00 0a 00  |T.3.7.0.0.5.....|
    00000080  54 00 33 00 37 00 30 00  30 00 36 00 0d 00 0a 00  |T.3.7.0.0.6.....|
    00000090  54 00 33 00 37 00 30 00  31 00 31 00 0d 00 0a 00  |T.3.7.0.1.1.....|
    
    2. In addition to 0xFFFE, there are carriage-returns in the filenames, which appear as \r characters in the Terminal output.

    These also need to be stripped out.


    3. Please provide a list of the files that should have been moved but weren't, and ones that were actually moved.

    In the Terminal output, the first file is clearly not moved, because its attempted name contains the 0xFFFE (which appear as ?? in the output).

    On further inspection, it looks to me like every file failed, due to the \r. So according to the Terminal output, NO FILES were moved. There are 1645 instances of "No such file or directory" and 1645 files in the input list.
     
  15. Moderator

    balamw

    Staff Member

    Joined:
    Aug 16, 2005
    Location:
    New England
    #15
    You beat me to it.

    I didn't see this when I opened the file in TextEdit and was able to fix it just by saving it as UTF-8 instead of UTF-16 in TextEdit.

    The last one seemed to work OK as there was no \r.

    Code:
    + mv '/Users/mvineski/Desktop/DVD2012/T72001_T98004/TA67923*' /Users/mvineski/Desktop/DVD2012/zDD/
    mv: rename /Users/mvineski/Desktop/DVD2012/T72001_T98004/TA67923* to /Users/mvineski/Desktop/DVD2012/zDD/TA67923*: No such file or directory
    + echo 'copying TA67923'
    copying TA67923
    
    B
     
  16. macrumors 603

    Joined:
    Aug 9, 2009
    #16
    The textutil command should be able to change the encoding in a shell script. I don't have time right now.

    Also, adding \r to the value assigned to IFS should let the shell strip \r's as whitespace.

    And I recommend this:
    I typed that hastily, so it could be wrong. But it also gives the drift: use conditional execution to echo failures.

    And note the quoting.
     
  17. macrumors member

    Joined:
    Aug 28, 2008
    #17
    a bit of a hack but I think it'll work for this instance. see the posts by kryten2, chown33 and balamw if this isn't just a one time thing.

    Code:
    #!/bin/bash
    target="/Users/mvineski/Desktop/DVD2012/T72001_T98004"
    destination="/Users/mvineski/Desktop/DVD2012/zDD"
    list="/Users/mvineski/Desktop/BashScript_MoveFiles/DiscontinuedSince2007.txt"
    
    #fix_list
    tr -d '\r' < "$list" >fixed_list && cat fixed_list|sed 's/^[^a-zA-Z0-9][^a-zA-Z0-9]//g' >tmp && mv tmp fixed_list
    fnames=$(cat fixed_list)
    for i in $fnames; do
      mv "${target}/${i}*" "${destination}/"
      echo "copying $i"
    done
    rm fixed_list
    echo "done"
    
     
  18. thread starter macrumors newbie

    Joined:
    Apr 25, 2012
    #18
    Windows users send me the list as a Excel document (.xlsx) containing one column and multiple rows of Tnumbers. I'm on a Mac, 2x2.4GHz Quad-Core Intel Xeon running MacOS10.6.8. I open that .xlsx file in my Mac 2004 version of Excel and Save As a UTF-16 format text file (no UTF-8 format available). I also tried saving the UTF-16 text file to UTF-8 in TextEdit, but that didn't work either.

    Here is a partial list of items that need to be moved:
    T72309
    T72312
    T72316
    T23322
    T72325
    T72327
    T72329
    T72330
    T72334
    T72335
    T72339
    T73017
    T73012
    T1151
    No files were moved when I executed the script.

    This process is not a one time thing. I either need to get a script to work, or move files manually (what a drag!).

    Can you please let me know of a procedure or correct format I should select in Excel for saving out a workable text file list that doesn't include any unnecessary hidden characters that make the script fail? Thanks, Mike
     
  19. macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
    #19
    @chown33, @balamw, @sero : Thank you for your feedback.
    I saw MikeVine's post, took a quick look at the provided files and yelled it's those carriage-returns again plus an added bonus of two ?? at the start of the txt file. Had to do something else so I couldn't respond immediately. I've attached a clean txt file.
     

    Attached Files:

  20. macrumors 603

    Joined:
    Aug 9, 2009
    #20
    Please describe exactly what you did, and exactly what happened.

    There are many ways to interpret "didn't work". Examples:
    1. I tried to Save As in TextEdit, but I see no option to save as UTF-8. Here's a screen shot of the dialog <post screen shot here>.

    2. I was able to Save As UTF-8 in TextEdit, and I changed the name of the input file in the script, but even as UTF-8, the shell script still failed (new UTF-8 file, shell script, and output are included as a zip file here).
    The solution in each case will be different, because the exact problem is different. "Didn't work" is not a useful description of what you did or what happened.


    I'm confident we can come up with a workable solution, but we need to know exactly what's happening on your computer. Describing what happens is crucial, otherwise we have to invent test data and run script tests on that, assuming we do any testing at all.
     
  21. thread starter macrumors newbie

    Joined:
    Apr 25, 2012
    #21
    I apologize for not typing a clear explanation of my problem. I downloaded kryten2's "clean" text file and the script executed correctly! All discontinued image files were moved to the destination folder based on the text file.

    I now need an explanation on how to make a "clean" text file from an Excel document, using Mac software. Thanks everyone!
     
  22. macrumors 603

    Joined:
    Aug 9, 2009
    #22
    1. Open the UTF-16 file in TextEdit.
    2. File > Save As...
    3. A dialog appears.
    4. Near bottom of dialog, find "Plain Text Encoding:" and a popup.
    5. Choose "Unicode (UTF-8)" from the popup.
    6. Click the Save button.

    If you don't see what's described in step 4, post a screenshot of your dialog.

    If you don't see a "Unicode (UTF-8)" item in step 5, again, post a screenshot with the popup expanded. Or try choosing "Customize Encodings List..." and add UTF-8 to the list


    I don't have that version of Excel, but I find it hard to believe there isn't a similar option in the Excel dialog. It's pretty much the standard dialog for saving text files.
     
  23. macrumors member

    Joined:
    Aug 28, 2008
    #23
    I would save the file as csv rather than text. Also, if you can get your clients to send you files in .xls (rather than .xlsx), you could script the conversion to csv. Check out this perl script that converts xls to csv.
     
  24. thread starter macrumors newbie

    Joined:
    Apr 25, 2012
    #24
    I tried saving the Excel .xls doc to .csv and the Excel unicode UTF-16 to UTF-8 in TextEdit, and neither one worked. Both contained those returns. Whatever kryten2 did to my text file to make it "clean" worked great.
     
  25. kryten2, Apr 30, 2012
    Last edited: Apr 30, 2012

    macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
    #25
    As chown33 mentioned in post #16 I looked at the textutil command and added \r to the value assigned to IFS to let the shell strip \r's as whitespace. I tested with reading from the new file converted from your original "DiscontinuedSince2007.txt" file. It worked. The ?? are gone and it doesn't trip on the carriage-returns. Change target,destination, filelistUTF16 and filelistUTF8 to your paths.

    Code:
    #!/bin/bash -x
    SAVEIFS=$IFS
    IFS=$(echo -en "\n\b\r")
    target="/Users/test/Desktop/Images"
    destination="/Users/test/Desktop/Chapter 1"
    filelistUTF16="/Users/test/Downloads/ZipFiles/Archive/DiscontinuedSince2007.txt"
    filelistUTF8="/Users/test/Downloads/ZipFiles/Archive/DiscontinuedSince2007UTF-8.txt"
    textutil -convert txt -encoding UTF-8 -output "${filelistUTF8}" "${filelistUTF16}"
    fnames=$(cat "$filelistUTF8")
    
    for i in $fnames; do
      mv "${target}/${i}"* "${destination}/"
    #  exit 1
      echo "copying $i"
    done
    IFS=$SAVEIFS
    echo "done"
    
     

Share This Page