Rsync and non standard characters

[rpsibewl110]

Hello
I am running rsync in automater to back up the music folder. In this folder there are some non standard characters in some of the names. ie Josh Groban/Noël/songs.

What happens is when I have the delete command in the command line it will not see the song in the Noël folder and then it will delete it from the back up folder. I took the delete command out so it does not delete the backed up files.

My question is is there a command or switch to ignore the characters and back up the files?

rsync -avu ~/Documents/ /Volumes/Macbook/Documents
rsync -avu ~/Music/ /Volumes/Macbook/music
rsync -avu ~/Pictures/ /Volumes/Macbook/Pictures
rsync -avu ~/Desktop/ /Volumes/Macbook/Desktop

edited to remove the term ascii

 

[casperes1996]

Not an answer to your question, but something that might be helpful for you if you want to search for this more yourself;
ë is not an ASCII character. I think it is in extended ASCII, possibly, but it is not ASCII. Pure ASCII is only 7 bits and thus only the first 0->127 characters. 128->255 is extended ASCII. You’d probably want to consider ë a Unicode character and forego the ASCII v extended ASCII debate entirely

 

[mj_]

I've tried just that on my MacBook right now using the Josh Groban/Noël/songs structure as described above and had no issues. My Mac correctly detected all files in the songs folder within Josh Groban/Noël and copied them over:

martin@MacBook test % mkdir -p "Josh Groban/Noël/songs"
martin@MacBook test % cd Josh\ Groban/Noël/songs
martin@MacBook songs % touch a b
martin@MacBook songs % cd ../../../..
martin@MacBook ~ % rsync -av --delete test test2
building file list ... done
created directory test2
test/
test/Josh Groban/
test/Josh Groban/Noël/
test/Josh Groban/Noël/songs/
test/Josh Groban/Noël/songs/a
test/Josh Groban/Noël/songs/b

sent 254 bytes  received 88 bytes  684.00 bytes/sec
total size is 0  speedup is 0.00
martin@MacBook ~ % find test
test
test/Josh Groban
test/Josh Groban/Noël
test/Josh Groban/Noël/songs
test/Josh Groban/Noël/songs/a
test/Josh Groban/Noël/songs/b
martin@MacBook ~ % find test2
test2
test2/test
test2/test/Josh Groban
test2/test/Josh Groban/Noël
test2/test/Josh Groban/Noël/songs
test2/test/Josh Groban/Noël/songs/a
test2/test/Josh Groban/Noël/songs/b

Not sure why it isn't working on your side. What's your locale set to and what shell are you using?

 

[chown33]

What are the files being backed up to? Please be specific, identifying the model, version, etc. of the target host or device, and what format the storage is in (e.g. FAT, ExFAT, HFS+, etc.).

Different file-systems use different encodings for characters outside the basic 7-bit ASCII area. Furthermore, accented characters in Unicode have different representations. For example, the ë can be represented by a single Unicode code point U+00EB or by the combining diaresis followed by small letter e. This can be significant because HFS and its derivatives will convert different forms into a canonical form with combining marks, and store that on disk. I'm almost certain that APFS does the same.

I don't know offhand how 'rsync' compares names. If the target doesn't use the canonicalized combining-mark form produced by HFS or APFS, and rsync doesn't attempt any conversion, just doing a byte-by-byte comparison, then the canonicalized form won't necessarily match whatever file-system representation the target is using.

Here's some info on the Unicode character block where Small Letter e with diaresis resides:

Latin-1 Supplement - Wikipedia

en.wikipedia.org

Here's one of the Unicode combining marks pages:

Combining Diacritical Marks - Wikipedia

en.wikipedia.org

 

[rpsibewl110]

What are the files being backed up to? Please be specific, identifying the model, version, etc. of the target host or device, and what format the storage is in (e.g. FAT, ExFAT, HFS+, etc.).

I apologize about the lack of information.

Source APFS is MacBookAir 6,2 running Catalina , target is a Mac Pro 3,1 on the same network. Running El Capitan drive is OS X extended Journaled.

I ran ChronoSync in trial mode and it moved the files correctly.

 

[chown33]

If you want to dig into this deeper, I can outline a test.

First, reduce the amount of data to at most a handful of files and folders. Make a folder with a diacritic'ed character in its name (ë is fine for now), and a folder without. Then in each folder, make a file with a diacritic'ed character in its name. Again, é is fine at first. Later on, it might be informative to try different characters with different diacritical marks, and see if there's any difference.

Next, work out how to use the 'tcpdump' command so it tells you all the network traffic going between the machines. Wireshark is another tool that may be helpful in collecting the network traffic.

I'm pretty sure rsync has a standard TCP port, but it may be worthwhile to cast a broader net to collect data initially, then run a few tests, then narrow the criteria once you're sure of what the traffic is.

Once the traffic data is captured (e.g. as hex in a plain text file), refer to the rsync protocol to decode the data. As I vaguely recall, there's a pathname and some rolling checksums exchanged. The pathnames are likely the only thing you're interested in, so look at what the actual binary of each name is. My first guess would be UTF-8 encoding, but it wouldn't surprise me if it were UTF-16.


A variant of the basic test from MBA to Mac Pro would be to go the other direction: Mac Pro to MBA, and see what the rsync traffic data looks like. Or just do that test and see if it works. If it does, then maybe the problem is something simple, like slightly incompatible versions of rsync or its protocol.

Ask rsync to output its version:
rsync --version

Ask rsync to use a different protocol version:
rsync --protocol=NUM

 

[rpsibewl110]

Today I upgraded to Big Sur and the problem seems to have disappeared. Unless the protocol version has changed? Is 31 the correct version? I did type in rsync --protocol=28 then 31 it seemed to stay at 31. So without knowing 100% of the usage I am guessing it needs to be used in the rsync command as I type it into terminal which I did not do as you can tell by the commands I post in the message.

$ rsync --version
rsync version 3.2.3 protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, no atimes,
batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
symtimes, no prealloc, stop-at, crtimes, file-flags
Optimizations:
SIMD, asm, openssl-crypto
Checksum list:
xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.


Command run in Terminal
Deleted Feliz Navidad on target, file transferred

$ rsync -avu ~/Music/ /Volumes/Macbook/Music
sending incremental file list
Media/Media.localized/José Feliciano/
Media/Media.localized/José Feliciano/Feliz Navidad/
Media/Media.localized/José Feliciano/Feliz Navidad/01 Feliz Navidad.m4a

sent 7,195,231 bytes received 721 bytes 214,804.54 bytes/sec
total size is 25,511,747,310 speedup is 3,545.29

Command run in Terminal, Deleted Feliz Navidad on target, file transferred

$ rsync -avu ~/Music/Media/Media.localized/José\ Feliciano/Feliz\ Navidad/ /Volumes/Macbook/Music/Media/Media.localized/José\ Feliciano/Feliz\ Navidad/
sending incremental file list
./
01 Feliz Navidad.m4a

sent 7,026,015 bytes received 38 bytes 4,684,035.33 bytes/sec
total size is 7,024,163 speedup is 1.00

Command run in Terminal, Deleted 13 songs on target, all transferred

$ rsync -avu ~/Music/Media/Media.localized/Josh\ Groban/Noël/ /Volumes/Macbook/Music/Media/Media.localized/Josh\ Groban/Noël/
sending incremental file list
./
01 Silent Night.m4a
02 Little Drummer Boy (feat. Andy McKee).m4a
03 I'll Be Home for Christmas.m4a
04 Ave Maria.m4a
05 Angels We Have Heard On High.m4a
06 Thankful.m4a
07 The Christmas Song.m4a
08 What Child Is This_.m4a
09 The First Noel.m4a
10 Petit Papa Noel.m4a
11 It Came Upon a Midnight Clear.m4a
12 Panis Angelicus.m4a
13 O Come All Ye Faithful.m4a

sent 114,306,768 bytes received 266 bytes 10,886,384.19 bytes/sec
total size is 114,277,832 speedup is 1.00

Commands run in an automater app, Deleted files on target, Files transferred
Command in app is rsync -avu ~/Music/ /Volumes/Macbook/Music

building file list ... done

sent 107639 bytes received 20 bytes 3417.75 bytes/sec
total size is 13261956122 speedup is 123184.83
building file list ... done
Media/Media.localized/Jose\#314\#201 Feliciano/Feliz Navidad/
Media/Media.localized/Jose\#314\#201 Feliciano/Feliz Navidad/01 Feliz Navidad.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/
Media/Media.localized/Josh Groban/Noe\#314\#210l/01 Silent Night.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/02 Little Drummer Boy (feat. Andy McKee).m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/03 I'll Be Home for Christmas.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/04 Ave Maria.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/05 Angels We Have Heard On High.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/06 Thankful.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/07 The Christmas Song.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/08 What Child Is This_.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/09 The First Noel.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/10 Petit Papa Noel.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/11 It Came Upon a Midnight Clear.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/12 Panis Angelicus.m4a
Media/Media.localized/Josh Groban/Noe\#314\#210l/13 O Come All Ye Faithful.m4a

sent 121479010 bytes received 340 bytes 2669875.82 bytes/sec
total size is 25511747310 speedup is 210.01