Linux file names (/ and ? not allowed)

[plugsnpixels]

I am mainly a Mac user who dinks around in Windows as needed and Linux just because I can (both dedicated computers and VMs).

I recently decided I wanted a non-Mac-friendly backup of my data in case I am ever without a Mac, so I have been copying my data from various HFS+ external drives to an 8TB Seagate formatted as NTFS out the door as well as a 5TB exFAT drive intended to be able to transfer data like a mule among all systems.

I have a Mac that can read/write to NTFS (super glacially slow, so I have turned to alternatives: a 2011 PC build running Windows 10 and a Linux box about the same age running the latest Linux Lite). The Windows and Linux boxes copy the data much faster.

Here in the office I am writing you from a Mac but am using the old Linux box to do the copying from an HFS+ backup drive to the exFAT drive (I just learned Linux reads HFS+ natively, which is very helpful). I will later use the exFAT drive to transfer files to the NTFS drive at home.

But lo and behold, any file with a / or a ? or some other unknown permissions issue has to be skipped – and that is a LOT of them. This is true whether I do a drag-and-drop or use FreeFileSync. Linux CAN open and play these (mostly music) files from the source however. A web search revealed this is indeed a limitaiton of Linux.

I guess when I get back to the Mac or Windows box at home I can top off this exFAT backup with FreeFileSync and add these skipped files.

So not being able to write these particular files to backup in Linux renders it sort of "not useful" for this current purpose, fast as the copying is going. That was a disappointing thing to discover as Linix is otherwise very powerful.

 

[bradl]

I am mainly a Mac user who dinks around in Windows as needed and Linux just because I can (both dedicated computers and VMs).

I recently decided I wanted a non-Mac-friendly backup of my data in case I am ever without a Mac, so I have been copying my data from various HFS+ external drives to an 8TB Seagate formatted as NTFS out the door as well as a 5TB exFAT drive intended to be able to transfer data like a mule among all systems.

I have a Mac that can read/write to NTFS (super glacially slow, so I have turned to alternatives: a 2011 PC build running Windows 10 and a Linux box about the same age running the latest Linux Lite). The Windows and Linux boxes copy the data much faster.

Here in the office I am writing you from a Mac but am using the old Linux box to do the copying from an HFS+ backup drive to the exFAT drive (I just learned Linux reads HFS+ natively, which is very helpful). I will later use the exFAT drive to transfer files to the NTFS drive at home.

But lo and behold, any file with a / or a ? or some other unknown permissions issue has to be skipped – and that is a LOT of them. This is true whether I do a drag-and-drop or use FreeFileSync. Linux CAN open and play these (mostly music) files from the source however. A web search revealed this is indeed a limitaiton of Linux.

I guess when I get back to the Mac or Windows box at home I can top off this exFAT backup with FreeFileSync and add these skipped files.

So not being able to write these particular files to backup in Linux renders it sort of "not useful" for this current purpose, fast as the copying is going. That was a disappointing thing to discover as Linix is otherwise very powerful.

This doesn't just go for Linux, but for any Unix operating system, MacOS included. If you get down to the CLI on MacOS, you'll notice that / is used for directory structures/paths on a Unix filesystem, and ? is a special character as well, used for pattern matching, or finding out the return status of a given process that has run. In short, this isn't a limitation of Linux, but actually the directory structure for the filesystems in any Unix or Unix-like environment (in case of / ) or pattern matching to find a file you are looking for (in the case of ? ). It is always best to avoid using those in filenames, because unless properly escaped, they can cause problems when looking for files or directories across the entire filesystem.

BL.

 

[plugsnpixels]

This doesn't just go for Linux, but for any Unix operating system, MacOS included. If you get down to the CLI on MacOS, you'll notice that / is used for directory structures/paths on a Unix filesystem, and ? is a special character as well, used for pattern matching, or finding out the return status of a given process that has run. In short, this isn't a limitation of Linux, but actually the directory structure for the filesystems in any Unix or Unix-like environment (in case of / ) or pattern matching to find a file you are looking for (in the case of ? ). It is always best to avoid using those in filenames, because unless properly escaped, they can cause problems when looking for files or directories across the entire filesystem.

BL.

Thanks, on the Mac I seem to have no issues with these characters in file names (naming, copying, accessing), but I understand what you're explaining.

 

[bradl]

Thanks, on the Mac I seem to have no issues with these characters in file names (naming, copying, accessing), but I understand what you're explaining.

You'll only see this when you get to the filesystem level, not at the UI. For example, you'll need to run Terminal.app and have an understanding of how to use the CLI to see that. once you do that, you can run ls -al to see the files in what the UI calls your Home (as if you did Shift-Command-H). If you did a cd /usr/bin and run that ls -al command again, you'll be in a completely different directory, and seeing what actually are commands that can be run at the CLI. The key here is that notice how you used the / in that cd command; The backup program you would use wouldn't be able to discern between the / being in the name of a file, or a path to a file.

Basically, the UI that you are seeing is the window manager for the underlying Unix OS that MacOS uses.

BL.

 

[maflynn]

A web search revealed this is indeed a limitaiton of Linux.

Correct, People who design/develop file systems will reserve certain characters that cannot be used as part of the file name. You've been using those characters that many operating systems would have a problem with. MacOS' new file system appears to have done away with these forbidden characters but its probably safest not to use them so you can interact with other platforms.

I generally only use alphanumeric characters but then I'm old school

 

[plugsnpixels]

Correct, People who design/develop file systems will reserve certain characters that cannot be used as part of the file name. You've been using those characters that many operating systems would have a problem with. MacOS' new file system appears to have done away with these forbidden characters but its probably safest not to use them so you can interact with other platforms.

I generally only use alphanumeric characters but then I'm old school

Thanks Mike, and BTW, I've long enjoyed your posts about flip flopping!

As for these files, I use "/" all the time for music and data files, adding bitrate info to music file titles in the Finder such as "24/96" and dates to genealogy-related items such as "01/01/2022".

After spending a lot of time today sudo pkill -f fsck'ing to get the 5TB exFAT drive to mount on Mac (it keeps locking up, maybe it's dying though it repairs successfully) and watching ._ and .DS_Store files being added and subtracted during syncing (depending on which platform I was using for the task), I think I've come to this conclusion:

I like working in the macOS Finder (bad as it behaves sometimes). I like the columns which Windows and Linux don't have (is there an app?), and I find it very easy to organize and back up my stuff. So I will stop worrying about Windows-native file systems.

Now that I know Linux can natively read HFS+ (and I think Windows as well with some software), the native Mac format is an option for accessing my files should I be on a Linux box for any length of time for some reason. And I've seen Windows recognize my HFS+ drive (mounted on a Mac) over the home network, so the files are accessible in that manner as well.

Copying and syncing files on exFAT and NTFS is a grinding nightmare! At least when a Mac is involved.

PS: When viewed in Linux, "/" becomes ":" which is odd! I know that is a no-no character on Mac.

PPS: Notice I haven't mentioned backing up to SSD and APFS, since I don't have any large SSDs to work with... I'm sure that's another cross-platform nightmare.

PPPS: Now what about making the 5TB NTFS? Block size and maybe reliability/compatibility might be better (exFAT block size is too big and wastes space). I've also read exFAT on Mac is not very good.

 

[maflynn]

Oddly, this sort of information is hard to come by for each of the platforms

I found this link https://amigotechnotes.wordpress.com/2015/04/02/invalid-characters-in-file-names/

It details the various platform's use of forbidden characters

HFS Plus Invalid Characters
According to How Macintosh Filenames Are Translated on Microsoft, : (colon) is the only invalid characters in file names.

/ is not allowed in terminal due to Unix.

ext4 Invalid Characters
According to ext4 on Wiki, / NULL are invalid characters in file names.

Here's what I found from MSFT

• Use any character in the current code page for a name, including Unicode characters and characters in the extended character set (128–255), except for the following:
  • The following reserved characters:
    • < (less than)
    • > (greater than)
    • : (colon)
    • " (double quote)
    • / (forward slash)
    • \ (backslash)
    • | (vertical bar or pipe)
    • ? (question mark)
    • * (asterisk)
  • Integer value zero, sometimes referred to as the ASCII NUL character.
  • Characters whose integer representations are in the range from 1 through 31, except for alternate data streams where these characters are allowed. For more information about file streams, see File Streams.
  • Any other character that the target file system does not allow.

 

[plugsnpixels]

Oddly, this sort of information is hard to come by for each of the platforms

I found this link https://amigotechnotes.wordpress.com/2015/04/02/invalid-characters-in-file-names/

It details the various platform's use of forbidden characters





Here's what I found from MSFT

Thanks for the research! This is an interesting topic. Somehow Apple has made this type of thing "friendly" to the point where Mac users only had to avoid ":"

 

[plugsnpixels]

Still attempting to move data to the 8TB NTFS drive from the 5T exFAT drive (which was itself populated on the Mac from the HFS+ source), currently on a Windows box. Still slow as it can possibly be. Plus on these copies all of my illegal-character names originating on the Mac had to be substituted (/ became _).

Such a miserable experience! If it weren't for the time already invested I would just wipe everything and go HFS+ all around (no SSDs involved) and use Windows and Linux on their own merits with maybe minor file transfers among them and the Macs over an ethernet network or via Dropbox.

In the course of this misery I learned that Linux can directly read the HFS+ drives and I've got Windows reading them too with an app, so maybe that's good enough to access and copy over small bits of data as needed (photos to edit, etc.), but no good at all for making large clones/backups of the original Mac source archive.

For now I'll use the 5TB to carry most of my data to the office for another set of backup, it'll be good enough for that.

 

[maflynn]

Still attempting to move data to the 8TB NTFS drive from the 5T exFAT drive

Is there a specific reason that you're trying to make a backup that is compatible to other platforms?

If there's no critical reason, play the long game. Start with new habits with file naming. Use a utility to batch rename files, not everything all at once but small bites and something you can validate.

It took you a long time to get 5TB of data, it will take time to shift gears and use file names that would be compatible on other platforms.

Also some perspective Macs are not going away, if there's no hard and fast reason - what harm is there in just keep doing what you're doing, or at the very least not try to change/rename all of your archive data? I can see leaving your existing data as is, and then just use compatible file names for any new stuff. Eventually the old stuff will age out, or if its still needed, be accessible on your mac

 

[plugsnpixels]

Good points Mike. I was (for no good reason) hedging my bets. I am mainly a Mac user (been so for 30 years and even work in higher ed IT supporting Macs). Even so, I like to tinker in Windows and Linux to keep up my awareness and to use the odd non-Mac app.

As I discovered during this adventure, non-Apple platforms can still access Mac-formatted data via direct connection, network or cloud, so it seems a dedicated backup in a universal format is not strictly necessary.

I have utilities for file renaming so that wouldn't be a big deal, except that I like to name my files "in friendly English". Being a Mac user is all about the "friendly computing", right? ;-)

 

[cyb3rdud3]

Good points Mike. I was (for no good reason) hedging my bets. I am mainly a Mac user (been so for 30 years and even work in higher ed IT supporting Macs). Even so, I like to tinker in Windows and Linux to keep up my awareness and to use the odd non-Mac app.

As I discovered during this adventure, non-Apple platforms can still access Mac-formatted data via direct connection, network or cloud, so it seems a dedicated backup in a universal format is not strictly necessary.

I have utilities for file renaming so that wouldn't be a big deal, except that I like to name my files "in friendly English". Being a Mac user is all about the "friendly computing", right? ;-)

Friendly English - I guess you mean quiet verbose and with spaces etc. Again not something I'd recommend as those characters may have to be substituted and then you'll see %20 on many other platforms.

Likewise I don't understand why you want to put meta data in the file names thinks like bitrate and dates belong in meta data. Makes it way easy to find, sort and deal with opposed to in file names that you have to read.

 

[plugsnpixels]

Friendly English - I guess you mean quiet verbose and with spaces etc. Again not something I'd recommend as those characters may have to be substituted and then you'll see %20 on many other platforms.

Likewise I don't understand why you want to put meta data in the file names thinks like bitrate and dates belong in meta data. Makes it way easy to find, sort and deal with opposed to in file names that you have to read.

Good points, thanks for your input!