RAID vs file sync for usb drives

[lifereinspired]

Hi,

I’m running a M1 Mac mini and have some data on external USB drives (currently as exFAT). I keep my data on a primary drive with a second drive that duplicates it.

At this point, a hardware RAID setup isn’t doable. I keep the data current on the 2nd drive using file sync software.

I’ve known that MacOS supports RAID for awhile but only recently learned that it can support software RAID using USB drives. I’m only looking for a simple mirror setup (I think that would be RAID 1). I’ve been reading & can’t tell if it’s a benefit or not to setup RAID as opinions seem split. I’d love to hear thoughts on stability, use, swapping drives/rebuilding if/when there are issues down the road, and any other things that might be of concern or benefit vs just continuing as I have with file sync. Any info & advice would be much appreciated. Thanks!

 

[throAU]

as someone who's dealt with raid/storage arrays/etc. for the past 25 years for my job...

wouldn't try doing RAID1 over USB connected drives for anything you want to rely on, USB can be flaky and the way traditional RAID works, if one of the drives drops out it will need to re-sync the ENTIRE DRIVE (e.g., even if you are using 100 GB of a 4 TB drive, the full 4 TB needs to re-sync - which will take freaking AGES - many hours) once it is reconnected. and if the other drive drops out halfway through? who knows! Its a recipe for pain - pain that may not present itself (in the form of silent data corruption or an unmountable file system) until you've started relying on it for some time.

traditional raid really needs to be on reliable well connected hardware and even then some modern drives can cause their own issues with RAID unless they're specifically enabled for proper RAID support (e.g.: https://www.abmx.com/blog/what-is-tler).

I'd consider (if you have one) attaching the other drive to a different machine and sync to it once a day or just use it as a time machine backup location (or similar). That way you'll get a second copy of your data with some sort of multi-version history to protect against file deletion or corruption as opposed to just drive failure, and you won't be risking data loss due to RAID misbehavior on USB connected drives.

If you want to do RAID with these, stick them in a NAS or disk array intended to do this job. And make sure that array either doesn't need error recovery enabled or your drives have the required features.


Again, in this situation you're far better off just using one as a data drive and another as a backup copy that is periodically updated.

 

[Fishrrman]

The dumb ol' Fishrrman's way of keeping data on two drives (that aren't boot drives):

One drive serves as "the primary/external storage" drive.
The second drive is the backup of the primary storage drive.

At regular intervals, I use CarbonCopyCloner to "dupe" the contents of the primary drive to the backup drive.

No, it's not "simulataneous" backup.
But it works, it's reliable, and it's easy.

 

[lifereinspired]

@throAU, thanks so much for taking the time so share your knowledge & experience, especially since you’ve worked with RAID for so long. This is exactly what I needed to know and wasn’t sure about. I didn’t realize that RAID would have to rebuild the entire array if there was any anomaly, disconnect, etc. I’m gonna take your advice and avoid RAID. Again, I really appreciate your advice.

@Fishrrman, I’ll look into CarbonCopyCloner. I’ve been using FreeFileSync (though I’ve donated for the support & extra features) and it works pretty well but CCC might have some additional features that are worth considering.

I agree that it’s not simultaneous backup, but there is a safety net in that, too. If one accidentally deletes a file, it’s not instantly deleted off the backup. As @throAU said, it gives a degree of versioning with the time lapse.

I hadn’t kept up my backup super regularly the last few months due to other life issues & felt a little urgency about it. Decided to catch up & backed up my drives. One has been throwing error -50 when I’ll try to copy files on it and I rebooted my Mac to see if that would help. When it came back, over 1/2 a terabyte of files were gone or in some weird cases, partly gone. So the first part was there (in some cases just a few MB of large files). Because I had just backed up, I was able to copy everything back. I have absolutely no idea what happened or why all those files disappeared.

I downloaded DriveDX to check out the drive that’s been acting weird & it said it was fine (don’t know whether to believe that), BUT the drive I’ve been using as the backup is in imminent failure. Gotta upgrade now for the full version but unless I hear of a better option, DriveDX seems to be well regarded & the price is fair.

I’m gonna copy all the data to a different drive & reformat the drive throwing the errors to see if that fixes things. I had trouble running FirstAid but was able to add Disk Utility to security giving it full disk access & it ran without issues but didn’t fix things.

Again, I really appreciate you saving me tons of time, pain, and frustration down the road.

 

[throAU]

No, it's not "simulataneous" backup.
But it works, it's reliable, and it's easy.

This is a feature, not a bug.

Constant sync without versioning (e.g., RAID 1 mirror) = you destroy some data, it is is destroyed in both locations

 

[throAU]

@throAU, thanks so much for taking the time so share your knowledge & experience, especially since you’ve worked with RAID for so long. This is exactly what I needed to know and wasn’t sure about. I didn’t realize that RAID would have to rebuild the entire array if there was any anomaly, disconnect, etc. I’m gonna take your advice and avoid RAID. Again, I really appreciate your advice.

I will add that some non-traditional RAID platforms CAN do partial rebuilds; anything based on ZFS for example, due to the integration of the filesystem into the disk management, can understand exactly what data was changed between the disconnect and the reconnect, but again that's not macOS RAID and so out of scope of what you can do without spending money on more hardware.

 

[ChrisA]

Hi,

I’m running a M1 Mac mini and have some data on external USB drives (currently as exFAT). I keep my data on a primary drive with a second drive that duplicates it.

Both setups are poor and don't offer good backup. Here is why? Lets say one file get corrupted, using eithrer RAID or some kind of file sync the corruption is copied to the second drive in real time.

A better use of the second drive is to use Time Machine. This keeps a VERSIONED copy on the second drive. Now if a file is corrupted, the corrupted data never overwrites the good data. You will always have the option to revert back to the older version of the file.

Apple makes it dead easy to set this up. The downside is that the second drive needs to be a little larger to hold the old versions but APFS makes this OK because only the differences are stored, and if the data does not change a lot, like say an active SQL database would, then you can do fine for a long time with using two identical drives. (get rid of all exFAT file systems, use APFS. APFS does to things you want, checksums the metadata, can "checkpoint" the data (this is a zero-cost operation), and COW, Copy on Write.

People think data loss happens because a disk drive fails. Yes that might happen but the big causes are (1) user error, (2) software issues, (3) theft of the equipment, (4) lightning striking the power lines. Drives don't fail as often as these other cases.

You don't want RAID, file sync, or even periodic cloning of the data. You should be keeping a versioned history

 

[gilby101]

(currently as exFAT)

Yoiks! If you want reliability, you should format as APFS (or even HFS+) on macOS. (Of course, backing up before reformatting!). exFAT has no protection against simple issues like the cable falling out.

As the others have said you want versioned backup to protect against hardware failure, data corruption, etc. Just add the source drive to your Time Machine backup or get CCC for more fine control over the backup.

 

[throAU]

Both setups are poor and don't offer good backup. Here is why? Lets say one file get corrupted, using eithrer RAID or some kind of file sync the corruption is copied to the second drive in real time.

A better use of the second drive is to use Time Machine. This keeps a VERSIONED copy on the second drive. Now if a file is corrupted, the corrupted data never overwrites the good data. You will always have the option to revert back to the older version of the file.

Apple makes it dead easy to set this up. The downside is that the second drive needs to be a little larger to hold the old versions but APFS makes this OK because only the differences are stored, and if the data does not change a lot, like say an active SQL database would, then you can do fine for a long time with using two identical drives. (get rid of all exFAT file systems, use APFS. APFS does to things you want, checksums the metadata, can "checkpoint" the data (this is a zero-cost operation), and COW, Copy on Write.

People think data loss happens because a disk drive fails. Yes that might happen but the big causes are (1) user error, (2) software issues, (3) theft of the equipment, (4) lightning striking the power lines. Drives don't fail as often as these other cases.

You don't want RAID, file sync, or even periodic cloning of the data. You should be keeping a versioned history

Yup. Raid is about equipment uptime.

Backups are about data protection. Raid is not a backup technology and backups are potentially useless unless you have a copy off site (eg house fire).

 

[benwiggy]

Both setups are poor and don't offer good backup. Here is why? Lets say one file get corrupted, using either RAID or some kind of file sync, the corruption is copied to the second drive in real time.

Or indeed if the file is deleted.
As said, TM is perfectly good for directly connected drives. Effortless, automatic, and comprehensive. You can use third-party syncing apps -- some of them do provide versioning and archiving of deleted files, but it's usually more complex to set up and manage.

RAID isn't a backup.