Copying Time Machine Disk - one more permission problem

[toke lahti]

I might have some corrupted backups in my TM drive.
And my TM does not always succeed to make a backup maybe because of this.
Anyway the drive is full, so I bought a double sized disk (8TB->16TB).

For my past experience, when I notice something is missing, it is usually acidentally deleted months ago.
So, I want to keep the old backups, but I also want to format the old drive to clear it's corruption and use it to something else.
I also want to use the new TM disk with older macs, so this means that both old and new disk are HFS+.
I guess that this should also speed up the copying since TM with APFS disk is a whole different beast than with HFS+.

For the discussions I've googled, I understood that Finder is the only way to do that.
First it takes about 30 hours for Finder to Find Out how many items it needs to copy (over 25M).
Then it starts to copy and hilariously tells that this will take about 30 days. Fun.
It is of course a bit faster but after about one day it had copied about 2.7TB of 7.8TB and tells me:

Anybody have experience with this?

Monterey changes TM disk to "read only" for the (super)user and now it can't even read it by itself.
Even after I have removed it from Time Machine Preferences to be used as a TM disk.
Ie. TM seems to not change the settings back, when the disk is removed from it's duties as a TM disk.

What next?

 

[gilby101]

TM seems to not change the settings back, when the disk is removed from it's duties as a TM disk.

Correct. Removing it from duties for backup does not change anything - in case you want to recover files as some point. If you want to re use the disk (and you have no further use for anything on it), you just need to erase it.

Are you are using one physical disk for the backup of many Macs and connecting it to each Mac as required. If so, I recommend a separate partition for the backup of each Mac. Those for macOS 11/12 Macs should be APFS whilst those for older Macs should be HFS+. I must admit I don't like that scenario, too many opportunities for corrupt backup (particularly on HFS+ partition) when disks are unmounted and removed. I would much rather have a TM disk for each computer.

Or do you have the single external disk permanently connected to one Mac and the others backup to it over the network?

 

[toke lahti]

Correct. Removing it from duties for backup does not change anything - in case you want to recover files as some point. If you want to re use the disk (and you have no further use for anything on it), you just need to erase it.

Are you are using one physical disk for the backup of many Macs and connecting it to each Mac as required. If so, I recommend a separate partition for the backup of each Mac. Those for macOS 11/12 Macs should be APFS whilst those for older Macs should be HFS+. I must admit I don't like that scenario, too many opportunities for corrupt backup (particularly on HFS+ partition) when disks are unmounted and removed. I would much rather have a TM disk for each computer.

Or do you have the single external disk permanently connected to one Mac and the others backup to it over the network?

One disk for multiple macs.
Was thinking that one disk is more efficient since it needs to have only one free space, that all macs are using. My dozen of external disks have so much space wasted...

But the idea of 2 partitions is good. HFS+ is no good for this long time archiveing.After (/ if) I get the old backups copied, shouldn't it be possible to convert the whole disk to APFS?
(And waste some space and use some other disk for the older macs...)

Or if I would format it before copying, does the copying still happen so that the incrementality of the backups are preserved?
Since, again, TM for HFS? is very different than TM for APFS.
If it is possible, would like to know how many weeks it would take...

 

[ChrisA]

For the discussions I've googled, I understood that Finder is the only way to do that.

If all you need to do is copy files, why not use "cp" from the terminal?

 

[Taz Mangus]

One disk for multiple macs.
Was thinking that one disk is more efficient since it needs to have only one free space, that all macs are using. My dozen of external disks have so much space wasted...

If you are going to use one drive for all backups, make sure you also do a secondary backup to a separate drive. If either the first or second backup drive fails, you won't loose everything that you have backed up.

 

[Taz Mangus]

View attachment 2011556
Anybody have experience with this?

Monterey changes TM disk to "read only" for the (super)user and now it can't even read it by itself.
Even after I have removed it from Time Machine Preferences to be used as a TM disk.
Ie. TM seems to not change the settings back, when the disk is removed from it's duties as a TM disk.

What next?

Have you tried enabling root user an then try to do the copying from the root user account?

 

[planteater]

Tread carefully with that backup. Any signs of corruption in TM backups is a serious sign for concern. And any attempt to move it could make matters worse. Consider creating a new archive.

I went through a similar situation a couple of years ago, and the result ended up being unreliable backup to or restores from my archive. Given that loss and considering how important backups are led me to getting a third party solution that is rock solid reliable.

I love TM functionality but have a hard time trusting it as a primary backup solution, secondary sure.

 

[ThrowerGB]

I was never able to successfully "copy" files from an old TM disk to a new one in a way that allowed new backups to be added to the old ones in a single Backups.backupdb. I eventually go it to work by using Disk Utility to create an Image of the old drive, copy that Image to the new drive, and then unpack it onto the new drive. I'm not sure which MacOS I was using at the time. I suspect I did it June 3, 2020 using whatever version of MacOS and Disk Utility was most recent at the time.
You'll want to keep the old drive and it's Image for a while until you're satisfied with the result. Actually I retired the old drive with it's Backups.backupdb. I have several of those retired drives taking up physical space on a shelf.

 

[toke lahti]

2nd try and always something new...

...well, that took only 40 hours...

 

[toke lahti]

If you are going to use one drive for all backups, make sure you also do a secondary backup to a separate drive. If either the first or second backup drive fails, you won't loose everything that you have backed up.

Yep. Done this for years. But I think now, that I need two secondary backups; hfs+ and apfs.
I never knew that hfs+ is so sensitive with big archives.

 

[gilby101]

I never knew that hfs+ is so sensitive with big archives

HFS has always been somewhat fragile. Use APFS for all new TM backups.

 

[toke lahti]

If all you need to do is copy files, why not use "cp" from the terminal?

Mainly, because it doesn't work.

Time Machine needs hard links in hfs+, which are not copied with cp.
If I would copy my 8TB TM drive, which has about 60 backups of about 1TB each, with cp, I'd need to have 60TB destination disk.

Does cp work with APFS Tm disk? I don't know...

 

[toke lahti]

HFS has always been somewhat fragile. Use APFS for all new TM backups.

I have a 10.11, 10.12 & 10.14 systems to backup with TM.

 

[toke lahti]

Tread carefully with that backup. Any signs of corruption in TM backups is a serious sign for concern. And any attempt to move it could make matters worse. Consider creating a new archive.

I went through a similar situation a couple of years ago, and the result ended up being unreliable backup to or restores from my archive. Given that loss and considering how important backups are led me to getting a third party solution that is rock solid reliable.

I love TM functionality but have a hard time trusting it as a primary backup solution, secondary sure.

I might have one corrupted backup of those 60 in that old disk. Maybe 2.
I don't know if they would infect the other backups?
I don't know the TM so deeply.
Does the hard links always point to

1. Last link = the chain goes through every backup
2. Last change = the chain goes through only few backups

...to the original version of that file.

@planteater, would you like to share what is your primary backup solution now?

 

[toke lahti]

Does anybody know, if HFS+ TM disk can be converted to APFS?

Or, can I copy my TM archive from HFS+ disk to APFS disk and then keep using that archive?

 

[maverick100]

This is what happened to me. When I went from Mojave to Monterey; I kept using the same TM disk (2tb ssd); Monterey at some point converted it to APFS. It seems to be working ok. I have not had to restore any files from pre Monterey. Just what happened to me.

 

[Taz Mangus]

You would be better off taking the pre-Monterey backup drives and just storing them away. Start a brand new backup of Monterey. Leave the old drives like they are, they will still be usable if every needed. This is what I did.

 

[gilby101]

Does the hard links always point to

1. Last link = the chain goes through every backup
2. Last change = the chain goes through only few backups

...to the original version of that file.

The links all point to the same file. A key thing with HFS TM disks is that it also uses hard links to directories.

Or, can I copy my TM archive from HFS+ disk to APFS disk and then keep using that archive?
We can only repeat so many times. Copying any TM archive is likely to cause trouble. Don't do it.

I will say it only one more time.

You would be better off taking the pre-Monterey backup drives and just storing them away. Start a brand new backup of Monterey. Leave the old drives like they are, they will still be usable if every needed. This is what I did.

^^^^^ DO THIS ^^^^^

 

[toke lahti]

This is what happened to me. When I went from Mojave to Monterey; I kept using the same TM disk (2tb ssd); Monterey at some point converted it to APFS. It seems to be working ok. I have not had to restore any files from pre Monterey. Just what happened to me.

Monterey secretely converted your TM disk from hfs+ to apfs?
Quite amazing.

 

[toke lahti]

The links all point to the same file. A key thing with HFS TM disks is that it also uses hard links to directories.

Or, can I copy my TM archive from HFS+ disk to APFS disk and then keep using that archive?
We can only repeat so many times. Copying any TM archive is likely to cause trouble. Don't do it.

I will say it only one more time.

^^^^^ DO THIS ^^^^^

You don't have to repeate.
That's not a good argument.

Do you think that Apple added the ability to copy TM archive with Finder to get in trouble?
Or to get out of a trouble?

"Links all point to a same file."

Does this mean that when I have 60 backups in my drive and then the backup #58 gets corrupted, then the #59 and #60 and any future backups won't work?

And same thing with APFS?

So the only thing to avoid corruption is to keep ALL the backups not corrupted?
One corruption will destroy all the following backups?

This would mean that you should use very small drives and stop using them very often?

 

[Taz Mangus]

You don't have to repeate.
That's not a good argument.

Do you think that Apple added the ability to copy TM archive with Finder to get in trouble?
Or to get out of a trouble?

"Links all point to a same file."

Does this mean that when I have 60 backups in my drive and then the backup #58 gets corrupted, then the #59 and #60 and any future backups won't work?

And same thing with APFS?

So the only thing to avoid corruption is to keep ALL the backups not corrupted?
One corruption will destroy all the following backups?

This would mean that you should use very small drives and stop using them very often?

If your backup is still in HFS+ format try a program call DiskWarrior. It is able to repair disk drives that may have the drive damaged. I have used it for over a decade. It does not work at present with APFS formatted drives. Maybe it is worth a try for you. It isn’t free and not cheap but it might help you.

 

[toke lahti]

You would be better off taking the pre-Monterey backup drives and just storing them away. Start a brand new backup of Monterey. Leave the old drives like they are, they will still be usable if every needed. This is what I did.

I bought an expensive 16TB drive to get the old 8TB drive to be used in something else.
This is economically quite a non efficient way that you can't copy things from a drive to another.

How much better I would be, if I wouldn't copy my backups to a new drive?

Let's say that there's 1% chance the backup gets corrupted when it's made and same when it's copied.
Then there is 99% chance that the backup is good after it's made and 98.01% chance that the backup is good after one copy. Big difference?

If the hard link from backup #60 points to a corrupted file in backup #58, it's lost regardless of copying it or not.
But this may be the reason why copying does not succeed. Maybe then the filesystem notices, that the file is corrupted and refuses to copy the rest 500 000 files.

 

[Taz Mangus]

I bought an expensive 16TB drive to get the old 8TB drive to be used in something else.
This is economically quite a non efficient way that you can't copy things from a drive to another.

How much better I would be, if I wouldn't copy my backups to a new drive?

Let's say that there's 1% chance the backup gets corrupted when it's made and same when it's copied.
Then there is 99% chance that the backup is good after it's made and 98.01% chance that the backup is good after one copy. Big difference?

If the hard link from backup #60 points to a corrupted file in backup #58, it's lost regardless of copying it or not.
But this may be the reason why copying does not succeed. Maybe then the filesystem notices, that the file is corrupted and refuses to copy the rest 500 000 files.

It is hard to know why something got corrupted on your backup. Try repairing the drive, then try the copy. I would suggest you do the copy as root user.

 

[toke lahti]

It is hard to know why something got corrupted on your backup. Try repairing the drive, then try the copy. I would suggest you do the copy as root user.

I have done First Aid for the disk every time there has been a problem.
When I first started getting messages last winter, that TM backup failed, I needed to do many of them, since one First Aid for the partition didn't succeed, after some 20-30 hours.
Then I did it first for the disk and then to the partition, and finally First Aid thought all is good.

Apple's support is also asking their engineers about the case, since Apple has removed their official support documents about copying TM to a new disk. This might be that they just don't support it any more or there are some problems with the system or the documentation.

How do you use Finder as a root user?

 

[toke lahti]

A side note:
Just 2 years ago TM worked only with hfs+. Like it had worked with for 13 years.

So I don't fully ingest the idea that hfs+ is bad for TM "all the time".

Types of disks you can use with Time Machine on Mac

Use Time Machine on your Mac with a Time Capsule and with USB, FireWire, and Thunderbolt disks.

support.apple.com