Big Sur finally uses APFS snapshots for Time Machine backups!

zorinlynx

macrumors 603
Original poster
May 31, 2007
6,029
8,505
Florida, USA
I've been waiting for this feature a long time and I'm so happy to finally see it! Using APFS snapshots for backups is so much faster and more efficient than the old way of walking the entire filesystem and using hard links. I did some tests backing up my test installation and it just flies.

I'm a UNIX sysadmin and work extensively with ZFS and using ZFS snapshots for backups, and this is basically the same thing. I think this is one of the reasons Apple considered using ZFS so many years ago, but then decided to implement their own. Backups are going to be MUCH faster, especially deltas with lots of small files. No more TM sitting there for 20 minutes at "Backing up 646MB of 650MB" with the disks cranking away.
 

zorinlynx

macrumors 603
Original poster
May 31, 2007
6,029
8,505
Florida, USA
So presumably it’s converted your TM drive to APFS (as opposed to forcing it to be hfs+ previously)?
I haven't tested that. I'm using a completely separate test machine and am backing up to an APFS disk.

I should try a backup to an HFS+ disk to see if it still backs up the old way for that.
 

casperes1996

macrumors 601
Jan 26, 2014
4,901
2,762
Horsens, Denmark
I haven't tested that. I'm using a completely separate test machine and am backing up to an APFS disk.

I should try a backup to an HFS+ disk to see if it still backs up the old way for that.
I have Big Sur but am not going to use Time Machine with it, but I’m curious what your findings will be. If it will convert the drive or just continue the old way
 

bdart2

macrumors newbie
Oct 29, 2014
28
0
With the news that Big Sur will support Time Machine in APFS - I had the following issue arise today.

I just bought a new MBP a few weeks ago and today bought a WD easystore 8 TB HDD for use as both a time machine and to store pictures and videos - one partition for TM and the other for other files.

I was just about to reformat and started looking at the format types and saw APFS as the preferred format for newer OSs. Then I saw that Time Machine is not compatible with APFS, but would be in Big Sur.

Question which one should I do:

1.) Format the new HDD in MacOS Journaled and just partition part of the drive for Time Machine and the other for files?

or


2.) Format an old spare drive to MacOS journaled and use it as a TM backup now and now Format the new HDD in APFS and partition the drive as above and wait to move the TM backups from the old drive to the new drive once I install Big Sur later this year?

thanks in advance
 

Taz Mangus

macrumors 601
Mar 10, 2011
4,003
602
With the news that Big Sur will support Time Machine in APFS - I had the following issue arise today.

I just bought a new MBP a few weeks ago and today bought a WD easystore 8 TB HDD for use as both a time machine and to store pictures and videos - one partition for TM and the other for other files.

I was just about to reformat and started looking at the format types and saw APFS as the preferred format for newer OSs. Then I saw that Time Machine is not compatible with APFS, but would be in Big Sur.

Question which one should I do:

1.) Format the new HDD in MacOS Journaled and just partition part of the drive for Time Machine and the other for files?

or


2.) Format an old spare drive to MacOS journaled and use it as a TM backup now and now Format the new HDD in APFS and partition the drive as above and wait to move the TM backups from the old drive to the new drive once I install Big Sur later this year?

thanks in advance
I am not sure I would use the same drive for backups and storing other valuable items. If the drive fails, for whatever reason, you have lost everything.
 

JimmyjamesEU

macrumors member
Jun 28, 2018
39
12
I've been waiting for this feature a long time and I'm so happy to finally see it! Using APFS snapshots for backups is so much faster and more efficient than the old way of walking the entire filesystem and using hard links. I did some tests backing up my test installation and it just flies.

I'm a UNIX sysadmin and work extensively with ZFS and using ZFS snapshots for backups, and this is basically the same thing. I think this is one of the reasons Apple considered using ZFS so many years ago, but then decided to implement their own. Backups are going to be MUCH faster, especially deltas with lots of small files. No more TM sitting there for 20 minutes at "Backing up 646MB of 650MB" with the disks cranking away.
Hmmm, not so much. It's true that apfs disks can now be used for Time Machine, and it's also true that snapshots are created on both the source and target volumes...

However, TM still isn't using a block based backup I'm afraid. It's easy enough to verify this. Do an initial backup (which is pretty fast), then change a large file just a little (for example, I created a 10 gig file before the first backup, then added 10 megs to it). Do a subsequent backup and you will see that the entire 10 gig + 10 meg file is recopied. Not blocks unfortunately, just file copies.

It's still an early beta, so perhaps that will come later. It is also true to say that they have improved performance of TM overall. Still, disappointing that block based backups still aren't here.
 
  • Like
Reactions: NazgulRR

zorinlynx

macrumors 603
Original poster
May 31, 2007
6,029
8,505
Florida, USA
Hmmm, not so much. It's true that apfs disks can now be used for Time Machine, and it's also true that snapshots are created on both the source and target volumes...

However, TM still isn't using a block based backup I'm afraid. It's easy enough to verify this. Do an initial backup (which is pretty fast), then change a large file just a little (for example, I created a 10 gig file before the first backup, then added 10 megs to it). Do a subsequent backup and you will see that the entire 10 gig + 10 meg file is recopied. Not blocks unfortunately, just file copies.

It's still an early beta, so perhaps that will come later. It is also true to say that they have improved performance of TM overall. Still, disappointing that block based backups still aren't here.
Did you verify that the entire 10GB file is being copied though? Maybe TM just tells you 10GB basing that information on what's changed?

Now I feel like testing this myself but unfortunately my test Macbook is at the office. Argh.
 

JimmyjamesEU

macrumors member
Jun 28, 2018
39
12
Did you verify that the entire 10GB file is being copied though? Maybe TM just tells you 10GB basing that information on what's changed?

Now I feel like testing this myself but unfortunately my test Macbook is at the office. Argh.
I got the disk size after the first backup and then after the subsequent backups. Each time the space had been consumed the total size of the file, as indicated by the Finder, disk utility, and via command line. I also used the 'asr' command line tool that does use block based transfer. The difference was noticeable. Block based initial backup was 4 minutes for a 30 gig disk. Subsequent delta transfers were around 10-20 seconds. Time machine was fast, but not nearly as fast as the block based method.
 

chama98

macrumors regular
Feb 13, 2014
185
68
London
so, tell me.. Did anyone have their time machine drive converted automatically to APFS from HFS+? I backup to a NAS drive, so wondering if I need to migrate backups to a external thunderbolt drive or something.
 

yangm

macrumors member
Apr 16, 2014
31
33
so, tell me.. Did anyone have their time machine drive converted automatically to APFS from HFS+? I backup to a NAS drive, so wondering if I need to migrate backups to a external thunderbolt drive or something.
Mine is on a NAS (SMB3, zfs backed), it didn't (nor did it offer to) get converted into APFS.


Does this work if you back up to a NAS?
I've created another share, so it creates another backup, on my NAS (same as above) and it does default to a APFS sparsebundle.
1593356115758.png
 

donawalt

macrumors 6502
Sep 10, 2015
459
197
So it sounds like you will have to start with a freshly formatted disk to use Time Machine with APFS...is that correct?
 
  • Like
Reactions: me55

yangm

macrumors member
Apr 16, 2014
31
33
Seems like it, at least for now. Though that's coming from the very first developer beta... There's still a possibility that the final release is going to introduce some upgrade path.
 
  • Like
Reactions: donawalt

zorinlynx

macrumors 603
Original poster
May 31, 2007
6,029
8,505
Florida, USA
I got the disk size after the first backup and then after the subsequent backups. Each time the space had been consumed the total size of the file, as indicated by the Finder, disk utility, and via command line. I also used the 'asr' command line tool that does use block based transfer. The difference was noticeable. Block based initial backup was 4 minutes for a 30 gig disk. Subsequent delta transfers were around 10-20 seconds. Time machine was fast, but not nearly as fast as the block based method.
Ugh, this is frustrating! Why would they half-ass the concept? My only theory is maybe they don't full trust the asr incremental stream code yet? I know Apple has to be super conservative with software they deploy to millions of users, but it's already been multiple OS releases so I'm surprised they wouldn't have the kinks worked out.

Either way, at least using snapshots is most likely more reliable than the crazy chains of hard links they were using before. One step at a time I suppose.
 
  • Like
Reactions: yangm

mikecwest

macrumors 65816
Jul 7, 2013
1,015
368
I hope that using APFS will make it faster on external SSD backup drives. TM is so dang slow.
 

JimmyjamesEU

macrumors member
Jun 28, 2018
39
12
Ugh, this is frustrating! Why would they half-ass the concept? My only theory is maybe they don't full trust the asr incremental stream code yet? I know Apple has to be super conservative with software they deploy to millions of users, but it's already been multiple OS releases so I'm surprised they wouldn't have the kinks worked out.

Either way, at least using snapshots is most likely more reliable than the crazy chains of hard links they were using before. One step at a time I suppose.
Yeah, I get your frustration believe me! I can only assume it's a work in progress. They haven't talked about it at WWDC or on the website at all, so they know it's not enough to boast about. The current method is probably the most flexible because it works with both hfs and apfs.

Also worth considering, currently macOS allows only 24 hourly snapshots. For delta snapshots to work, the previous snapshot must exist on both systems. If a user were to wait a week or so between backups, no deltas would be possible. The way other os's with cow filesystems deal with this is to have way more than 24 hourly snapshots. This could easily be done on macOS... but the problem here is mac users! I think apple is worried about users complaining that "all their space is being used up". I have seen hundreds of posts like this. Complaining, whinging about snapshots, where users on linux, freebsd etc know and appreciate the immense benefits they provide. Its a case where macOS is being held back by the ignorance of it's user base.

One good thing, is there are people like John Siracusa with knowledge and interest in apfs. He just asked about Time machine and delta backups on Twitter. Hopefully he can have some influence.
 

casperes1996

macrumors 601
Jan 26, 2014
4,901
2,762
Horsens, Denmark
Yeah, I get your frustration believe me! I can only assume it's a work in progress. They haven't talked about it at WWDC or on the website at all, so they know it's not enough to boast about. The current method is probably the most flexible because it works with both hfs and apfs.

Also worth considering, currently macOS allows only 24 hourly snapshots. For delta snapshots to work, the previous snapshot must exist on both systems. If a user were to wait a week or so between backups, no deltas would be possible. The way other os's with cow filesystems deal with this is to have way more than 24 hourly snapshots. This could easily be done on macOS... but the problem here is mac users! I think apple is worried about users complaining that "all their space is being used up". I have seen hundreds of posts like this. Complaining, whinging about snapshots, where users on linux, freebsd etc know and appreciate the immense benefits they provide. Its a case where macOS is being held back by the ignorance of it's user base.

One good thing, is there are people like John Siracusa with knowledge and interest in apfs. He just asked about Time machine and delta backups on Twitter. Hopefully he can have some influence.
To be fair the issue with uninformed users is always the responsibility of the creator who failed to properly inform them.
There's no easy storage management section with a GUI showing snapshots and their storage usage and such. It's fine for us who just pop the Terminal, but instead of just saying that System is using over a hundred gigs, users would be more able to understand if the storage management section you can get to with About This Mac actually showed more information about Snapshots and allowed you to control it without needing to use diskutil ap snapshots /
 

yangm

macrumors member
Apr 16, 2014
31
33
I guess much of the frustration comes from: we've waited so long... for this? Given they're pretty much reusing the same machinery which powers the current HFS+ replication, why didn't they do this earlier? Or, since we've waited for so long, why isn't this a block based snapshot replication like asr has been doing?
 
Last edited:

JimmyjamesEU

macrumors member
Jun 28, 2018
39
12
To be fair the issue with uninformed users is always the responsibility of the creator who failed to properly inform them.
There's no easy storage management section with a GUI showing snapshots and their storage usage and such. It's fine for us who just pop the Terminal, but instead of just saying that System is using over a hundred gigs, users would be more able to understand if the storage management section you can get to with About This Mac actually showed more information about Snapshots and allowed you to control it without needing to use diskutil ap snapshots /
I definitely agree that the tools could and should be better. Space should be accurately reported and consistent. I'm not sure I agree that users have no responsibility to learn and educate themselves, but tbh I don't care enough to argue the point!

Some (hopefully) promising news. If you run 'strings' on the backupd utility deep in the bowels of Big Sur, you can see references to diffing snapshots and delta blocks, so it seems it's something they're working on at least.
 

casperes1996

macrumors 601
Jan 26, 2014
4,901
2,762
Horsens, Denmark
I definitely agree that the tools could and should be better. Space should be accurately reported and consistent. I'm not sure I agree that users have no responsibility to learn and educate themselves, but tbh I don't care enough to argue the point!

Some (hopefully) promising news. If you run 'strings' on the backupd utility deep in the bowels of Big Sur, you can see references to diffing snapshots and delta blocks, so it seems it's something they're working on at least.
Users should educate themselves, yes. I agree with that too. But macOS is a platform used by a wide variety of people and you can't expect Joe Bluepants to investigate things that are completely irrelevant to the way he uses his computer without even any indication from the system that there is something to be investigated in the first place. It just won't let him save his new photo to the computer, saying it's out of space even though he hasn't saved that many things, and apparently system eats up a lot of space. I don't think it's unreasonable for Joe Bluepants to then just complain about macOS eating up all his disk space. There's nothing that tells him to look into snapshots and a fix for his issue. There's just "system eating space". And if you tell him to use Terminal, he'll rightfully look at you like you're suggesting the dark arts, because we, the software developers, have been making GUIs precisely for average Joe Bluepants, telling them not to be scared of computers. It clearly looks like it's a problem that the system is eating up his space to him, and not a helpful feature. It becomes the user's responsibility as soon as the information has been presented to them. It's OK to get pissy with someone you see every day if they never remember your name. It's not OK to be pissy with them the very first time when you've not even told them your name yet. It's the user's responsibility as soon as they've been given the tools to know it, but not before that. Before that it's our responsibility (our referring to developers and creators).
That's my opinion anyway :)

But yeah that does sound promising
 
  • Like
Reactions: mikecwest

mikecwest

macrumors 65816
Jul 7, 2013
1,015
368
Users should educate themselves, yes. I agree with that too. But macOS is a platform used by a wide variety of people and you can't expect Joe Bluepants to investigate things that are completely irrelevant to the way he uses his computer without even any indication from the system that there is something to be investigated in the first place. It just won't let him save his new photo to the computer, saying it's out of space even though he hasn't saved that many things, and apparently system eats up a lot of space. I don't think it's unreasonable for Joe Bluepants to then just complain about macOS eating up all his disk space. There's nothing that tells him to look into snapshots and a fix for his issue. There's just "system eating space". And if you tell him to use Terminal, he'll rightfully look at you like you're suggesting the dark arts, because we, the software developers, have been making GUIs precisely for average Joe Bluepants, telling them not to be scared of computers. It clearly looks like it's a problem that the system is eating up his space to him, and not a helpful feature. It becomes the user's responsibility as soon as the information has been presented to them. It's OK to get pissy with someone you see every day if they never remember your name. It's not OK to be pissy with them the very first time when you've not even told them your name yet. It's the user's responsibility as soon as they've been given the tools to know it, but not before that. Before that it's our responsibility (our referring to developers and creators).
That's my opinion anyway :)

But yeah that does sound promising

I agree, most end users would not ben know what any of that means. I like the old school Mac logic, that is "just works."
 

gilby101

macrumors 6502
Mar 17, 2010
314
125
Tasmania
They haven't talked about it at WWDC or on the website at all, so they know it's not enough to boast about.
My interpretation:
Apple would not want boast about a feature that might or might not be in the final release. My guess is that Apple are wanting feedback from beta testers before deciding to include it. Remember 2 years ago there was APFS on Fusion drives in the beta, but not in the release.
 

zorinlynx

macrumors 603
Original poster
May 31, 2007
6,029
8,505
Florida, USA
Convert to APFS is greyed out for my TimeCapsule, but the option to Erase it to APFS is there. Is it safe to do so?
Make sure you have an additional backup of your system before wiping the current backup.

It would really suck if you wiped your current backup then had a storage failure while creating a new backup.

Of course if you're just on a test system this is moot.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.