APFS Bug in macOS High Sierra Can Cause Data Loss When Writing to Disk Images

[ikir]

High Sierra has been a complete disaster for me. My computer locks up for 5-10 seconds every few hours. Probably some memory leaks or something. Either way this is really pathetic, even windows works better than this.

I work with many costumers very day, no one have this issue. Probably it is a software you are using or even an hardware problem. Try to boot from an external new SSD and check if you have the issue again You could bring it in a support center / Apple Store, maybe it is just something you are loading in launchagents

 

[chucker23n1]

How about Dave Nanian’s SuperDuper? He took the more conservative approach (vs Bombich’s CCC) throughout this entire development ‘race’ to field an APFS bootable backup system without putting client data in jeopardy.

A disk image isn't meaningfully bootable anyways. This is a different use case.

I know SuperDuper! can also write to an image. I'm not sure right now if it uses sparse, though. If not, it's not affected.

 

[seanbperiod]

Finally vindication!

This is exactly what happened to me a week ago. I kept deleting files, but no space was being freed. I tried reinstalling macOS, but then it wouldn't boot. Fortunately I had a backup.

 

[chucker23n1]

Finally vindication!

This is exactly what happened to me a week ago. I kept deleting files, but no space was being freed. I tried reinstalling macOS, but then it wouldn't boot. Fortunately I had a backup.

That's a completely different scenario and isn't a bug. Deleted files are first moved to a snapshot for backup purposes.

 

[scousekenny]

Apple's APFS file system included in macOS High Sierra suffers from a disk image vulnerability that in certain circumstances can lead to data loss, according to the creator of Carbon Copy Cloner.

In a blog post last Thursday, software developer Mike Bombich explained that he had uncovered the data writing flaw in the Apple File System, or APFS, through his regular work with "sparse" disk images.

For those who aren't familiar with the term, a sparse disk image is basically a file that macOS mounts on the desktop and treats as if it was a physically attached drive with a classic disk volume structure. The flexibility of sparse disk images means they are commonly used in the course of performing backup and disk cloning operations, hence Bombich's extensive experience with them.
Two related problems are identified by Bombich, above. The first is that the free space on the APFS-formatted sparse disk image doesn't update as it should when the free space on the underlying physical host disk is reduced. The second problem is the lack of error reports when write requests fail to dynamically grow the disk image, resulting in data being "written" into a void. Bombich tracks both bugs back to macOS's background "diskimages-helper" application service, which he has since reported to Apple.

Bombich's video demonstrating the APFS bug​

Every installation of High Sierra converts the existing file system to APFS, which is optimized for modern storage systems like solid-state drives. However, as Bombich notes, ordinary APFS volumes like SSD startup disks are not affected by the problem described above, so the vast majority of users won't be affected by it - the flaw is most applicable when making backups to network volumes. Bombich says Carbon Copy Cloner will not support AFPS-formatted sparse disk images until Apple resolves the issue.

The APFS flaw follows the discovery of another bug in Apple's operating systems that received extensive coverage last week. That bug is induced by sending a specific character in the Indian language Telugu, which causes certain apps on iPhones, iPads, and Macs to freeze up and become unresponsive. The Telugu character bug has already been fixed in Apple's upcoming iOS 11.3 and macOS 10.13.4 software updates.

Article Link: APFS Bug in macOS High Sierra Can Cause Data Loss When Writing to Disk Images

[doublepost=1519058254][/doublepost]High Sierra, works fine with my Macbook pro (2016), not had one problem to date. everything is running as should be, files are backing up without any problems , all backup drives are APFS as is Mac Os, may it always be this way (hoping)

 

[nt5672]

Ok, come on. All of the Apple updates and the Key Note for High Sierra just looked awesome, what else matters?

Surely, no one expects a toy manufacturer like Apple to get the details right any more?

 

[LinkinJunior]

Apple this is a wakeup call to actually focus on releasing a bug free Mac OS.

 

[coolfactor]

Apple makes no mistakes. I love the notch, I embrace the notch. I think therefore I am, the notch.

Hey Notch, I've got the solution for you! Let's just remove the front-facing True Depth camera! Ready? Okay, see ya!

Now regarding this disk image bug... inexcusable for Apple to have missed this. Seems like a pretty basic test to perform. But if this has been fixed AND they knew about it when High Sierra was released, that's even worse!

 

[nt5672]

Get with it man.

If Apple gave us a choice to convert to APFS during the upgrade, this bug wouldn't be so bad. However, a forced conversion to APFS when it's not really ready for prime time simply stinks.

The problem is that Apple management thought it was ready for prime time so they forced it on their users. Either management did not listen, or did not care. Either way, it reflects on poor management. The engineers knew how thoroughly it was tested and they knew the risks. This is simply a reflection of Apple strategy to release on release day no matter the consequences. customers be dammed.

 

[Mac 128]

Not cool. Data loss is a very serious bug. I hope Apple releases the fix soon.

This is exactly what I was afraid of when they announced moving to APFS. And I waited months before I upgraded to make sure they worked out any serious bugs. Who knows now whether any of my converted data was corrupted by the process.

 

[BigJohno]

This release is a joke. I'm still on Sierra on my MacBook Pro. I'll stay there until the next release.

 

[iVoid]

Good, it seems it only affects APFS formed disk images and I haven't created any of those. HFS+ only, all created before High Sierra.

Of course, they are on old platter drives for backups and archived stuff, so wouldn't really benefit from APFS anyway.

 

[ravenstar]

That said, in everyday life hopefully people aren’t filling up drives.

Oh, if only that were true. Data has a way of filling available space and then some.

 

[EM2013]

Well at least we never have problems with our emojis.

 

[cmaier]

This is exactly what I was afraid of when they announced moving to APFS. And I waited months before I upgraded to make sure they worked out any serious bugs. Who knows now whether any of my converted data was corrupted by the process.

This is when you use sparse disk images, not when you create a volume. Has nothing to do with converting your boot drive.

 

[pat500000]

I'm glad i'm on sierra. Apple is telling me through telekinesis not to update.

 

[neliason]

Oh, if only that were true. Data has a way of filling available space and then some.

Yes, but you really should monitor your system. If you are getting close to filling a drive you need to make space. OSs don't do well with no disk space. Bad things will happen if you fill a disk. I had more in mind my iMac which has 0.5 TB free. But I do have a MacBook Air with a puny 128GB SSD that has to be monitored. And my backup drives have gotten dangerously close to full before. So I guess I made your point with my MacBook and backup drives!

 

[madmin]

If you have APFS formatted an external disk drive that you use to store spare bundle disk images to and the drive runs low on free space you will likely have issues. You might be doing that for example, if it's a big disk that you use to store backups from several computers. In which case you should keep a close eye on the free space. A better and safer option would be to create separate volumes for each backup and do normal backups to them. Since with APFS free disk space is shared between all the volumes, they will adjust as needed. No need for sparse images and less risk of corruption. It's probably faster too.

 

[name99]

"The first is that the free space on the APFS-formatted sparse disk image doesn't update as it should when the free space on the underlying physical host disk is reduced."

This is not a problem related to APFS, it is a problem with the Finder. It has been in Finder for at least ten years, and Apple appears to have zero interest in fixing it.

Specifically, Finder caches a huge amount of state about files and the file system (icons, size and names, what files are present in what directories, etc). Caching is a fine idea BUT is only acceptable if there is a cache invalidation protocol --- a way that ANY cached information gets flushed as soon as it has changed. Finder appears to have no such cache invalidation protocol and (judging by the fact that every year they ignore my bug reports) no interest in creating one.

You see similar phenomena in a dozen different ways. You'll see that Time Machine volumes frequently don't update either the free space they display in Finder, or new backup directories they've created. Likewise remote volumes (eg mounted via SMB) will not update to display changes that were made on the remote machine.
Basically ANY change made to the file system outside Finder has only a random chance of being noticed by Finder, meaning that practically all Finder data has a chance of being stale.

Windows used to have this problem; the main difference is that Windows KNEW and ADMITTED it had the problem, hence the infamous F5 key to update the display of an Explorer Window. Apple has gone backwards in this respect --- they used to not need an F5 key because they always maintained cached data properly. But over the past 10 years, they no longer maintain the cache properly, but won't admit or provide an F5 key.
It's a sad situation when your file UI is less robust and reliable than Windows 95...

 

[WatchFromAfar]

Probably it is a software you are using or even an hardware problem.

So the software Apple provides (the file system) is not a concern to you?

 

[jigzaw]

I'm trapped on Yosemite on my 2013 Macbook Pro because, while I'd be happy to upgrade to El Capitan or even Sierra if I determine that the apps I need for work will still function, I am given only one choice by Apple- High Sierra and there is no way in hell I'm risking my work computer on that.

I also have an iMac at home that I use to edit video, and it's a side business for me. It's on Sierra and for the foreseeable future it's staying that way because I have heard too many horror stories of apps that are simply unable to work on High Sierra and the fact that it reformats and re-writes all your data makes me worry about compatibility with other people and their OS's. Even if it's overblown, the perception that High Sierra just doesn't work is a problem.

 

[Zwhaler]

I blame the forced annual release cycle. Please, make it stop!
[doublepost=1519063743][/doublepost]

I'm trapped on Yosemite on my 2013 Macbook Pro because, while I'd be happy to upgrade to El Capitan or even Sierra if I determine that the apps I need for work will still function, I am given only one choice by Apple- High Sierra and there is no way in hell I'm risking my work computer on that.

I also have an iMac at home that I use to edit video, and it's a side business for me. It's on Sierra and for the foreseeable future it's staying that way because I have heard too many horror stories of apps that are simply unable to work on High Sierra and the fact that it reformats and re-writes all your data makes me worry about compatibility with other people and their OS's. Even if it's overblown, the perception that High Sierra just doesn't work is a problem.

You know it's pretty easy to get your hands on the Sierra / El Capitan installer and use it on another machine? Try the "purchased" section on the Mac App Store. If you've never downloaded it before, it is still pretty easy to obtain through searching.

 

[JMPATLANTA]

I wonder if this is related to high sierra suddenly invalidating my time machine backups to my NAS, and demanding to do a new full. (Loosing two years worth of history). Then doing it again 2 weeks later.

This really sounds familiar to the TimeMachine issue you illustrate here. I have been having this way before HS was released.

 

[springsup]

For a fully released filesystem, I don't think so. This is core to Apple's filesystem and should have been caught. I would say the same for any filesystem. If it's a public release, simple bugs like this should not exist.

But, hey, if you don't mind losing data, that's your prerogative.

It took quite a while for anybody to notice, and the one who did was a filesystems specialist running tests. That's not to say that nobody was affected, but it puts things in to perspective a bit.

I think most people here have no idea what a filesystem is or does, or how complex it is. For comparison, Microsoft have been trying to replace their filesystem for the last 6 years (https://en.wikipedia.org/wiki/ReFS). Read the wiki page, google a bit - you'll quickly find that it has a host of holes and is still not bootable. Apple deserve commendations for how quickly and smoothly their filesystem transition has been, despite this bug.

 

[LarryJoe33]

Mike is the man! Nice catch. Big time CCC user here and fully believe there is no better way to back up and restore quickly and accurately.