APFS Bug in macOS High Sierra Can Cause Data Loss When Writing to Disk Images

[Greg3523]

That’s not true. I migrated a RAID6 array consisting entirely of spinning HDD to APFS. Have also migrated several standard HDDs.

The RAID migration, for the record, was a bad idea. I get kernel panics routely, and have heard from others that RAID + APFS + encryption (enabled on my drive) is causing other users to see kernel panics. The panic logs on my machine point toward APFS and encryption, and the crashes only occur when using that drive (which was stable before migrating to APFS).

I too would be surprised. First, APFS formats only on drives that are SSD. It won't format on Fusion drives, nor will it format on standard platters. So the only people who will encounter this bug are those who have an all-SSD drive who try to create a sparse disk image on it. Yes, it's a serious bug (which Apple has patched for the next update), but it would be evident only in a limited number of special case uses.

 

[WatchFromAfar]

Actually, not. I don't say it's GUI-simple; but you CAN avoid APFS if you REALLY want to...

https://en.wikipedia.org/wiki/Apple_File_System#Support
[doublepost=1519083387][/doublepost]
And, you can actually avoid it...

https://en.wikipedia.org/wiki/Apple_File_System#Support

I like your terminology, "you CAN avoid APFS" yes you can but that's not the default way of doing things. just like anything other than NTFS is the default for Microsoft and they didn't spend six years pushing the successor down your throat.
[doublepost=1519107375][/doublepost]"Why would saying alpha/beta testing make me feel good?" because you say "alpha/beta testing" in your post. Really at which point do you think the Homepod was only alpha tested? Then when do you think it was beta tested? Please tell us when do you actually think the Homepod was, honest to God, actually tested.
[doublepost=1519107824][/doublepost]

It is absolutely comparable. Both NTFS and HFS+ were aging systems, stretched far beyond their expected use-cases in modern environments. Don't try to pretend that anybody loved HFS+. Apple pushed APFS out to hundreds of millions of iOS devices in a minor update. I haven't heard of any issues. They did the same to all of the millions of macOS devices out there, and this is the first time I've heard of a significant bug. All software has bugs; they get shaken-out over time by reports like this from real users. The kinds of real users Microsoft doesn't have testing ReFS.

About this bug: it's well-defined, easily reproducible and fixable with a driver update. It would be 1000x worse if there was a poorly-defined, hard-to-reproduce issue which was, for example, writing corrupted metadata for your original files. This is simply a silent failure which should instead produce an error. The consequences, of course, can be severe, but the bug itself is simple to fix and fairly limited in scope. What percentage of Mac users do you think are even in danger of encountering this bug? Maybe 5%?

These kinds of bugs are perfectly normal in any software. What is new is the level of publicity they get.

I think you've missed the point "all software has bugs" yes but Microsoft is not pushing out a new file system to their billions of users, it's successor may have bugs but they're not pushing it on users until it's fixed, Apple is pushing out it's new file system regardless of bugs.

 

[bartvk]

the bug is particularly evil if you use a backup tool, because that tool is particularly likely to use sparse disk images.

But which backup tool would that be? No backup tools currently exist that use APFS sparse images on network shares.

Note I'm not denying this is a bug -- it most obviously is.

 

[chucker23n1]

[doublepost=1519111010][/doublepost]

But which backup tool would that be? No backup tools currently exist that use APFS sparse images on network shares.

Note I'm not denying this is a bug -- it most obviously is.

Well, CCC, for one. Plus solutions people may have rolled on their own.

 

[DeepIn2U]

Every news item that comes out is like a feeding frenzy for the piranha

I'm just a minow trying to get a nibble

Well placed statement to be honest and I too have fallen for the bait. Human nature??

 

[iPaintCode]

Steve Jobs once said, "Real artists ship."

Was it Tim Cook who said Apple still has Steve's D.N.A.?

Maybe he confused that statement with, "Real artists ship alpha products.".

(I say that somewhat sarcastically)

 

[fermat-au]

But opt to have security vulnerabilities instead.

Security update should be separate from feature updates and new versions. This is in fact a weakness of the Mac. There are many very large and security organisation still running Windows 7, as Microsoft ships security update for Windows 7.

 

[drct]

I'm truly fed up with companies using regular customers as beta-testers.

 

[fermat-au]

We are not talking about a day or even a week or even a month on this.

I probably should have added a year to a day, a week or even a month

 

[AlloftheCoffee]

High Sierra has been a complete disaster for me. My computer locks up for 5-10 seconds every few hours. Probably some memory leaks or something. Either way this is really pathetic, even windows works better than this.

Same. My Fusion drive has been UN-Fusioned (sp?) and I can no longer let the computer go to sleep or it will call the blinking folder question mark of death. Just an all-around bad deal over here.

 

[BritishApple]

You read about more bugs than rumours on here these days. Who is the hell is in charge of QA at Apple?

 

[Bacillus]

You read about more bugs than rumours on here these days. Who is the hell is in charge of QA at Apple?

The problem with rumors about bugs is that they're unmistakenly true, generally
With the apparent lack of a single QA responsible, my best guess would be a guy named T. Cook.

The more sensible answer would be that Apple is such a complex company, that QA has to be implemented inside employee's heads from the bottom to the top. This requires a kind of professional stubbornness and rigorousness that Cook doesn't possess - let alone that he can implement it...
Same for the billionaire Board members that did anything besides staying hungry, lean or mean (as Steve asked them)

 

[gmavrikas]

I wonder if this is related to high sierra suddenly invalidating my time machine backups to my NAS, and demanding to do a new full. (Loosing two years worth of history). Then doing it again 2 weeks later.

Network TM backups are using a sparse volume, so from a first read it seems plausible.
AFAIK though, TM does not allow backing up to an APFS target, so your sparse volume is probably HFS+ formatted.

 

[Bacillus]

Kudo's to Apple for their transparency, immediate acknowledgement and precise description of the bug, instructions how to avoid it, and immaculate action to prevent customers' loss of data.
This is how the World's biggest IT-company supports its customers in the highest order (RED) emergency
(snore, snore, .... postPC era...snore..

 

[ravenstar]

That was changed before High Sierra shipped, wasn't it? It's supposed to be exclusive to SSD for now.

Nope, I installed it on an external drive from the official release. It's just not the default option (or necessarily recommended.) Don't do it on a drive you want to use for your Time Machine backups, though....

 

[ventricle]

CCC is the best! - No need for RAID - just copy one external drive to another daily.

It's the Apple Way or the Highway, which is why not all follow the latest, un-greatest. Until CCC gives me a green light, I'm staying away from new hardware, which as a multi-Mac Pro user, I've had to do anyway for I-can't-even-count-how-long-now!

 

[macguru212]

Happy to be a paying CCC user; thanks!

It's a beautiful piece of software!

 

[springsup]

I think you've missed the point "all software has bugs" yes but Microsoft is not pushing out a new file system to their billions of users, it's successor may have bugs but they're not pushing it on users until it's fixed, Apple is pushing out it's new file system regardless of bugs.

As I've said elsewhere, I disagree. I consider the APFS rollout to be one of Apple's great engineering successes, together with the PPC/Intel and 32/64-bit transitions.

The truth is that APFS has been used for critical data storage for hundreds of millions of people for about a year. We've found one minor bug. Even if/when Microsoft actually ships a version of ReFS which they recommend as the default for critical data storage and booting, it will not have zero bugs. Again, if they do ever ship, Microsoft's ReFS team will be praying that an issue like this is the biggest thing they find after an entire year.

When you hear people talk about "filesystem bugs" as being so awful, they typically refer to issues which can corrupt your data. This isn't one of those kinds of bugs. The copy operation should (and does) fail; the bug is that it doesn't tell you about it, so you might think it was successful. You could easily work-around it by opening the disk image and validating the copied file (which, one might argue, is a good idea anyway for a mission-critical backup).
[doublepost=1519140535][/doublepost]

Most people have been using HFS+ for over a decade with no problems, and they don't even know what it is. You make it sound like it's been causing issues for average users. The only frustration I can think of is sizing a very large directory.

I don't mean to imply that at all; my point is that HFS+ has been used by real users every day for 20 years, and maintained by Apple's engineers over that same time. Of course they have had plenty of time to investigate every minor edge-case. HFS+ 1.0, released in 1998, certainly had issues which were fixed in later updates.

APFS has lots of benefits over HFS+, and I'm yet to hear of an actual, serious bug.

Everybody is quick to shout at Apple's engineers for every minor issue; really, if people understood the complexity of these systems and looked dispassionately at the overall results of Apple's efforts, I think they'd come away very impressed.

 

[adream]

APFS is a time consuming dangerous roll out

to have no choice if you run a ssd system is ridiculous.

trashed two machines here

don't force beta level software on customers especially in such a critical system.

 

[DeadSeaMac]

Speaking of problems with APFS - I had to revert (using CCC actually) all the systems I had upgraded to High Sierra back to the old Mac OS Extended format due to the whole "not releasing disk space" bug which certainly should have been taken care of before High Sierra went GM. I have numerous users on limited-space SSD equipped MacBook Airs and MacBook Pros (education environment - staff laptops), so they know how to manage disk-space by offloading old files to network storage to free disk-space, and it just wasn't working; leaving users unable to work on large projects (i.e. importing large video-files for use in iMovie, etc.) The systems they deleted 30gb or 40gb of files from would throw up the "not enough free space" errors when trying to copy new files to the drive, even though those were generally far smaller than the space that should have just been freed-up. Unbelievable. Oh, wait, unbelievable for the old Apple we knew and loved. Totally believable, even expected, from the new Apple.

However, the upshot is that the systems running High Sierra on Mac OS Extended also seem to be far less buggy overall than when they were running on APFS, so if you are experiencing tons of High Sierra unreliability and have the know-how & time, you might try cloning the system, reformatting the drive back to GUID and Mac OS Extended (journaled) and then cloning everything back. Seemed to help me, but that's an admittedly limited-data set (around a dozen system to date).

Remember that on High Sierra, SSDs are automatically converted to APFS as the default file system. Your staff with SSDs using High Sierra had no choice - you should have known that. And if you did, then you should have taken your own advice and tested it first before downloading High Sierra on your Staff's SSD Macs. Assume nothing as an IT Manager. Going back to Mac OS Extended, you had to downgraded to Sierra with those SSD Macs.

 

[9081094]

And this why I will always prefer Windows, "it just works"

Lol, this post made me laugh. I know you meant it sarcastically but it's true, sometimes you wish you could switch... unfortunately how tempting the other side might look, I still prefer the look and feel of Mac OS. However Windows 10 is very stable these days and all programs from Adobe run circles around the mac versions.
[doublepost=1519145461][/doublepost]

Apple should have stuck with ZFS years ago. I'll stay on El Capitain for a number of years because it works on most modern and previous Apple hardware.

Has Apple ever used ZFS out in the open? Can't remember.... the only thing I remembered that ZFS wasn't ready for prime time too.

 

[ApfelKuchen]

I wonder if this is related to high sierra suddenly invalidating my time machine backups to my NAS, and demanding to do a new full. (Loosing two years worth of history). Then doing it again 2 weeks later.

Not too likely. Time Machine backups to NASes have had this issue for a long time, regardless of the file system being backed up. It's been happening on backups of my HFS+ formatted, spinning HDD-equipped Early 2008 iMac for quite some time. I've been putting up with it because backup history for that particular machine means nothing to me - I just need crash protection. My primary machine backs up to an attached HDD for this reason.

 

[skinned66]

Glad I don't use sparse disk images.

 

[panjandrum]

Remember that on High Sierra, SSDs are automatically converted to APFS as the default file system. Your staff with SSDs using High Sierra had no choice - you should have known that. And if you did, then you should have taken your own advice and tested it first before downloading High Sierra on your Staff's SSD Macs. Assume nothing as an IT Manager. Going back to Mac OS Extended, you had to downgraded to Sierra with those SSD Macs.

Incorrect. You just have to clone, reformat, clone back. High Sierra runs just fine on SSDs which are not APFS. Wish Apple simply provided an option NOT to convert to APFS; it would be easier. I did of course do some testing of High Sierra first (including upgrades on all my personal systems), but because this issue doesn't become apparent until a user nearly fills a drive, then attempts to delete files to reclaim some available free-space, it isn't a problem that would become apparent through normal testing.

 

[chucker23n1]

Has Apple ever used ZFS out in the open? Can't remember.... the only thing I remembered that ZFS wasn't ready for prime time too.

Apple had a team of ZFS engineers for a few years.

10.5 shipped with read-only ZFS support, with a beta version for read-write support. For 10.6 Server, read-write support was originally announced, but dropped before release. There was even an Apple-hosted open source project.

It was weird. Clearly not the way they had planned.
[doublepost=1519151781][/doublepost]

Wish Apple simply provided an option NOT to convert to APFS; it would be easier.

Does --converttoapfs NO no longer work?