Apple to Drop Support for Encrypted Mac OS Extended Drives Next Year

[extrachrispy]

That's not true. APFS is a more modern filesystem with many improvements over HFS+, but HFS+ had numerous optimizations for spinning disks which are still relevant to those who use them (and which were more cost effective at large sizes and even more so with the spike in SSD prices).

This is an interesting claim: that HFS+ performs better on spinning disks than APFS. I would be interested in seeing data and, if one exists, an explanation as to why.

The elevator algorithm is ancient. I would be surprised if APFS did not simply use what already exists.

By far, the biggest optimization for spinning disks was invented thirty-four years ago: the log-structured file system. For whatever reason, it was not widely adopted.

 

[extrachrispy]

An easy way to see the performance issue is to compare file enumeration in a search query with a lot of results. It takes much longer on APFS.

An analysis of APFS enumeration performance on rotational hard drives

My APFS-formatted rotational disks have always felt slower than when they were HFS+ formatted. The speed of copying files to them felt about the same, bu...

bombich.com

That's very interesting; however, it's also seven years old. It would be interesting to see it repeated today.

Bombich's explanation as to why is certainly plausible: directories being scattered all over the platter rather than in a pre-allocated location would indeed lead to terrible problems with seeking and rotational latency. A nit: he describes file sizes in his article; they are, however, completely irrelevant to the speed of enumeration. All you are doing when enumerating (unless you are also doing a stat on each file) is reading directories.

 

[IvyKing]

Totally with you on the frustrations regarding how APFS falls short of even being "ZFS-lite". I had high hopes that Apple would adopt ZFS like its upstream parent FreeBSD did, but then they rolled their own.

IIRC, MacOS 10.6 had ZFS read capability, which was removed in the next release. The scuttlebutt was this was done because Jonathan Schwarz (Sun's CEO at the time) had upstaged Jobs about ZFS being included in OS X. I was rather disappointed when that happened as ZFS had the capabilities of "Time Machine" built in. Another nice aspect of ZFS was reading infrequently accessed sectors on the disk to check for soft errors and moving files to new sectors when too many soft errors (errors that were recoverable) occurred on a give sector.

 

[djplong]

"Undocumented", "hard to clone" and "inaccessible APIs" sound like they're also (maybe accidental) security features to me.

Yeah, Apple is being a bit of a gatekeeper here, no argument there. But as someone who's job included having to "think like a black hat", the fewer ways into a system, the fewer access points you had to worry about.

 

[JimenyC]

For single drive encrypted storage on HDD, you still can continue using them but with APFS. For incidental writes and reads, it will be fine but not great for frequent writes. You could also use a AFPS DMG or Sparse Image with encryption, although it won't be very fast as well.

Performance tests here.

Thousands of dollars? 16 or 24 bay units are that expensive. Four or five bay NAS without disks costs around 400-700 dollar, depending on brand, model and options of course. Mine is an eight bay rackmounted model (RS1221+). Bought it 3 years ago without disks for around 1100 euro with tax. Migrated 5 existing HDD's one by one from previous enclosures to this NAS, leaving 3 bays for future expansion. One doesn't always have to buy a NAS with new disks installed. I still use a couple that are nearly 10 years old and still perform superb.

A sparse image is no good for my needs. You're right about the cost of a 4-5 bay NAS without disks. But one does have to incorporate the cost of the disks (none of my existing external drives are suitable). By the time you've done that, you're into the thousands.

 

[extrachrispy]

IIRC, MacOS 10.6 had ZFS read capability, which was removed in the next release. The scuttlebutt was this was done because Jonathan Schwarz (Sun's CEO at the time) had upstaged Jobs about ZFS being included in OS X. I was rather disappointed when that happened as ZFS had the capabilities of "Time Machine" built in. Another nice aspect of ZFS was reading infrequently accessed sectors on the disk to check for soft errors and moving files to new sectors when too many soft errors (errors that were recoverable) occurred on a give sector.

There's way more than that, but yes. ZFS solves the URE (Unrecoverable Read Error) problem in a different way than does object storage such as Panasas's (now VDURA's) system.

Primer for those who haven't heard of the URE problem: in a RAID volume of sufficient size, the probability of having an unrecoverable read error when reconstructing the volume approaches 1 when the volume is large enough. For enterprise-grade spinning drives, this threshold is about 10TB. This means you have lost a disk and are reconstructing from parity plus the remaining disks: you fail to read a sector, and now you cannot reconstruct it. Traditional RAID offerings hard-fail the reconstruction at this point and you lose the entire volume. You will not know that a sector has gone bad until you do reconstruction, so you have a ticking time bomb on your hands.

Object storage solves this by doing RAID across the individual objects--think of them as component blobs that when assembled form a complete file or directory. Hit a URE on an object component, and you lose the object, but not the entire volume; other objects can still be reconstructed (and PanFS could reconstruct them on-the-fly in addition to in the background).

ZFS solves the problem in two ways: 1) You can keep multiple copies of the data, and ZFS keeps checksums with the data. If a block fails the checksum, ZFS can quietly repair it from the copy. 2) ZFS can do slow patrol reads and find corrupted data that you haven't touched in awhile, and repair it, if you have a copy. No more ticking time bomb.

You also get cheap COW snapshots with ZFS, and the ability to send snapshot differentials over the wire for backups and disaster recovery (say your basement home office got flooded or your house burned down). This is something that APFS does not have.

 

[zapmymac]

That's September if you still want to use Microsoft Office, which only supports the current OS, plus the two before that.

Oh frack! Forgot about that as well. Thanks for the heads up. Hmmmm. I’ve been okay with Libre office…

 

[MilaM]

That's very interesting; however, it's also seven years old. It would be interesting to see it repeated today.

Time Machine + APFS on a spinning drive was a horrible experience, likely because APFS was not optimized for it. You had to wait 10 Minutes just to restore a specific file and backups also took easily 10 times as long as with a SSD.

It's ironic that some users will probably buy cheaper spinning storage these days for backups and archives due to SSDs becoming so expensive. My advice, don't do it.

 

[Powerbooky]

A sparse image is no good for my needs. You're right about the cost of a 4-5 bay NAS without disks. But one does have to incorporate the cost of the disks (none of my existing external drives are suitable). By the time you've done that, you're into the thousands.

Most enclosures contain regular HDD’s, either small or large 3.5inch. Open it and see if these are regular SATA. If positive then they fit in most NAS trays. I moved mine like this to cut costs.

It's ironic that some users will probably buy cheaper spinning storage these days for backups and archives due to SSDs becoming so expensive. My advice, don't do it.

Don’t! New HDD’s have always had more capacity for the dollar than SSD’s, even today. And a HDD still has a longer lifespan than SSD. If left unplugged, wil lose its data in 1 to 3 years. Quicker if the room temperature is regularly higher than standard room temperature.
Magnetic drives can last 10 years up to 20, in normal conditions. I have several disks left from my 2008 and 2001 Mac’s that still work perfectly even after years on the shelf.

Not Seagate drives btw, these are famous for premature failures since the 90ies.

 

[bzgnyc2]

This is an interesting claim: that HFS+ performs better on spinning disks than APFS. I would be interested in seeing data and, if one exists, an explanation as to why.

It was kind of amazing that opening say Applications on MacOS X was almost instant on mid-2000s laptop with a spinning laptop hard drive (10ms average seek times) despite the complicated .app structure.

The issue is that HFS+ included many optimizations for rotating disks while these are actually counterproductive to SSD. Spending microseconds of CPU time to minimize seeks make sense when seeks takes milliseconds but not so much when they too can complete in microseconds. In some cases HFS+ creates extra drives writes which aren't really an issue with HDD all else being equal but not ideal for SSD with low TBW.

Here are some of the strategies that I understand HFS uses (note this is AI summarized and I didn't validate by say going into HFS source code):

1. Next-fit allocation​

Instead of always searching the free space bitmap from the beginning, HFS+ remembered roughly where the last successful allocation occurred.

---

2. Clump allocation​

Instead of allocating exactly what an application requested, HFS+ often allocated extra contiguous space.

---

3. Delayed extent creation​

The allocator tried hard to extend the existing extent before giving up and creating another.

---

4. Extent overflow B-tree​

A file's catalog record stores only the first eight extents directly. If more were needed, additional extents went into the extent overflow B-tree.

---

5. Hot File Adaptive Clustering​

The filesystem monitored which files were accessed most frequently and periodically it would reorganize them so they occupied a special "hot zone" near the outer tracks of the disk, where sequential transfer rates were highest.

---

6. On-the-fly defragmentation of small files​

HFS+ could transparently rewrite fragmented files into a contiguous region when they were opened.


I remember seeing a experimental performance comparison of the APFS versus HFS+ that simulated long-term use (i.e. after fragmentation inherently slips in) but I can't find it.

In the meantime, here is a pretty good summary of the APFS versus HFS+ (mostly highlighting APFS's advanced features but noting the greater propensity for fragmentation on APFS and fragmentation is of course far more painful on HDD than SSD):

Why use APFS?

Deciding which file system to use for hard disks can be difficult. Here are the advantages and disadvantages explained in detail, for HFS+ and APFS.

eclecticlight.co

The elevator algorithm is ancient. I would be surprised if APFS did not simply use what already exists.

By far, the biggest optimization for spinning disks was invented thirty-four years ago: the log-structured file system. For whatever reason, it was not widely adopted.

https://en.wikipedia.org/wiki/Log-structured_file_system#Disadvantages

 

[extrachrispy]

It was kind of amazing that opening say Applications on MacOS X was almost instant on mid-2000s laptop with a spinning laptop hard drive (10ms average seek times) despite the complicated .app structure.

The issue is that HFS+ included many optimizations for rotating disks while these are actually counterproductive to SSD. Spending microseconds of CPU time to minimize seeks make sense when seeks takes milliseconds but not so much when they too can complete in microseconds. In some cases HFS+ creates extra drives writes which aren't really an issue with HDD all else being equal but not ideal for SSD with low TBW.

Here are some of the strategies that I understand HFS uses (note this is AI summarized and I didn't validate by say going into HFS source code):

[...lengthy list of HFS+ specific spinning drive optimizations...]​

Thank you for that. I'm actually quite surprised that a lot of that wasn't retained in APFS during a period of transition from spinning drives to flash (or maybe I shouldn't be, given that Apple exited the enterprise hardware space sometime back, and their focus is on integrated products--an OS that supports their current-generation hardware, which does not contain spinning drives).

 

[verdi1987]

That's very interesting; however, it's also seven years old. It would be interesting to see it repeated today.

Bombich's explanation as to why is certainly plausible: directories being scattered all over the platter rather than in a pre-allocated location would indeed lead to terrible problems with seeking and rotational latency. A nit: he describes file sizes in his article; they are, however, completely irrelevant to the speed of enumeration. All you are doing when enumerating (unless you are also doing a stat on each file) is reading directories.

I still experience this problem today. (I found the Bombich article as a result of my real-world experience.) I have backups of HFS+ drives on APFS. The drives are the same model and capacity. File enumeration is much slower on APFS. Granted, I have a lot of files.

I expected performance improvements with APFS on mechanical drives in the intervening years, but none ever materialized.

 

[Felix Lyran]

I still experience this problem today. (I found the Bombich article as a result of my real-world experience.) I have backups of HFS+ drives on APFS. The drives are the same model and capacity. File enumeration is much slower on APFS. Granted, I have a lot of files.

I expected performance improvements with APFS on mechanical drives in the intervening years, but none ever materialized.

I wonder where the basis for recommending APFS in Carbon Copy Cloner (the software Bombich develops) comes from then? I guess maybe it's the assumption that the issues that APFS presents when used on HDDs aren't a big deal for the purpose of backups? Maybe. I'd like to see it more clearly articulated though as I'm not sold on the idea that APFS is to be preferred on HDDs, even for that purpose. (For the record, I do use APFS on my HDD backup volumes, but that's only due to CCC recommending that I do that. I did use Time Machine previously and, if I'm remembering correctly, it too recommended using APFS.)