ZFS Comes to OS X Courtesy of Apple's Former Chief ZFS Architect

[ratsg]

Why not EXT4?
Or Btrfs?

The technical advantages of ZFS over the (2) file systems you mentioned aside, because they would have been stuck with worse licensing issues (i.e. GNU GPL) than dealing with CDDL.

----------

Why couldn't Apple with its near $100 billion in the bank just outright buy ZFS if it's that good? I mean HFS+ can't be all that bad....

Not that I am suggesting that $100 billion USD is any small sum, but ZFS is owned by Oracle, and that is where primary development occurs.

I just don't see Oracle being bought out any time soon, either by Apple or anyone else.

 

[KnightWRX]

Not that I am suggesting that $100 billion USD is any small sum, but ZFS is owned by Oracle, and that is where primary development occurs.

I just don't see Oracle being bought out any time soon, either by Apple or anyone else.

You do know that everyone was speculating about Apple acquiring Sun instead of Oracle right ?

 

[ratsg]

You do know that everyone was speculating about Apple acquiring Sun instead of Oracle right ?

I do, but that ship sailed long ago.

 

[KnightWRX]

I do, but that ship sailed long ago.

Unfortunately. However, with Apple's current consumer-only direction, I don't see what they could have gotten out of Sun. Heck, I'm not even sure ZFS is really needed in OS X in a consumer only world.

 

[ratsg]

Unfortunately. However, with Apple's current consumer-only direction, I don't see what they could have gotten out of Sun. Heck, I'm not even sure ZFS is really needed in OS X in a consumer only world.

Apple's current consumer oriented direction raises many flags and concerns for me as well.

That aside, ZFS brings many useful features to desktop and laptop computing. And it works well enough on its own/built in defaults, that for the non-technical end users, it (ZFS) just stays out of the way. At least as much as HFS+ does.

 

[KnightWRX]

That aside, ZFS brings many useful features to desktop and laptop computing. And it works well enough on its own/built in defaults, that for the non-technical end users, it (ZFS) just stays out of the way. At least as much as HFS+ does.

Sure, but is it needed. Apple obviously has a thing about investing in things they don't need, just because they would be a nice to have for power users.

My own stuff at home is all managed under LVM on my Linux servers (yes, with an S, yes for my home network). But do I really need volume management if I were some consumer reading facebook and storing stuff in iCloud ? Probably not.

 

[ratsg]

Sure, but is it needed.

That is something only you can answer for yourself. As a person who has been working with ZFS since it was in beta, its difficult to imagine life without it.

Apple obviously has a thing about investing in things they don't need, just because they would be a nice to have for power users.

When I made my previous comment, I specifically commented on its (ZFS) usefullness for in desktop and laptop areas, and specifically left server implementation out. For some odd reason I have yet to understand, some are reading this as only useful in an enterprise environment.

In a desktop area, its great to be able to dynamically enable compression on file system(s) that compression would be beneficitial to, and that I am not going to add external drives.

Or if you are a person who does all of their work from a laptop, with one drive slot, and no possibility of even a raid 1 mirror, I can have multiple copies of data

(i.e. zfs set copies=2 rpool/<fs-name> )

This one feature is the next/best thing to having a drive mirror in a laptop without room for (2) drives.

If you can successfully perform all of your computing needs on an iPad, nothing stated here by any of us will make any sense anyway.

My own stuff at home is all managed under LVM on my Linux servers (yes, with an S, yes for my home network).

And I have multiple Solaris, different Solaris based distro, and Apple servers in my house.

But do I really need volume management if I were some consumer reading facebook and storing stuff in iCloud ? Probably not.

I realize that "the cloud" is all the fad right now, but anyone who tells me that all their data is stored on "the cloud" at some 3rd party location that they have little to no leverage over is esentially telling me that their data means little to nothing to them.

 

[ratsg]

One additional comment I would make, is that it great we (Mac OS X users) now have this, its sad it took so long for us to get a commercial release. Yes, I am aware of the open source effort.

Obviously, Solaris, and Solaris based distro's have had ZFS forever.

The BSD distros, and primarily FreeBSD have done an incredible job of integrating ZFS into *BSD.

The lunux distros can use ZFS either as a FUSE file system or via a kernel module (i.e. http://zfsonlinux.org/ )

From a desktop/end-user perspect, only dos/windows/os/2 groups are left out.

Again, its great that ZFS is finally with us in a commercial form, its just sad we were esentially last.

 

[50548]

We need a decent file system. I've experienced corruption with hfs+ but nothing as bad as with fat32 or ext2.

And HOW is this ZFS solution integrated into OS X?

 

[hchung]

If I remember correctly, ZFS doesn't play well with SSD's. It plays, but not optimized.

Interesting thing I read was actually quite the opposite. Can't seem to find it now, but from watching behaviors, seems quite likely.

Because ZFS does a heavy amount of caching in memory when writing, it writes in relatively big sequential swaths. For example, if you're copying a large number of small files to a ZFS volume, a bunch of these will go into RAM first and then get written in chunks around 512K at a time.

Why does this matter? Well, remember the old crappy JMicron 601/602 SSDs and the stutter issues? Writing in large sequential blocks (relative to the NAND block size) cuts down on the stutter dramatically and increases the performance.

I read this after getting a 16GB one for the boot drive... maybe should have moved it to being the ZIL. But oh well, too lazy.

 

[sennekuyl]

Unfortunately. However, with Apple's current consumer-only direction, I don't see what they could have gotten out of Sun. Heck, I'm not even sure ZFS is really needed in OS X in a consumer only world.

ZFS specifically no. But I would love a filesystem that has end to end checksum like ZFS as a consumer filesystem. AFAIK, NTFS et al except ZFS & BTRFS checksumming is an after the fact occurrence if it occurs.

Pools as you say are not needed and may confuse the issue --- till they can shrink.

The desirable features in an ideal consumer filesystem imho would be check-sum, fail-safe distributed filesystem & zfs-esque snapshots. Maybe a hot-pluggable mirror would be advantageous but as you say that is getting into volume management.

----------

One additional comment I would make, is that it great we (Mac OS X users) now have this, its sad it took so long for us to get a commercial release. Yes, I am aware of the open source effort.

Obviously, Solaris, and Solaris based distro's have had ZFS forever.

The BSD distros, and primarily FreeBSD have done an incredible job of integrating ZFS into *BSD.

The lunux distros can use ZFS either as a FUSE file system or via a kernel module (i.e. http://zfsonlinux.org/ )

From a desktop/end-user perspect, only dos/windows/os/2 groups are left out.


Again, its great that ZFS is finally with us in a commercial form, its just sad we were esentially last.

:jawdrop: linux now has kernel module for ZFS? That does away with the fuse issues, no?

Hadn't realised it had advanced so!

Essentially last? Windows can't even read ZFS. It has to be shared as a CIFS link. What are you talking about?
--- Edit: Whoops apparently there was project to read ZFS . No activity though.

 

[ratsg]

:jawdrop: linux now has kernel module for ZFS? That does away with the fuse issues, no?

Hadn't realised it had advanced so!

The linux people I work with have been more impressed with this, than what they have seen from BTRFS. YMMV. I don't have personal opionion of this yet. Sun/Oracle Solaris and Cisco networking skills keep a roof over my head, not linux.

Essentially last? Windows can't even read ZFS. It has to be shared as a CIFS link. What are you talking about?
--- Edit: Whoops apparently there was project to read ZFS . No activity though.

I appoligise for my vague statement. What I meant to say, is that I don't expect to ever see any capability in dos/windows or OS/2 to read/write to ZFS pools. Of (what I consider) all the other commonly used desktop environments, we (Mac OS X users) are last to get "up to speed" on ZFS. Everyone else is esentially there.

As a side note, for any OS/2 fans who aren't up to speed, the latest release of OS/2, now EcomStation, now has the ability to boot from JFS2 volumes/file systems. Good times everywhere.

 

[sennekuyl]

The linux people I work with have been more impressed with this, than what they have seen from BTRFS. YMMV. I don't have personal opionion of this yet. Sun/Oracle Solaris and Cisco networking skills keep a roof over my head, not linux.

I appoligise for my vague statement. What I meant to say, is that I don't expect to ever see any capability in dos/windows or OS/2 to read/write to ZFS pools. Of (what I consider) all the other commonly used desktop environments, we (Mac OS X users) are last to get "up to speed" on ZFS. Everyone else is esentially there.

As a side note, for any OS/2 fans who aren't up to speed, the latest release of OS/2, now EcomStation, now has the ability to boot from JFS2 volumes/file systems. Good times everywhere.

Ah. I guess from that perspective OSX is the last. No need to apologise.

Interesting about OS/2; in '95 I was intending to get the latest OS/2 (warp? 3 i think actually) but the cost and 'convenience' diverted me to Win95.

 

[Prodo123]

I contacted Ten's Complement, and they responded that ZEVO will not allow you to install Mac OS X onto a ZFS partition, even from a backup restore or a hard drive cloning.

 

[The Man]

Problem with ZEVO is, how do I know how long they (as a company) will be around in a few years time? Yes, my data will be okay, but if there's no Mac software to connect the drive, what good is it to have all my data on ZFS drives?

 

[KnightWRX]

I realize that "the cloud" is all the fad right now, but anyone who tells me that all their data is stored on "the cloud" at some 3rd party location that they have little to no leverage over is esentially telling me that their data means little to nothing to them.

I agree, but you and I are not looking at this from a consumer perspective. My data is stored here locally on a mirrored RAID array. But in the consumer world, Dropbox and iCloud and Amazon et al are it right now.

From Apple's consumer oriented perspective, does it make sense to invest and work on local storage solutions ? That's the key to OS X getting better storage management and I just don't see it anymore.

 

[haravikk]

Sure, but is it needed.

I would love to have block-level snapshots as a replacement to Time Machine (I believe ZFS can perform its snapshots onto a target volume?), as Time Machine's timestamp based full-file copying is doing my head in; change the label on a 4gb HD movie and you have 4gb of copying to look forward to!

Data deduplication (hate that term) would be nice, as I know that my own system has a lot of (necessary) redundancy in places, but I wouldn't mind at all if the system used a single chunk of data for all copies to conserve space. Not a huge issue for full-sized hard-drives, but less copying is never a bad thing. Compression as well could be nice.

I think that being able to have a single pool of all my current drives would be great, and if Mac OS X could do this in a sensible way automatically then I could see it greatly benefiting your average consumer. Think of people with an SSD, a regular hard drive, and a Time Machine back-up drive. With clever ZFS (or other pooled storage) those three drives could be a single volume, with the SSD invisibly caching commonly accessed (but rarely written) files for speed, while using the HD for capacity and the Time Machine disk for redundancy. Who wouldn't want that? Currently I have my user folder (and some other choice folders) haphazardly symbolically linked from my SSD on my standard hard-drive(s), which is a huge pain, just to get a decent, but far from perfect, mix of speed and capacity.

Even for laptop users with a single internal drive and an ad-hoc external drive for backups I believe ZFS can still reduce that to a single volume, using the external drive for redundancy when available?

By comparison HFS+ is fine, but it lacks so many modern features by comparison, and when it comes down to it ZFS has been around now for years so it's not like those features are even that modern anymore.


My main regret is that it seems that Apple was wasting its time with ZFS due to the licensing issues (I imagine they don't apply for single users, but because Apple is a company it has trouble?). The result is that we'll have to wait for Mac OS 10.8 at the earliest before we see anything of Apple's own solution, and given the necessary amount of development and heavy testing I'm not even sure we'll see it in 10.8.

 

[KnightWRX]

I would love to have block-level snapshots as a replacement to Time Machine (I believe ZFS can perform its snapshots onto a target volume?), as Time Machine's timestamp based full-file copying is doing my head in; change the label on a 4gb HD movie and you have 4gb of copying to look forward to!

At this point, I will point this out : this is ZFS' volume management you're talking about, not its filesystem portion. LVM (under HP-UX and Linux) implements block level snapshots and it works with any overlying filesystem (xfs, ext2, ext3, ext4, vxfs, hfs (HP's version of UFS, not the Mac thing)).

Again, I see all the points for volume management, what I'm asking, is what ZFS features are consumery. You would love, I would love, power users and professionals of all categories the world over would love, but does Apple's consumer base really need it ?

Honestly ask yourself that question, which eyes are you looking at this with ? My stepfather doesn't know about block-level snapshotting (which enables data deduplication and tons of other fun features, yes I managed NetApp appliances for a while), compression, or volume management. In his mind, adding a new disk to a system is transparent because he takes it to a repair person to do, not because he can go "vgextend, lvextend/pvmove" (sorry, for the past year or so I've been stuck in LVM land with HP-UX and Linux as the primary platforms at my work, we left VxVM behind and never quite moved to ZFS).

Apple doesn't need to implement ZFS to get volume management. It needs to implement a volume manager. ZFS is nice in that it also updates your filesystem while you're there, killing 2 birds with 1 stone. But again, Apple hasn't done volume management. Do they really need to, in a purely consumer optic ?

 

[AidenShaw]

But again, Apple hasn't done volume management. Do they really need to, in a purely consumer optic ?

Any consumer running Windows is using the volume manager snapshot feature frequently.

VSS (Volume Shadow Copy Service) is used by System Restore, Previous Versions, and all of the Windows backup tools. VSS lets backup get consistent backups without any issues with files being written during the backups - including being able to get a fully consistent "bare metal backup" of a running system.

 

[haravikk]

but does Apple's consumer base really need it ?

I don't really care if it's ZFS or not, as I already noted, CoreStorage seems to point toward the start of a volume manager.

But I think it matters just fine to your average user, whether they know they want it or not. If it can be used to make Time Machine backups smaller, and faster, and also use them for data-integrity, then it'd be an ideal feature for everyone. As while your average user might not know what data integrity issues are, if you ask them if they'd prefer their data to invisibly degrade, or for Time Machine to automatically repair it, then I can hazard a guess at what most people would want! In particular now that we have Thunderbolt becoming more common, back-up drives can be connected at speeds effectively as good as having them fitted as internal drives.

Pooled storage is also very useful for machines with SSD options, as if you can advertise the SSD as a slap-on upgrade that speeds up the machine out-of-the-box with no effort whatsoever then that's a big improvement compared with the current situation of requiring a bit of work to manually balance speed, capacity, and wear on the SSD. SSD's are ideal as read-cache volumes, but there's not really a good way to use them as such right now as you still need a separate, bootable disk.

So I think there are plenty of good reasons your average user could do with a good volume manager, ZFS or no, and that's in addition to the many uses that other users could have for it. It's also worth remembering that Mac OS X is not a fully consumer-oriented OS; it's a very powerful OS with a pretty GUI on top that makes that power easily accessible, and it still has a server version/add-on so it's not like there is no market for OS X being able to manage disks properly.

 

[Sevenfeet]

All this seems interesting but I'm going to have to wait for the gold or platinum versions since I'm running Lion Server on my household machine these days. Silver is not considered to be compatible with OS X Server for some reason. Also, it would be nice to have AFP file sharing working and that's still on Zevo's "to-do" list.

Like with any "new" filesystem, you put things on it you can afford to lose. For my data, I put critical files (like my iTunes library of Lossless recordings and ripped DVDs) in two extra places. One is a Drobo which I know probably can't be used with Zevo (the logic of the Drobo would likely implode if you formatted it for anything else outside of HFS+, FAT or NTFS). But I could see putting it on another external volume as a test case for some months.

I do have some light ZFS experience. My ancient Sun Ultra 10 (running Solaris 10) in my basement is running with two ZFS volumes (including the boot volume). I look forward to the next release.

 

[blitzspear]

Well I bought silver

Bought silver, tried to format WD 3TB drives.

Works first time, then after a disconnect the drives never reconnect.

Contact their tech support and requested a refund...

Week later nothing.....

I managed to recover the 3TB disks by formatting them in windows as Disk Utility refused to touch them once ZFS had formatted them...

Wasted money... wish i hadn't bought it.

 

[grahamperrin]

booting from ZEVO volumes

… hoped you can convert your boot drive to ZFS, but apparently not. …

… I hope they continue and get the stuff working so that Mac OSX can be booted from ZFS …

Do they have a 'timeframe' when booting from zfs will be supported?

Support for booting from ZEVO volumes is on the product roadmap without a schedule.

I guess that there will be:

1. initial releases of all four editions

2. a schedule for key features that are currently not in any of the four.

 

[grahamperrin]

ZFS, encryption, integrity, Apple Core Storage

… while MacZFS says it doesn't support ZFS encryption, would it support CoreStorage encryption? …

Published by Apple for the first time in January 2012:

Cryptographic Services Guide: About Cryptographic Services

> … describes the encryption, decryption, signing, hashing, and other cryptographic technologies in Mac OS X and iOS.

However: in the PDF of that document, no mention of CoreStorage.

The man page for fsck_cs acknowledges bugs:

> fsck_cs does not perform an exhaustive validation,
> nor is it able to fix many of the inconsistencies that it does detect.

In the absence of documentation about Core Storage approaches to integrity, I assume that any mix of Core Storage with ZEVO or ZFS will introduce a risk that ZFS is designed to minimise or negate: corruption.

 

[msh]

Bought silver, tried to format WD 3TB drives.

Works first time, then after a disconnect the drives never reconnect.

Contact their tech support and requested a refund...

Week later nothing.....

I managed to recover the 3TB disks by formatting them in windows as Disk Utility refused to touch them once ZFS had formatted them...

Wasted money... wish i hadn't bought it.

I bought it and formatted about 5 drives connected via eSata, FW800 and internally and it is all working very well. I have found that it does disconnect and reconnect (export and import in ZFS parlance) flawlessly. I did initially have a performance (speed) issue but that was my mistake because I formatted my disks as "advanced format" when they weren't. When I reformatted without that option checked all was fine.