ZEVO Solution for ZFS on OS X Acquired by GreenBytes

[dolphin842]

Edit: incase anyone's wondering, it's currently used solely for FileVault 2's full disk encryption. That's about the extent of its capabilities for now. It's a full logical disk management system though, it's just in the early stages.

Core Storage, being a logical volume manager, doesn't solve the data integrity problems HFS+ has. The main appeal of ZFS to many is that it addresses silent data corruption.

I really don't want to run a linux or BSD box. The whole goal of using a Mac was to make getting software easy. I wanted to have iTunes running in the background to share music and movies in the home, along with a variety of DLNA software (my Samsung and Vizio TVs are incredibly picky about the host DLNA software).

The latest FreeNAS 8.2 release has support for plugins. Looks like DLNA and an iTunes server are already available. Everything (including volume management, sharing, etc) is configured through a web GUI. Honestly, it was easier for me to set up than OS X file sharing. Using an old mATX case/PSU and some leftover RAM/drives, I was able to get a Sandy Bridge-based FreeNAS box going for ~$100.

 

[KnightWRX]

Using an old mATX case/PSU and some leftover RAM/drives, I was able to get a Sandy Bridge-based FreeNAS box going for ~$100.

One of his requirements was low power. One of the mini-ITX based Intel Atom solutions might be better. Intel and others have quite a few boards that come with CPU already mounted, and being mini-ITX, you can get much smaller cases.

 

[SirOmega]

One of his requirements was low power. One of the mini-ITX based Intel Atom solutions might be better. Intel and others have quite a few boards that come with CPU already mounted, and being mini-ITX, you can get much smaller cases.

That was the whole reason I wanted to go Mac mini was that I would still get a good CPU and sip power (11W idle IIRC). If I wanted to throw a video on it for Handbrake to process overnight it would still be OK. Atom doesn't really offer that, nor a NAS-solution.

Is there any way to virtualize FreeNAS under OSX? Is that even a good idea?

 

[dolphin842]

That was the whole reason I wanted to go Mac mini was that I would still get a good CPU and sip power (11W idle IIRC). If I wanted to throw a video on it for Handbrake to process overnight it would still be OK. Atom doesn't really offer that, nor a NAS-solution.

Don't forget to add external drives to the power consumption. Just as a point of reference, my FreeNAS box (with two green drives, a 65W Sandy Bridge Celeron, and an old/inefficient PSU) idles at 43W.

Is there any way to virtualize FreeNAS under OSX? Is that even a good idea?

I believe it's possible... the online docs have more information.

 

[techsmith]

I think if you want ZFS you are going to have to get Solaris.

The problem with ZFS on Mac OS X is (1) How to present a full featured ZFS system to a typical Mac user in a way that he can understand the concepts and options. (2) What does a Mac user with one internal disk drive have to gain?

Since they are making their own propitiatory SSD add-in cards, they could in theory have them configured as smaller multiple drives. Configured in ZFS, one drive would be for redundancy, and the OS would see the others as just one drive. Then you can get the self-healing features of ZFS. Even with no redundancy, having the ability to detect corruption via checksums is a step ahead because the filesystem at least knows that corruption occurred. Coupled with time-machine, files detected as corrupted can be recovered from the time machine backup if a backup exists. As for how to present the options to the user, they could have a check box for redundancy. If checked, user gets "Enhanced data protection" along with the cost in GB. If unchecked, user gets "Maximum space availability". I'm not 100% sure but I think ZFS will allow these options to be changed, so that they can ship the computer with redundancy enabled and the user can disable it should they need the space and don't care about the protection.

I don't know about everyone else here but I for one have lost data due to crashes/lock-ups more than once, and I'm not just talking the data being worked on at the time. Any time the system goes down before the drives have been properly unmounted I get very nervous. I've never had these kinds of issues with Windows NTFS. HFS+ is crap and the sooner it can be replaced the better.

 

[dolphin842]

I don't know about everyone else here but I for one have lost data due to crashes/lock-ups more than once, and I'm not just talking the data being worked on at the time. Any time the system goes down before the drives have been properly unmounted I get very nervous. I've never had these kinds of issues with Windows NTFS. HFS+ is crap and the sooner it can be replaced the better.

Both filesystems have journaling, but both still have a risk of data loss if the power is cut mid-write. At least Microsoft has announced a solution moving forward... the frustration on the OS X side is the radio silence for years on end...

 

[TLewis]

The latest FreeNAS 8.2 release has support for plugins. Looks like DLNA and an iTunes server are already available. Everything (including volume management, sharing, etc) is configured through a web GUI. Honestly, it was easier for me to set up than OS X file sharing. Using an old mATX case/PSU and some leftover RAM/drives, I was able to get a Sandy Bridge-based FreeNAS box going for ~$100.

If he doesn't need plugins, I'd recommend that he take a look at "nas 4 free" (there are actually no spaces in the name, but the forum censors the name if I take out the spaces -- I also had to use a URL shortener, sorry), which appears to be a fork of freenas V7. In particular, nas 4 free uses ZFS version 28 (the last free version), while freenas seems to be stuck at version 15. For details on versions, see ZFS release history in wikipedia, under "ZFS pool version number".

 

[firedownunder]

Well this is incredibly disappointing.

I had planned on buying one of the new Mac minis that I expect to launch with 10.8 this week. That combined with a USB 3.0 UASP external enclosure for multiple disks and ZEVO would finally allow me to come up with a lower-power way to replace my aging Windows Home Server now that MS has killed that platform.

Still looking for solutions to build that in-home server that are easy, low power and expandable.

http://code.google.com/p/maczfs/

Suggested in an older thread, up and running about a year with no hiccups. Took a little reading and getting comfortable with terminal, but fairly straight forward. Loaded up my Mac Pro with drives and haven't looked back (got tired of waiting on Ten's Compliment).

 

[quasinormal]

Pardon my ignorance, but what does this mean for licence owners? I have my files backed up to a ZFS formatted volume. I just opened the pane in System preferences and downloaded update 1.0.3.

Would it be OK to keep on using ZFS?

 

[smuckola]

ZFS on Mac OS is at http://maczfs.org/

Hi guys. If you're wanting ZFS on Mac OS right now, especially if you want free software, come to http://maczfs.org/

We welcome Greenbytes to the MacZFS community and we hope that they'll collaborate well beyond just their legal obligation to release the source code under the CDDL.

 

[haravikk]

ZFS is overkill for a company (Apple) that is decidedly consumer and mobile.

Data integrity is a pretty important feature no matter how you use your machine, and with how much data.

I'm hoping that, at the very least, Apple has integrity checking in the works as a feature for CoreStorage.

After that, block-level snapshots would hugely benefit Time Machine, allowing for faster, more efficient backups. This wouldn't need any kind of user setup; just allow Time Machine to configure it automatically.

Finally, assigning an SSD as a read cache in a machine with both an SSD and one or more HDDs would be useful too, and benefit machines with those options; again, having a default option for machines so configured will cover that nicely, plus an installer option for quickly setting it back up if you re-install.


So, while ZFS may be most often used for setting up RAID style distribution and/or redundancy, integrity, snapshots and read/write caching are things that could definitely benefit users, the first two especially. That's not to mention proper (rather than bolted on) support for hard-linking and various other useful features that HFS+ can do, but only after a fashion.

Also, integrating the ability to create RAID through CoreStorage would be nice; definitely a power user feature but bringing everything under one heading would be much better overall.

 

[TLewis]

Hi guys. If you're wanting ZFS on Mac OS right now, especially if you want free software, come to http://maczfs.org/

Looks good, but is it really based upon the ancient ZFS version 9?

----------

Data integrity is a pretty important feature no matter how you use your machine, and with how much data.

It's interesting to note that one of the early ZFS developers was doing development on flaky hardware: occasionally, the data that ended up being written to disk was corrupted. However, ZFS automatically detected the corruption (when the data was read), and auto-corrected it. He supposedly ran for a while with this bad hardware.

 

[fmaxwell]

He's dead, Jim...

Once Apple dropped support for ZFS, it was dead on OS X. The very people who really want ZFS, those for whom their data is vitally important, are not going to accept a third-party implementation that could be broken at any time by any OS X upgrade, even necessary ones that address security vulnerabilities. They aren't going to accept the idea that the third-party supplier might fold, drop support for ZFS on OS X, or turn it into an internal product.

I'm all for ZFS, but it needs to be OS-independent and implemented in hardware -- the way that Drobo did, where the external box looked like a standard hard drive while actually being a RAID.

 

[manu chao]

I heard a lot about ZFS over the years. What I never hear is what ZFS does that will benefit me, as a Mac user. Until I hear that, I won't care.

Simple, it reduces the rate of file corruption noticeably. File corruption can happen with data just sitting on your HDD and with every copy operation (eg, the automatic defragging taking place on OS X in the background for small files, when upgrading your HDD, restore something from a backup, copy something from one device to another, upgrading to a new computer). And if the file corruptions happens in your disk directory, you can easily loose a lot of files in one fell swoop.

 

[Lindono]

Core Storage, being a logical volume manager, doesn't solve the data integrity problems HFS+ has. The main appeal of ZFS to many is that it addresses silent data corruption.

If I'm right about Apple's plans for Core Storage, they may be working on implementing their own file system, relying heavily on Core Storage. For example, HFS compression could in theory be implemented as full-disk compressing using Core Storage in the same way they implement encryption. They could even implement an entire file system backed by this new technology. It really is too early for you to claim any strengths or weaknesses in regard to Core Storage, seeing as how new it is. Once there is full support for all of its features in the OS, we should have some feel for where it's headed. As it stands, it isn't even clear what it does support, besides encryption.

 

[AidenShaw]

If I'm right about Apple's plans for Core Storage, they may be working on implementing their own file system, relying heavily on Core Storage. For example, HFS compression could in theory be implemented as full-disk compressing using Core Storage in the same way they implement encryption. They could even implement an entire file system backed by this new technology. It really is too early for you to claim any strengths or weaknesses in regard to Core Storage, seeing as how new it is. Once there is full support for all of its features in the OS, we should have some feel for where it's headed. As it stands, it isn't even clear what it does support, besides encryption.

But, isn't it true that most Apples hide the filesystem from the user?

Why do *anything* to improve something that the user never sees?

 

[Lindono]

But, isn't it true that most Apples hide the filesystem from the user?

Why do *anything* to improve something that the user never sees?

The user never sees 99% of what actually goes on in any modern OS... That's kinda the point of a GUI. Abstraction. They'll improve it because it will give them added flexibility to implement future features, as well as improve existing ones. Block-based Time Machine backups anyone? Simple to do with Core Storage.

 

[ChrisA]

...
I'm all for ZFS, but it needs to be OS-independent and implemented in hardware -- the way that Drobo did, where the external box looked like a standard hard drive while actually being a RAID.

Can't do that and still have it be ZFS. The point of ZFS is that off of RAID, Volume Management and the file system are ALL in the same layer. In other systems these functions are independent layers

What a Mac user would see in ZFS is that the concept of a "disk" goes away. there is on;y "storage" and if you need more of it you buy a physical disk and plug it in and if a disk fails you buy and other and plug it in then trash the failed part. Pretty much like what Drobo does, except that drobo looks like a disk to the OS, that part goes away.

You can't build it into the disk hardware because there needs to be a part of ZFS that sees all the drives and it is one layer thick so it can live in only one place. So it has to be above the hardware layer

Yes your idea would work but it would not be ZFS.

 

[steve-p]

Finally, assigning an SSD as a read cache in a machine with both an SSD and one or more HDDs would be useful too, and benefit machines with those options; again, having a default option for machines so configured will cover that nicely, plus an installer option for quickly setting it back up if you re-install.

I can't see anyone putting a great amount of effort into something specifically for SSD/HD combinations. Those combinations only exist at the moment because we are in a transition period between HD and SSD, purely because of the cost and capacity of SSD which is a relatively new technology. Within a fairly short timeframe it's likely that SSD storage cost will plummet and no-one will be buying conventional hard disks anymore anyway. 512GB SSDs today cost around 1/3 what they did 15 months ago.

 

[haravikk]

I can't see anyone putting a great amount of effort into something specifically for SSD/HD combinations. Those combinations only exist at the moment because we are in a transition period between HD and SSD, purely because of the cost and capacity of SSD which is a relatively new technology. Within a fairly short timeframe it's likely that SSD storage cost will plummet and no-one will be buying conventional hard disks anymore anyway. 512GB SSDs today cost around 1/3 what they did 15 months ago.

I'm not so sure; SSD's are fast approaching the point where it's starting to get more difficult to pack more data in without crippling efficiency, all SSD's today use a variety of tricks to keep the drives fast today.

So while the gap has certainly closed a bit, HDDs are still ahead in raw capacity, and don't have the same longevity concerns which, coincidentally, is another case to be made for integrity checking at the file-system/volume manager level 😀

Still, it's definitely a much more niché case, but read-caching is a fairly straightforward technology to implement, but it needs to be done at a fairly low level in the OS if you want to speed up boot-times, which is one of the big advantages of SSD's for a lot of users. Besides, while the gap in capacity and price have been going down, it's still going to be some time before it closes enough to justify going all SSD.

----------

Core Storage, being a logical volume manager, doesn't solve the data integrity problems HFS+ has. The main appeal of ZFS to many is that it addresses silent data corruption.

It could in theory though couldn't it? It just needs to maintain a store of checksums for each block, where isn't that important, and verify blocks as they're fetched, and update the checksum as they're stored. It's much the same idea of how the encryption/decryption works as blocks are fetched/stored.

The more complex part is marrying it to data-redundancy, so corrupt files can be restored from a copy, or adding layers of error correction, or some combination of the two. I'm a big fan of ZFS' ability to do redundancy on a single drive, useful for a laptop when you want to benefit from the integrity checking; it seems to handle it better than mirroring two drive partitions would as well.

 

[KnightWRX]

The user never sees 99% of what actually goes on in any modern OS... That's kinda the point of a GUI. Abstraction. They'll improve it because it will give them added flexibility to implement future features, as well as improve existing ones. Block-based Time Machine backups anyone? Simple to do with Core Storage.

Don't get caught up in Aiden's comment, it was tongue in cheek. He didn't mean that Apple hides HFS+ from the user by saying "the filesystem", he was pointing at the fact that Apple tries its hardest to make file management not be based on the old paradigm of files and folders, but rather on librairies (iTunes, iPhoto, etc..). This is based on a lot of people that are non technical and talk about the "filesystem" meaning the folders and files they see in Finder. That's not the filesystems, that's your files. Finder is a file manager application, not a filesystem manager application, but to some people the distinction is quite lost.

They don't know that Finder is simply a userspace application that interacts with a standard API that calls into syscalls in the Kernel to get handles to files, which are managed by a VFS layer (open(), write(), read(), close() are the same syscalls in the kernel for every underlying storage type, no matter its actual filesystem) which then maps the call to the actual filesystem driver depending on which handle received the call.

This is quite all besides the point of the kernel structures being used to actually address userspace queries and map them onto the hardware. And the GUI isn't what abstracts this, it's the userspace/kernel space seperation that does. 😉

The CLUI hides as much detail of the modern OS as a the GUI does. You don't directly manipulation kernel structures, set hardware registers or interfere with the process scheduler on the CLUI you know. You interact with the same abstracted view of it all that the GUI presents, just as a text based format rather than with shiny icons.

 

[Lindono]

Don't get caught up in Aiden's comment, it was tongue in cheek. He didn't mean that Apple hides HFS+ from the user by saying "the filesystem", he was pointing at the fact that Apple tries its hardest to make file management not be based on the old paradigm of files and folders, but rather on librairies (iTunes, iPhoto, etc..). This is based on a lot of people that are non technical and talk about the "filesystem" meaning the folders and files they see in Finder. That's not the filesystems, that's your files. Finder is a file manager application, not a filesystem manager application, but to some people the distinction is quite lost.

They don't know that Finder is simply a userspace application that interacts with a standard API that calls into syscalls in the Kernel to get handles to files, which are managed by a VFS layer (open(), write(), read(), close() are the same syscalls in the kernel for every underlying storage type, no matter its actual filesystem) which then maps the call to the actual filesystem driver depending on which handle received the call.

This is quite all besides the point of the kernel structures being used to actually address userspace queries and map them onto the hardware. And the GUI isn't what abstracts this, it's the userspace/kernel space seperation that does. 😉

The CLUI hides as much detail of the modern OS as a the GUI does. You don't directly manipulation kernel structures, set hardware registers or interfere with the process scheduler on the CLUI you know. You interact with the same abstracted view of it all that the GUI presents, just as a text based format rather than with shiny icons.

I suppose so. I sometimes lose sight of where the actual abstraction lies. Everything looks like a bunch of CGPoints and NSViews. Heh.

 

[till213]

...
Fights corruption by using checksums
...

THIS!

Most of these features are pretty much Enterprise level. ...

True. However one can't stress enough that in the age of precious Gigabyte (Terrabyte!) Disks it's becoming more and more crucial to detect read-errors as soon as a single bit flips!

Why is that so important? Why not wait until that harddisk sector becomes *physically* unreadable? Because then it's already way too late!

Why?

Because typically a hard disk dies "a slow death". First some sectors become corrupt, means: "Bits flip over randomly" - the current HFS+ file system WON'T DETECT THESE BIT-FLIPS as long as that sector remains physically readable! And if you have a backup solution (and I hope you have!) then all those flipped (wrong!) bits will get backup-ed - possibly for weeks.

At some point you'll realise that your hard-disk starts making funny noise and eventually it will refuse to read certain sectors (physically unreadable), so it is only now that you'll realise - with the current HFS+ - that your harddisk just died.

But you're also very likely to notice that possibly much more data has become corrupted ("you cannot open certain JPEG data or other documents - but in the worst case you can, but the image data has changed, so you won't possibly notice until much much later that some pixels (or text...) have changed"). And that corrupted data has been backuped, maybe for weeks!

So good luck (manually!) finding a state in your backup where all data is in a good state!


So how does ZFS help here? Exactly in this situation when a single (or multiple) bits get randomly flipped - because of checksums which are constantly evaluated and updated upon read/write. And it makes sure that the file system is always in a consistent state! Partial file writes simply won't get committed if you loose power in that very moment etc. Moving a file is much more secure.


So in the age of thousands of precious photos on your harddisk (1) you absolutely want a filesystem as ZFS which detects logical errors - also as "consumer"! YES, YOU WANT IT! Repeat after me! YOU WANT ZFS ON YOUR CONSUMER MAC! Go and write Apple about it, use the feedback forms found on Apple's homepage!


(1) if you now mention "iCloud" or the like you still haven't understood the problem of "backing up corrupted data" - read again what I said about "flipped bits" which are still physically readable, so the current HFS+ doesn't notice that something went wrong (because it doesn't know nada, niet, nothing about the underlying file structure such as JPEG data), so it will happily be backed up, be it some TimeMachine backup harddisk, or synced back into the "Cloud".

 

[KnightWRX]

I suppose so. I sometimes lose sight of where the actual abstraction lies. Everything looks like a bunch of CGPoints and NSViews. Heh.

As opposed to a bunch of printf()s and fgets()s ? 😉

I'm just used to working with CLUI based programs and environnements. GUIs for me are a way of managing multiple screen sessions simultaneously.

 

[haravikk]

And if you have a backup solution (and I hope you have!) then all those flipped (wrong!) bits will get backup-ed - possibly for weeks.

I'm not sure this is true; Time Machine for example only copies files that have been updated since the last successful backup (if any). It doesn't recheck files that it doesn't think have changed, so any corruption won't be backed up.

The problem is that your backup drive(s) could be suffering similar corruption, which means that even if your original copy is fine, the backed up copy could have become corrupted. In theory it's a lot less likely as your backup drive should be seeing much less disk activity over time, so it should last a lot longer.

Anyway, my point being that a file that's become corrupted will only be backed up if you were to successfully open it (in spite of the corruption) then re-save it with the corruption still in place. Even then Time Machine's historic data may save you.

It's no alternative to proper integrity checking and additional error correction, but your backups should be comparatively okay.