PDA

View Full Version : ZEVO Solution for ZFS on OS X Acquired by GreenBytes




MacRumors
Jul 23, 2012, 10:32 AM
http://images.macrumors.com/im/macrumorsthreadlogo.gif (http://www.macrumors.com/2012/07/23/zevo-solution-for-zfs-on-os-x-acquired-by-greenbytes/)


http://images.macrumors.com/article-new/2012/07/greenbytes_logo-150x118.jpg

Back in January, we reported on ZEVO (http://www.macrumors.com/2012/01/31/zfs-comes-to-os-x-courtesy-of-apples-former-chief-zfs-architect/), an effort by former Apple file system engineer Don Brady to finally bring the Sun-backed ZFS file system to OS X. Brady's company, Ten's Complement, had begun releasing a set of software packages to support the robust file system on OS X, but planned launches for the some of the more advanced packages had begun to slip in recent months.

As noted by TUAW (http://www.tuaw.com/2012/07/22/zevo-zfs-on-os-x-project-now-part-of-greenbytes/), Brady has now announced that ZEVO has been acquired by GreenBytes (http://www.getgreenbytes.com/) a storage appliance hardware company that already uses ZFS in its products. Brady will also be joining GreenBytes as a development engineer.As to the future of the project, GreenBytes' spokesman Michael Robinson responded to TUAW's inquiry: "ZEVO's ZFS on OS X is safe. Ten's Complement has joined GreenBytes to continue their work and now they have more development support." Robinson says that GreenBytes will have more to share regarding ZEVO's future "down the line."A blog post (http://www.getgreenbytes.com/blog/bid/80758/GreenBytes-Welcomes-ZEVO-and-Don-Brady) from GreenBytes expresses a similar sentiment, although it remains unclear exactly what the company's plans for ZFS on OS X will be, with some ZFS fans having expressed concern that GreenBytes will be refocusing the product for its own internal needs.GreenBytes has been an enthusiastic supporter of ZFS (including ZFS on OS X) for many years and is absolutely committed to the continued development and support of Don's work on ZEVO (ZFS on OS X) into the foreseeable future.With the transfer of ZEVO to GreenBytes, Ten's Complement has ceased sales of the Silver Edition software package that had been available, and it remains unclear how and when ZEVO will be redeployed for OS X by GreenBytes.

Article Link: ZEVO Solution for ZFS on OS X Acquired by GreenBytes (http://www.macrumors.com/2012/07/23/zevo-solution-for-zfs-on-os-x-acquired-by-greenbytes/)



nuckinfutz
Jul 23, 2012, 10:35 AM
It's ok.

I'm hoping we see the vestiges of a HFS+ replacement in 10.9

ChrisA
Jul 23, 2012, 10:42 AM
I think if you want ZFS you are going to have to get Solaris.

The problem with ZFS on Mac OS X is (1) How to present a full featured ZFS system to a typical Mac user in a way that he can understand the concepts and options. (2) What does a Mac user with one internal disk drive have to gain?

nuckinfutz
Jul 23, 2012, 10:44 AM
I think if you want ZFS you are going to have to get Solaris.

The problem with ZFS on Mac OS X is (1) How to present a full featured ZFS system to a typical Mac user in a way that he can understand the concepts and options. (2) What does a Mac user with one internal disk drive have to gain?

Bingo.

ZFS is overkill for a company (Apple) that is decidedly consumer and mobile.

slu
Jul 23, 2012, 10:57 AM
I heard a lot about ZFS over the years. What I never hear is what ZFS does that will benefit me, as a Mac user. Until I hear that, I won't care.

madrag
Jul 23, 2012, 11:00 AM
I heard a lot about ZFS over the years. What I never hear is what ZFS does that will benefit me, as a Mac user. Until I hear that, I won't care.

I feel the same (although I still care about it), I would like to know more about it, maybe someone can shed some light.

ryedarrow
Jul 23, 2012, 11:01 AM
Bingo.

ZFS is overkill for a company (Apple) and is decidedly consumer and mobile.

Agreed. The best benefits of ZFS are for RAID users and data protection. Considering the last (cough) update of the Mac Pro is laughable, I don't see this type of change happening any time soon.

nuckinfutz
Jul 23, 2012, 11:02 AM
I heard a lot about ZFS over the years. What I never hear is what ZFS does that will benefit me, as a Mac user. Until I hear that, I won't care.

RAID like features built into the fs
Fights corruption by using checksums
Snapshots
dedupe
Pooled Storage
ZFS Cache (L2ARC)
Copy on Write (CoW)
Dynamic striping

Most of these features are pretty much Enterprise level. Apple could deliver a lighter weight more consumer friendly version more tuned to consumer needs.

KnightWRX
Jul 23, 2012, 11:04 AM
(2) What does a Mac user with one internal disk drive have to gain?

On single disk ?

- Snapshotting
- Copy on write
- Data integrity
- Encryption (built-in rather than filevault type after thought)
- Quotas

ZFS is much more than just a HFS+ replacement, it's a volume and storage pool management suite, not just a filesystem.

SirOmega
Jul 23, 2012, 11:05 AM
Well this is incredibly disappointing.

I had planned on buying one of the new Mac minis that I expect to launch with 10.8 this week. That combined with a USB 3.0 UASP external enclosure for multiple disks and ZEVO would finally allow me to come up with a lower-power way to replace my aging Windows Home Server now that MS has killed that platform.

Still looking for solutions to build that in-home server that are easy, low power and expandable.

KnightWRX
Jul 23, 2012, 11:08 AM
Well this is incredibly disappointing.

I had planned on buying one of the new Mac minis that I expect to launch with 10.8 this week. That combined with a USB 3.0 UASP external enclosure for multiple disks and ZEVO would finally allow me to come up with a lower-power way to replace my aging Windows Home Server now that MS has killed that platform.

Still looking for solutions to build that in-home server that are easy, low power and expandable.

Linux LVM 2.0 or FreeBSD's ZFS implementation.

There you go.

ghostlines
Jul 23, 2012, 11:11 AM
Bingo.

ZFS is overkill for a company (Apple) and is decidedly consumer and mobile.

The filesystem should be a priority even for these sectors. It's what we store our precious data on for Pete's sake. We need a modern day filesystem. Snapshots and copy-on-write sound very cool to me:)

dolphin842
Jul 23, 2012, 11:12 AM
Still looking for solutions to build that in-home server that are easy, low power and expandable.

I run a FreeNAS box (uses ZFS as the filesystem). It offers AFP network shares (and can even advertise specific datasets as compatible with Time Machine).

That said, I was hoping ZEVO would get some traction so I could start protecting my everyday data.

ryedarrow
Jul 23, 2012, 11:13 AM
Well this is incredibly disappointing.

I had planned on buying one of the new Mac minis that I expect to launch with 10.8 this week. That combined with a USB 3.0 UASP external enclosure for multiple disks and ZEVO would finally allow me to come up with a lower-power way to replace my aging Windows Home Server now that MS has killed that platform.

Still looking for solutions to build that in-home server that are easy, low power and expandable.

I don't understand your disappointment. Were you expecting these changes to be a surprise announcement? Can you not still buy a Mac Mini when they update?

nuckinfutz
Jul 23, 2012, 11:15 AM
The filesystem should be a priority even for these sectors. It's what we store our precious data on for Pete's sake. We need a modern day filesystem. Snapshots and copy-on-write sound very cool to me:)

Oh I'm sure it is. I just don't think ZFS is that fs that we need. We do need anti corruption, pooled storage, snapshots(Time Machine +) and other features.

Lindono
Jul 23, 2012, 11:21 AM
ZFS has no future on OS X now that Apple has gone down the Core Storage path. Any feature they could ever want can now be baked into Core Storage or HFS+ (assuming they don't replace that old dog). I would like to see them revise their HFS Compression implementation though. If you ask me, using xattrs for something that's supposed to be a file system feature is a no-no.

bedifferent
Jul 23, 2012, 11:44 AM
I recall ZFS implementation hinted at in old Leopard beta releases in 2006. Apple, as far as developers were concerned, seemed full steam ahead with ZFS.

Theeeen SunSystems went the way of the dodo, and mobile devices/iOS became Apple's main priority in 2007+.

ZFS would be fantastic, as many have stated, for single and multi-arrays. As well, it handles large [single] volumes much better, which would benefit "Time Machine" and large data backup(s).

HFS+ is long in the tooth.

Lindono
Jul 23, 2012, 11:48 AM
I recall ZFS implementation hinted at in old Leopard beta releases in 2006. Apple, as far as developers were concerned, seemed full steam ahead with ZFS.

Theeeen SunSystems went the way of the dodo, and mobile devices/iOS became Apple's main priority in 2007+.

ZFS would be fantastic, as many have stated, for single and multi-arrays. As well, it handles large [single] volumes much better, which would benefit "Time Machine" and large data backup(s).

HFS+ is long in the tooth.

Look up Core Storage. It does this. The only issue is there is only minimal support for actually manipulating it, and mountain lion doesn't add much. Perhaps in OS X 10.9...

Edit: incase anyone's wondering, it's currently used solely for FileVault 2's full disk encryption. That's about the extent of its capabilities for now. It's a full logical disk management system though, it's just in the early stages.

Marx55
Jul 23, 2012, 11:50 AM
Apple should bring ZFS to the Mac:

ZFS Self Healing (VIDEO)
http://hub.opensolaris.org/bin/view/Community+Group+zfs/selfheal

KnightWRX
Jul 23, 2012, 11:53 AM
I recall ZFS implementation hinted at in old Leopard beta releases in 2006. Apple, as far as developers were concerned, seemed full steam ahead with ZFS.

Theeeen SunSystems went the way of the dodo, and mobile devices/iOS became Apple's main priority in 2007+.

I think the issue was more than Sun ran into some legal issues with ZFS. NetApp held a patent over their block-level technology (WAFL) :

http://www.theregister.co.uk/2007/09/05/netapp_sues_sun_over_zfs/

It is speculated that that was the cause of dropping the project and it makes darn sense :

http://www.zdnet.com/apple-drops-zfs-project-3039838931/

nuckinfutz
Jul 23, 2012, 11:55 AM
Look up Core Storage. It does this. The only issue is there is only minimal support for actually manipulating it, and mountain lion doesn't add much. Perhaps in OS X 10.9...

Edit: incase anyone's wondering, it's currently used solely for FileVault 2's full disk encryption. That's about the extent of its capabilities for now. It's a full logical disk management system though, it's just in the early stages.

I'd like to see where Core Storage goes. Didn't see a whole lot of change in the WWDC '12 materials about Core Storage. I'm sure engineers are at hard work on adding a significant amount of features for a future release.

bedifferent
Jul 23, 2012, 11:58 AM
I think the issue was more than Sun ran into some legal issues with ZFS. NetApp held a patent over their block-level technology (WAFL) :

http://www.theregister.co.uk/2007/09/05/netapp_sues_sun_over_zfs/

It is speculated that that was the cause of dropping the project and it makes darn sense :

http://www.zdnet.com/apple-drops-zfs-project-3039838931/

Exactly. Unfortunately.

----------

Look up Core Storage. It does this. The only issue is there is only minimal support for actually manipulating it, and mountain lion doesn't add much. Perhaps in OS X 10.9...

Edit: incase anyone's wondering, it's currently used solely for FileVault 2's full disk encryption. That's about the extent of its capabilities for now. It's a full logical disk management system though, it's just in the early stages.

I'll check it out, thanks! :)

Lindono
Jul 23, 2012, 11:59 AM
I'd like to see where Core Storage goes. Didn't see a whole lot of change in the WWDC '12 materials about Core Storage. I'm sure engineers are at hard work on adding a significant amount of features for a future release.

Their only hint was something along the lines of "keep your eyes on this", in typical Apple fashion. Something's coming, but I'll be darned if I know what that something is.

SirOmega
Jul 23, 2012, 12:01 PM
I run a FreeNAS box (uses ZFS as the filesystem). It offers AFP network shares (and can even advertise specific datasets as compatible with Time Machine).

That said, I was hoping ZEVO would get some traction so I could start protecting my everyday data.

I really don't want to run a linux or BSD box. The whole goal of using a Mac was to make getting software easy. I wanted to have iTunes running in the background to share music and movies in the home, along with a variety of DLNA software (my Samsung and Vizio TVs are incredibly picky about the host DLNA software).

Oddly enough, I'm looking at running Windows 8 now instead of OSX because of things like storage spaces (while not completely like ZFS, it does offer data redundancy). I'd like to go with ReFS, which is a lot like ZFS, in the new Windows 8 Server OS but its $425 for the license, not to mention any hardware.

KnightWRX
Jul 23, 2012, 12:13 PM
I really don't want to run a linux or BSD box. The whole goal of using a Mac was to make getting software easy. I wanted to have iTunes running in the background to share music and movies in the home, along with a variety of DLNA software (my Samsung and Vizio TVs are incredibly picky about the host DLNA software).

My QNAP runs a busybox installation (Linux for embedded devices), works with iTunes and most picky DLNA server (Twonky Media). Linux boxes are easier to run this stuff on than Mac unfortunately.

Heck, why not just get a QNAP ? Low power, cheap, expandable (Optware for software, hot swap disk bays).

dolphin842
Jul 23, 2012, 12:32 PM
Edit: incase anyone's wondering, it's currently used solely for FileVault 2's full disk encryption. That's about the extent of its capabilities for now. It's a full logical disk management system though, it's just in the early stages.

Core Storage, being a logical volume manager, doesn't solve the data integrity problems HFS+ has. The main appeal of ZFS to many is that it addresses silent data corruption.

I really don't want to run a linux or BSD box. The whole goal of using a Mac was to make getting software easy. I wanted to have iTunes running in the background to share music and movies in the home, along with a variety of DLNA software (my Samsung and Vizio TVs are incredibly picky about the host DLNA software).

The latest FreeNAS 8.2 release has support for plugins (http://sourceforge.net/projects/freenas/files/FreeNAS-8.2.0/RELEASE/x64/plugins/). Looks like DLNA and an iTunes server are already available. Everything (including volume management, sharing, etc) is configured through a web GUI. Honestly, it was easier for me to set up than OS X file sharing. Using an old mATX case/PSU and some leftover RAM/drives, I was able to get a Sandy Bridge-based FreeNAS box going for ~$100.

KnightWRX
Jul 23, 2012, 12:38 PM
Using an old mATX case/PSU and some leftover RAM/drives, I was able to get a Sandy Bridge-based FreeNAS box going for ~$100.

One of his requirements was low power. One of the mini-ITX based Intel Atom solutions might be better. Intel and others have quite a few boards that come with CPU already mounted, and being mini-ITX, you can get much smaller cases.

SirOmega
Jul 23, 2012, 12:45 PM
One of his requirements was low power. One of the mini-ITX based Intel Atom solutions might be better. Intel and others have quite a few boards that come with CPU already mounted, and being mini-ITX, you can get much smaller cases.

That was the whole reason I wanted to go Mac mini was that I would still get a good CPU and sip power (11W idle IIRC). If I wanted to throw a video on it for Handbrake to process overnight it would still be OK. Atom doesn't really offer that, nor a NAS-solution.

Is there any way to virtualize FreeNAS under OSX? Is that even a good idea?

dolphin842
Jul 23, 2012, 01:45 PM
That was the whole reason I wanted to go Mac mini was that I would still get a good CPU and sip power (11W idle IIRC). If I wanted to throw a video on it for Handbrake to process overnight it would still be OK. Atom doesn't really offer that, nor a NAS-solution.

Don't forget to add external drives to the power consumption. Just as a point of reference, my FreeNAS box (with two green drives, a 65W Sandy Bridge Celeron, and an old/inefficient PSU) idles at 43W.


Is there any way to virtualize FreeNAS under OSX? Is that even a good idea?

I believe it's possible... the online docs have more information.

techsmith
Jul 23, 2012, 02:10 PM
I think if you want ZFS you are going to have to get Solaris.

The problem with ZFS on Mac OS X is (1) How to present a full featured ZFS system to a typical Mac user in a way that he can understand the concepts and options. (2) What does a Mac user with one internal disk drive have to gain?

Since they are making their own propitiatory SSD add-in cards, they could in theory have them configured as smaller multiple drives. Configured in ZFS, one drive would be for redundancy, and the OS would see the others as just one drive. Then you can get the self-healing features of ZFS. Even with no redundancy, having the ability to detect corruption via checksums is a step ahead because the filesystem at least knows that corruption occurred. Coupled with time-machine, files detected as corrupted can be recovered from the time machine backup if a backup exists. As for how to present the options to the user, they could have a check box for redundancy. If checked, user gets "Enhanced data protection" along with the cost in GB. If unchecked, user gets "Maximum space availability". I'm not 100% sure but I think ZFS will allow these options to be changed, so that they can ship the computer with redundancy enabled and the user can disable it should they need the space and don't care about the protection.

I don't know about everyone else here but I for one have lost data due to crashes/lock-ups more than once, and I'm not just talking the data being worked on at the time. Any time the system goes down before the drives have been properly unmounted I get very nervous. I've never had these kinds of issues with Windows NTFS. HFS+ is crap and the sooner it can be replaced the better.

dolphin842
Jul 23, 2012, 02:31 PM
I don't know about everyone else here but I for one have lost data due to crashes/lock-ups more than once, and I'm not just talking the data being worked on at the time. Any time the system goes down before the drives have been properly unmounted I get very nervous. I've never had these kinds of issues with Windows NTFS. HFS+ is crap and the sooner it can be replaced the better.

Both filesystems have journaling, but both still have a risk of data loss if the power is cut mid-write. At least Microsoft has announced a solution moving forward... the frustration on the OS X side is the radio silence for years on end...

TLewis
Jul 23, 2012, 03:05 PM
The latest FreeNAS 8.2 release has support for plugins (http://sourceforge.net/projects/freenas/files/FreeNAS-8.2.0/RELEASE/x64/plugins/). Looks like DLNA and an iTunes server are already available. Everything (including volume management, sharing, etc) is configured through a web GUI. Honestly, it was easier for me to set up than OS X file sharing. Using an old mATX case/PSU and some leftover RAM/drives, I was able to get a Sandy Bridge-based FreeNAS box going for ~$100.
If he doesn't need plugins, I'd recommend that he take a look at "nas 4 free (****************Gtg3O)" (there are actually no spaces in the name, but the forum censors the name if I take out the spaces -- I also had to use a URL shortener, sorry), which appears to be a fork of freenas V7. In particular, nas 4 free uses ZFS version 28 (the last free version), while freenas seems to be stuck at version 15. For details on versions, see ZFS release history (https://en.wikipedia.org/wiki/Zfs#Release_history) in wikipedia, under "ZFS pool version number".

firedownunder
Jul 23, 2012, 03:09 PM
Well this is incredibly disappointing.

I had planned on buying one of the new Mac minis that I expect to launch with 10.8 this week. That combined with a USB 3.0 UASP external enclosure for multiple disks and ZEVO would finally allow me to come up with a lower-power way to replace my aging Windows Home Server now that MS has killed that platform.

Still looking for solutions to build that in-home server that are easy, low power and expandable.

http://code.google.com/p/maczfs/

Suggested in an older thread, up and running about a year with no hiccups. Took a little reading and getting comfortable with terminal, but fairly straight forward. Loaded up my Mac Pro with drives and haven't looked back (got tired of waiting on Ten's Compliment).

quasinormal
Jul 23, 2012, 03:57 PM
Pardon my ignorance, but what does this mean for licence owners? I have my files backed up to a ZFS formatted volume. I just opened the pane in System preferences and downloaded update 1.0.3.

Would it be OK to keep on using ZFS?

smuckola
Jul 23, 2012, 04:11 PM
Hi guys. If you're wanting ZFS on Mac OS right now, especially if you want free software, come to http://maczfs.org/

We welcome Greenbytes to the MacZFS community and we hope that they'll collaborate well beyond just their legal obligation to release the source code under the CDDL.

haravikk
Jul 23, 2012, 04:25 PM
ZFS is overkill for a company (Apple) that is decidedly consumer and mobile.
Data integrity is a pretty important feature no matter how you use your machine, and with how much data.

I'm hoping that, at the very least, Apple has integrity checking in the works as a feature for CoreStorage.

After that, block-level snapshots would hugely benefit Time Machine, allowing for faster, more efficient backups. This wouldn't need any kind of user setup; just allow Time Machine to configure it automatically.

Finally, assigning an SSD as a read cache in a machine with both an SSD and one or more HDDs would be useful too, and benefit machines with those options; again, having a default option for machines so configured will cover that nicely, plus an installer option for quickly setting it back up if you re-install.


So, while ZFS may be most often used for setting up RAID style distribution and/or redundancy, integrity, snapshots and read/write caching are things that could definitely benefit users, the first two especially. That's not to mention proper (rather than bolted on) support for hard-linking and various other useful features that HFS+ can do, but only after a fashion.

Also, integrating the ability to create RAID through CoreStorage would be nice; definitely a power user feature but bringing everything under one heading would be much better overall.

TLewis
Jul 23, 2012, 05:32 PM
Hi guys. If you're wanting ZFS on Mac OS right now, especially if you want free software, come to http://maczfs.org/
Looks good, but is it really based upon the ancient ZFS version 9?

----------

Data integrity is a pretty important feature no matter how you use your machine, and with how much data.
It's interesting to note that one of the early ZFS developers was doing development on flaky hardware: occasionally, the data that ended up being written to disk was corrupted. However, ZFS automatically detected the corruption (when the data was read), and auto-corrected it. He supposedly ran for a while with this bad hardware.

fmaxwell
Jul 23, 2012, 05:55 PM
Once Apple dropped support for ZFS, it was dead on OS X. The very people who really want ZFS, those for whom their data is vitally important, are not going to accept a third-party implementation that could be broken at any time by any OS X upgrade, even necessary ones that address security vulnerabilities. They aren't going to accept the idea that the third-party supplier might fold, drop support for ZFS on OS X, or turn it into an internal product.

I'm all for ZFS, but it needs to be OS-independent and implemented in hardware -- the way that Drobo did, where the external box looked like a standard hard drive while actually being a RAID.

manu chao
Jul 23, 2012, 05:56 PM
I heard a lot about ZFS over the years. What I never hear is what ZFS does that will benefit me, as a Mac user. Until I hear that, I won't care.
Simple, it reduces the rate of file corruption noticeably. File corruption can happen with data just sitting on your HDD and with every copy operation (eg, the automatic defragging taking place on OS X in the background for small files, when upgrading your HDD, restore something from a backup, copy something from one device to another, upgrading to a new computer). And if the file corruptions happens in your disk directory, you can easily loose a lot of files in one fell swoop.

Lindono
Jul 23, 2012, 09:31 PM
Core Storage, being a logical volume manager, doesn't solve the data integrity problems HFS+ has. The main appeal of ZFS to many is that it addresses silent data corruption.


If I'm right about Apple's plans for Core Storage, they may be working on implementing their own file system, relying heavily on Core Storage. For example, HFS compression could in theory be implemented as full-disk compressing using Core Storage in the same way they implement encryption. They could even implement an entire file system backed by this new technology. It really is too early for you to claim any strengths or weaknesses in regard to Core Storage, seeing as how new it is. Once there is full support for all of its features in the OS, we should have some feel for where it's headed. As it stands, it isn't even clear what it does support, besides encryption.

AidenShaw
Jul 23, 2012, 11:04 PM
If I'm right about Apple's plans for Core Storage, they may be working on implementing their own file system, relying heavily on Core Storage. For example, HFS compression could in theory be implemented as full-disk compressing using Core Storage in the same way they implement encryption. They could even implement an entire file system backed by this new technology. It really is too early for you to claim any strengths or weaknesses in regard to Core Storage, seeing as how new it is. Once there is full support for all of its features in the OS, we should have some feel for where it's headed. As it stands, it isn't even clear what it does support, besides encryption.

But, isn't it true that most Apples hide the filesystem from the user?

Why do *anything* to improve something that the user never sees?

Lindono
Jul 23, 2012, 11:53 PM
But, isn't it true that most Apples hide the filesystem from the user?

Why do *anything* to improve something that the user never sees?

The user never sees 99% of what actually goes on in any modern OS... That's kinda the point of a GUI. Abstraction. They'll improve it because it will give them added flexibility to implement future features, as well as improve existing ones. Block-based Time Machine backups anyone? Simple to do with Core Storage.

ChrisA
Jul 24, 2012, 12:16 AM
...
I'm all for ZFS, but it needs to be OS-independent and implemented in hardware -- the way that Drobo did, where the external box looked like a standard hard drive while actually being a RAID.

Can't do that and still have it be ZFS. The point of ZFS is that off of RAID, Volume Management and the file system are ALL in the same layer. In other systems these functions are independent layers

What a Mac user would see in ZFS is that the concept of a "disk" goes away. there is on;y "storage" and if you need more of it you buy a physical disk and plug it in and if a disk fails you buy and other and plug it in then trash the failed part. Pretty much like what Drobo does, except that drobo looks like a disk to the OS, that part goes away.

You can't build it into the disk hardware because there needs to be a part of ZFS that sees all the drives and it is one layer thick so it can live in only one place. So it has to be above the hardware layer

Yes your idea would work but it would not be ZFS.

steve-p
Jul 24, 2012, 03:02 AM
Finally, assigning an SSD as a read cache in a machine with both an SSD and one or more HDDs would be useful too, and benefit machines with those options; again, having a default option for machines so configured will cover that nicely, plus an installer option for quickly setting it back up if you re-install.
I can't see anyone putting a great amount of effort into something specifically for SSD/HD combinations. Those combinations only exist at the moment because we are in a transition period between HD and SSD, purely because of the cost and capacity of SSD which is a relatively new technology. Within a fairly short timeframe it's likely that SSD storage cost will plummet and no-one will be buying conventional hard disks anymore anyway. 512GB SSDs today cost around 1/3 what they did 15 months ago.

haravikk
Jul 24, 2012, 03:38 AM
I can't see anyone putting a great amount of effort into something specifically for SSD/HD combinations. Those combinations only exist at the moment because we are in a transition period between HD and SSD, purely because of the cost and capacity of SSD which is a relatively new technology. Within a fairly short timeframe it's likely that SSD storage cost will plummet and no-one will be buying conventional hard disks anymore anyway. 512GB SSDs today cost around 1/3 what they did 15 months ago.
I'm not so sure; SSD's are fast approaching the point where it's starting to get more difficult to pack more data in without crippling efficiency, all SSD's today use a variety of tricks to keep the drives fast today.

So while the gap has certainly closed a bit, HDDs are still ahead in raw capacity, and don't have the same longevity concerns which, coincidentally, is another case to be made for integrity checking at the file-system/volume manager level :D

Still, it's definitely a much more niché case, but read-caching is a fairly straightforward technology to implement, but it needs to be done at a fairly low level in the OS if you want to speed up boot-times, which is one of the big advantages of SSD's for a lot of users. Besides, while the gap in capacity and price have been going down, it's still going to be some time before it closes enough to justify going all SSD.

----------

Core Storage, being a logical volume manager, doesn't solve the data integrity problems HFS+ has. The main appeal of ZFS to many is that it addresses silent data corruption.
It could in theory though couldn't it? It just needs to maintain a store of checksums for each block, where isn't that important, and verify blocks as they're fetched, and update the checksum as they're stored. It's much the same idea of how the encryption/decryption works as blocks are fetched/stored.

The more complex part is marrying it to data-redundancy, so corrupt files can be restored from a copy, or adding layers of error correction, or some combination of the two. I'm a big fan of ZFS' ability to do redundancy on a single drive, useful for a laptop when you want to benefit from the integrity checking; it seems to handle it better than mirroring two drive partitions would as well.

KnightWRX
Jul 24, 2012, 03:53 AM
The user never sees 99% of what actually goes on in any modern OS... That's kinda the point of a GUI. Abstraction. They'll improve it because it will give them added flexibility to implement future features, as well as improve existing ones. Block-based Time Machine backups anyone? Simple to do with Core Storage.

Don't get caught up in Aiden's comment, it was tongue in cheek. He didn't mean that Apple hides HFS+ from the user by saying "the filesystem", he was pointing at the fact that Apple tries its hardest to make file management not be based on the old paradigm of files and folders, but rather on librairies (iTunes, iPhoto, etc..). This is based on a lot of people that are non technical and talk about the "filesystem" meaning the folders and files they see in Finder. That's not the filesystems, that's your files. Finder is a file manager application, not a filesystem manager application, but to some people the distinction is quite lost.

They don't know that Finder is simply a userspace application that interacts with a standard API that calls into syscalls in the Kernel to get handles to files, which are managed by a VFS layer (open(), write(), read(), close() are the same syscalls in the kernel for every underlying storage type, no matter its actual filesystem) which then maps the call to the actual filesystem driver depending on which handle received the call.

This is quite all besides the point of the kernel structures being used to actually address userspace queries and map them onto the hardware. And the GUI isn't what abstracts this, it's the userspace/kernel space seperation that does. ;)

The CLUI hides as much detail of the modern OS as a the GUI does. You don't directly manipulation kernel structures, set hardware registers or interfere with the process scheduler on the CLUI you know. You interact with the same abstracted view of it all that the GUI presents, just as a text based format rather than with shiny icons.

Lindono
Jul 24, 2012, 04:09 AM
Don't get caught up in Aiden's comment, it was tongue in cheek. He didn't mean that Apple hides HFS+ from the user by saying "the filesystem", he was pointing at the fact that Apple tries its hardest to make file management not be based on the old paradigm of files and folders, but rather on librairies (iTunes, iPhoto, etc..). This is based on a lot of people that are non technical and talk about the "filesystem" meaning the folders and files they see in Finder. That's not the filesystems, that's your files. Finder is a file manager application, not a filesystem manager application, but to some people the distinction is quite lost.

They don't know that Finder is simply a userspace application that interacts with a standard API that calls into syscalls in the Kernel to get handles to files, which are managed by a VFS layer (open(), write(), read(), close() are the same syscalls in the kernel for every underlying storage type, no matter its actual filesystem) which then maps the call to the actual filesystem driver depending on which handle received the call.

This is quite all besides the point of the kernel structures being used to actually address userspace queries and map them onto the hardware. And the GUI isn't what abstracts this, it's the userspace/kernel space seperation that does. ;)

The CLUI hides as much detail of the modern OS as a the GUI does. You don't directly manipulation kernel structures, set hardware registers or interfere with the process scheduler on the CLUI you know. You interact with the same abstracted view of it all that the GUI presents, just as a text based format rather than with shiny icons.

I suppose so. I sometimes lose sight of where the actual abstraction lies. Everything looks like a bunch of CGPoints and NSViews. Heh.

till213
Jul 24, 2012, 04:45 AM
...
Fights corruption by using checksums
...


THIS!


Most of these features are pretty much Enterprise level. ...

True. However one can't stress enough that in the age of precious Gigabyte (Terrabyte!) Disks it's becoming more and more crucial to detect read-errors as soon as a single bit flips!

Why is that so important? Why not wait until that harddisk sector becomes *physically* unreadable? Because then it's already way too late!

Why?

Because typically a hard disk dies "a slow death". First some sectors become corrupt, means: "Bits flip over randomly" - the current HFS+ file system WON'T DETECT THESE BIT-FLIPS as long as that sector remains physically readable! And if you have a backup solution (and I hope you have!) then all those flipped (wrong!) bits will get backup-ed - possibly for weeks.

At some point you'll realise that your hard-disk starts making funny noise and eventually it will refuse to read certain sectors (physically unreadable), so it is only now that you'll realise - with the current HFS+ - that your harddisk just died.

But you're also very likely to notice that possibly much more data has become corrupted ("you cannot open certain JPEG data or other documents - but in the worst case you can, but the image data has changed, so you won't possibly notice until much much later that some pixels (or text...) have changed"). And that corrupted data has been backuped, maybe for weeks!

So good luck (manually!) finding a state in your backup where all data is in a good state!


So how does ZFS help here? Exactly in this situation when a single (or multiple) bits get randomly flipped - because of checksums which are constantly evaluated and updated upon read/write. And it makes sure that the file system is always in a consistent state! Partial file writes simply won't get committed if you loose power in that very moment etc. Moving a file is much more secure.


So in the age of thousands of precious photos on your harddisk (1) you absolutely want a filesystem as ZFS which detects logical errors - also as "consumer"! YES, YOU WANT IT! Repeat after me! YOU WANT ZFS ON YOUR CONSUMER MAC! Go and write Apple about it, use the feedback forms found on Apple's homepage!


(1) if you now mention "iCloud" or the like you still haven't understood the problem of "backing up corrupted data" - read again what I said about "flipped bits" which are still physically readable, so the current HFS+ doesn't notice that something went wrong (because it doesn't know nada, niet, nothing about the underlying file structure such as JPEG data), so it will happily be backed up, be it some TimeMachine backup harddisk, or synced back into the "Cloud".

KnightWRX
Jul 24, 2012, 05:18 AM
I suppose so. I sometimes lose sight of where the actual abstraction lies. Everything looks like a bunch of CGPoints and NSViews. Heh.

As opposed to a bunch of printf()s and fgets()s ? ;)

I'm just used to working with CLUI based programs and environnements. GUIs for me are a way of managing multiple screen sessions simultaneously.

haravikk
Jul 24, 2012, 08:00 AM
And if you have a backup solution (and I hope you have!) then all those flipped (wrong!) bits will get backup-ed - possibly for weeks.
I'm not sure this is true; Time Machine for example only copies files that have been updated since the last successful backup (if any). It doesn't recheck files that it doesn't think have changed, so any corruption won't be backed up.

The problem is that your backup drive(s) could be suffering similar corruption, which means that even if your original copy is fine, the backed up copy could have become corrupted. In theory it's a lot less likely as your backup drive should be seeing much less disk activity over time, so it should last a lot longer.

Anyway, my point being that a file that's become corrupted will only be backed up if you were to successfully open it (in spite of the corruption) then re-save it with the corruption still in place. Even then Time Machine's historic data may save you.

It's no alternative to proper integrity checking and additional error correction, but your backups should be comparatively okay.

macharborguy
Jul 24, 2012, 09:56 AM
I really like the ideas behind ZFS overall. Like the fact that ZFS looks at all of your storage as potentially useful for something other than "here is where my file is".

We really need to break away from the idea of a file system having this one-to-one relationship with the file browsing system (Finder, Explorer, etc). Just because I put 1 copy of a photo in my Photos folder, that doesn't mean there should only be 1 copy on the whole hard drive. Toss some extra copies around the free space further down the disk to use a backups (or on separate disks in the case of RAID-Z).

Another nice features is 'dedupe' which will try to save you on storage. Example, i do amateur web design and my hard drive is full of tons of backups of different websites, each one a wordpress blog. Most of the files in these wordpress directories are identical to eachother. Rather than having 10-100 copies of wp-login.php, it could just have 1 (plus 2 or 3 redundant copies spread around the drive) and have 10-100 references to that file.

The file system overall shouldn't simply be this thing that you dump files into and all it does is keep a record of where they are located on the drive. It needs to be active in keeping the files save, redundant, and ready to be moved in case of a bad sector. Its the difference between a filing cabinet and a secretary or archivist.

ChrisA
Jul 24, 2012, 12:03 PM
That was the whole reason I wanted to go Mac mini was that I would still get a good CPU and sip power (11W idle IIRC). If I wanted to throw a video on it for Handbrake to process overnight it would still be OK. Atom doesn't really offer that, nor a NAS-solution.

Is there any way to virtualize FreeNAS under OSX? Is that even a good idea?

You don't have to. FreeNAS is just Linux which is basically a UNIX-like OS. Mac OS X is already Unix. There is no need to "virtualize" anything. Mac OS can already serve files and uses about the same software that runs in the Linux based server.

Actually, if you have some spare Apple hardware around, maybe an old PPC mini. It can run Linux and would make a great server

dolphin842
Jul 24, 2012, 08:17 PM
You don't have to. FreeNAS is just Linux which is basically a UNIX-like OS. Mac OS X is already Unix. There is no need to "virtualize" anything. Mac OS can already serve files and uses about the same software that runs in the Linux based server.

Actually, if you have some spare Apple hardware around, maybe an old PPC mini. It can run Linux and would make a great server

FreeNAS is based on FreeBSD, not Linux, so it's actually closer in heritage to OS X than Linux. However, you can't just 'run' FreeNAS: It's a full FreeBSD OS distribution, so you'd have to virtualize it. You can't just pick out the FreeNAS-specific stuff and run it as-is on top of OS X.

Also, Linux usually doesn't run ZFS natively (i.e. in the kernel) due to licensing restrictions.

gbytes1
Jul 24, 2012, 09:34 PM
Greenbytes focus has been real time block level deduplication inside ZFS.
Greenbytes has developed technology which makes this fast and reliable.

I might imagine that 5 to 10x the usable storage on a MacBook SSD might
Be an attractive feature....stay tuned

AidenShaw
Jul 24, 2012, 09:45 PM
I might imagine that 5 to 10x the usable storage on a MacBook SSD might be an attractive feature....stay tuned

Those numbers are absurd.

A stock MacBook and apps might see a few percent savings with de-dupe. (There's one copy of Apple OSX, and one of each app.)

If one typically cloned photo or video libraries, the clones would take virtually no space until the clone is modified.

But "5x to 10x" is absurd.

till213
Jul 25, 2012, 07:01 AM
I'm not sure this is true; Time Machine for example only copies files that have been updated since the last successful backup (if any). It doesn't recheck files that it doesn't think have changed, so any corruption won't be backed up.

Okay, okay, so I was hyperventilating a bit ;) You're absolutely right off course!

However assume that the bits flip during a write operation - then the file becomes corrupted on your disk and the time stamp changes. Causing Time Machine to backup it in the next cycle... so I am also not totally off with my claim ;)


The problem is that your backup drive(s) could be suffering similar corruption, which means that even if your original copy is fine, the backed up copy could have become corrupted.

And I did not even mention that scenario...


Anyway, my point being that a file that's become corrupted will only be backed up if you were to successfully open it (in spite of the corruption) then re-save it with the corruption still in place. Even then Time Machine's historic data may save you.

It's no alternative to proper integrity checking and additional error correction, but your backups should be comparatively okay.

Off course you can go back in backup history for a particular file. But chances are if one file has become corrupted that other files have become corrupted and backed up since a long time! So you cannot simply "go back one week" (the time span where you found a good copy of that file you noticed went corrupt) and restore that entire state of your backup.

You would need to find the earliest point in time were "silent corruption" started. And that's a lot of pain...!

Koppenhoefer
Aug 8, 2013, 02:31 AM
I run a FreeNAS box (uses ZFS as the filesystem). It offers AFP network shares (and can even advertise specific datasets as compatible with Time Machine)..

Hello.. Are you able to use that AFP share for NetworkHomeDirectories and/or PortableHomeDirectories for 10.8 OS X (Mountain Lion) users? I'm concerned that ZFS doesn't play well with the Library/ folder?

dolphin842
Aug 8, 2013, 03:52 PM
Hello.. Are you able to use that AFP share for NetworkHomeDirectories and/or PortableHomeDirectories for 10.8 OS X (Mountain Lion) users? I'm concerned that ZFS doesn't play well with the Library/ folder?

I've never used networked home directories (either with HFS+ or ZFS) so I'm not sure. That said, if you browse the Zevo forums, you'll find that using ZFS for a home directory results in some cosmetic issues (e.g. custom icons not displaying). Serving ~/Library over AFP might or might not mask that behavior.

grahamperrin
Jun 8, 2014, 04:02 AM
Just for the record: around two weeks after Oracle announced its intention to acquire GreenBytes (http://www.oracle.com/us/corporate/acquisitions/greenbytes/index.html), I made an enquiry about source code. Addressed to someone at GreenBytes … but then the same day (2nd June), the acquisition completed.

GreenBytes' technology "is expected to enhance Oracle's ZFS Storage Appliances" – so I don't expect Oracle to develop ZEVO for OS X. Still, I hope that something good can be done with at least some of the code.