Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Apple Looking To Port ZFS for Mac OS X?
- Thread starter MacRumors
- Start date
- Sort by reaction score
My read of the Wikipedia article was that only the metadata is endian agnostic-- the files themselves are "just an array of bytes" that the application needs to sort out.Doctor Q said:
Of course, if the sum of my knowledge came from Wikipedia, I'm not standing on terribly solid ground with this...
So if I set up my external HD in this format, how compatible with windows is it?
I ask because the current format I use on my external, in order to be compatible with windows, limits the size of the files I can import in one hit to 5 gig or something.. I'd like to see that not be the case anymore...
I ask because the current format I use on my external, in order to be compatible with windows, limits the size of the files I can import in one hit to 5 gig or something.. I'd like to see that not be the case anymore...
Marx55 said:
Oh, hell no. If such a move would require applications written for OS X to be recompiled and altered so that they could run on a Mac Solaris then I think that'd be the straw that broke the camels back for developers. They've already been forced to rewrite their applications for OS X and now for Intel processors (and Adobe still hasn't done it) so what we need more than anything else at the moment is a period of stability.
At some point in the future it may well make sense that OS X be replaced with something new but I really don't think that time is now. One transition at a time, please...
mackeeper said:
Hear hear! At least tell me what this *might* be and when. Otherwise, what kind of rumor is it?
Marx55 said:
How is an operating system not even marketed in front of me anywhere at all going to be the best? It can be good, but the Mac is the best for me just because I've seen that and Windows, and not much more. At least Apple's Mac OS is getting some increasing coverage and public attention...
mackeeper said:
Hey, the average consumer only cares about the specs. Hit them with HFS+ can only store 2^63 bytes on a disk while ZFS can store 2^64 bytes. TWICE AS MUCH! That'll wow 'em.
Very American indeed
Well i'm glad to see that Apple are looking into something even bigger and better, how very american of them!
Looking forward to 10.6 when this may be implemented
Well i'm glad to see that Apple are looking into something even bigger and better, how very american of them!
Looking forward to 10.6 when this may be implemented
OK, I got curious and checked out some screenshots for Solaris 10. Umm, I'm unimpressed. Even if it's lightning fast and can pour coffee, it still looks horrid, just like Windows, and I can't see how that would be easy to use. Maybe easy for a geek, but not a guy like me.
Marx55 said:
Still living in the late 80's eh? Solaris stopping being the OS to gun for sometime in the mid 90's. Sun stopping being a company to watch shortly thereafter. Sad really, as Sun had the best thing going for almost 20 years before they diddled it away with mismanagement.
Isn't OS X modular or something, so wouldn't adding support for another file system be relativly simple? Yes, you'd need some major QC, but if Windows can read and write to HFS+ just by installing a program, doesn't that mean it's not hard to make an OS be able to read a file system? Why couldn't we see ZFS in 10.5?
Well, MacDrive is more than your typical Windows application--a lot more. It just looks like an application from the user perspective. File system support can be adjusted after installation without breaking things if you're adding and not replacing.thejadedmonkey said:
In the case of ZFS, I think it's likely that a Software Update will be provided to include this functionality sometime after the launch of 10.5 (but not likely in time for the launch--given the utter lack of details about Leopard so far, I get the distinct impression that they're going to be working right down to the wire and don't have time to test a new filesystem on top of that). After Leopard is updated with ZFS support (maybe they'll roll it into 10.5.2 or something), you'll be able to format and install using ZFS. They may even provide a conversion utility to "upgrade" your disks to ZFS, like Microsoft did with NTFS.
If they choose to replace HFS+ with ZFS, it won't be until 10.6, unless they've already been working on it and we don't officially know about it yet.
Here are some explanations of ZFS features and benefits taken from the comments made on ArsTechnica ( which was the first to pick up the news ) :
"I see ZFS as a perfect fit. It could really bring "ipod-like simplicity" to the file system.
Throw another drive in the system? Your system drive simply grows by the added size. Want to see what your files looked like 10 days ago? Drag a slider in "Apple TimeWarp" and see what the system looked like. Backup could be made so much more bulletproof with built in snapshot support."
---
"ZFS is all about increasing reliability through extra checks and redundant storage at the filesystem level. Reiser4 is largely about advancing the state of filesystem semantics, and does no more to protect your data from loss than any other current filesystem."
---
"I must admit that I am not intimate with all the details about ZFS but as far as I have understood there is no requirement of extra disk(s) from ZFS. You can use ZFS on a single disk, sure you can't get the RAID-Z capability but all of the other sweet things are there. Also I would not think the CPU cycles "lost" for checksumming are that big a thing with current CPUs and frankly we do "lose" lots of cycles already with Spotlight and mach kernel.
Some benchmarks I saw a few moths back on pre-versions of ZFS were very promising, speed will probably not be an issue."
---
"John, I think your claim that "ZFS isn't really appropriate for all of the markets the Apple serves" needs more substatiation than "there are file systems that do things faster and more reliably".
I won't argue that some filesystems are faster; ZFS is still pretty new and we there's still more tuning to do. Others mentioned checksumming... so far we haven't found it to be much of a burden on modern hardware, but it can be turned off ('zfs set checksum=off') if you determine that your CPU cycles are better used elsewhere.
However, I'm unaware of any filesystem that is more reliable. You mentioned that "to take advantage of all reliability features that ZFS has to offer, you need multiple disks". What features are you thinking of? It's true that you need multiple disks to use mirroring or raid-z, and therefore to use self-healing. However, that is true of any other filesystem/volume manager, so I don't see it as a disadvantage for ZFS. Even using only a single disk, everything is checksummed, it is always consistent on disk, and ZFS even stores multiple copies of the metadata.
So what features would be useful to home users? Others have already mentioned snapshots...
Home users are often less equipped to deal with data loss than enterprise customers, so I think that the increased data integrity will be a win on desktops (even single-disk) too. I've seen several instances of HFS+ (and other filesystems) becoming corrupt, presumably due to crashing at the wrong moment, that would have been prevented by ZFS. Plus, there are some non-catastrophic drive failures, which would be recoverable with ZFS (even with a single disk)."
---
"If this pans out, it will be great news for Apple customers. ZFS incorporates some excellent ideas, and may fundamentally change the way people look at storage. In a way, it provides an abstraction similar to virtual memory, only for disk based storage. As such, adding storage is hardly any more difficult than adding memory. Since Sun opened the ZFS source, it may become as relevant in the coming decades as UFS/FFS have been in the past. At the very least, it is slated for inclusion in DragonFlyBSD.
As far as performance, restructuring the storage stack actually provides some excellent opportunities. The level of integration with ZFS allows for much better IO scheduling, increased parallelism by avoiding unnecessary data dependencies, as well as eliminating the RAID-5 write hole. These are not small advantages. RAID-Z in particular has huge benefits over RAID-5, in that it doesn't require a read-modify-write cycle, or the added cost and complexity of NVRAM caches.
In any case, the utility of ZFS would be more than welcome, even on a single-disk desktop or laptop. There are really two issues here; the ability to protect your data from hardware errors, and to protect it from user error. HFS+J and UFS fail miserably in both ways. With either of them, a single bad sector or other minor problem can destroy an entire filesystem. On the other hand, the ZFS on-disk format is extremely robust, and the nature of the filesystem allows for simple and elegant solutions to both of these issues.
The amount of data corruption due to buggy ATA hardware and disk write caches is far greater than with enterprise class hardware. There are also the common problems with removable media and external drives being pulled. Beyond that, heat related problems are also common with laptops. These are all basically nonexistent with ZFS, since the on-disk data is robust, always consistent, and verifiable. While the infrastructure to support removable media may not be in place at present, there is no reason why ZFS couldn't be used on everything from flash media to iPod's as well. It would certainly be a welcome replacement for FAT, that is for sure.
The other problem plaguing desktop storage is in the ability to make backups convenient and often. ZFS snapshots would greatly simplify this problem. Together with the possibility of a native encrypted filesystem, it would also be far superior to the current FileVault hack. While FileVault is a great feature, the implementation sucks, and it really complicates backup. Snapshots open up a whole range of possibilities for full and incremental backups. User directories less than 4GB could be burned to DVD with a single click, with or without FileVault."
---
"Actually, ZFS stores *all* metadata redundantly. So we can actually correct some hardware errors, even on single disk pools. And we have some neat tricks for spreading out the copies as much as possible, so if you zero out (say) the first third of your disk, we can still navigate all the metadata and at least provide access to everything that's left. If you have multiple disks, the copies will be on different disks. (BTW, we call it the "uberblock", and it has even more copies.) FYI, these copies do cost something, but since we compress all metadata, the amount of disk space used by metadata is still similar to other filesystems."
"I see ZFS as a perfect fit. It could really bring "ipod-like simplicity" to the file system.
Throw another drive in the system? Your system drive simply grows by the added size. Want to see what your files looked like 10 days ago? Drag a slider in "Apple TimeWarp" and see what the system looked like. Backup could be made so much more bulletproof with built in snapshot support."
---
"ZFS is all about increasing reliability through extra checks and redundant storage at the filesystem level. Reiser4 is largely about advancing the state of filesystem semantics, and does no more to protect your data from loss than any other current filesystem."
---
"I must admit that I am not intimate with all the details about ZFS but as far as I have understood there is no requirement of extra disk(s) from ZFS. You can use ZFS on a single disk, sure you can't get the RAID-Z capability but all of the other sweet things are there. Also I would not think the CPU cycles "lost" for checksumming are that big a thing with current CPUs and frankly we do "lose" lots of cycles already with Spotlight and mach kernel.
Some benchmarks I saw a few moths back on pre-versions of ZFS were very promising, speed will probably not be an issue."
---
"John, I think your claim that "ZFS isn't really appropriate for all of the markets the Apple serves" needs more substatiation than "there are file systems that do things faster and more reliably".
I won't argue that some filesystems are faster; ZFS is still pretty new and we there's still more tuning to do. Others mentioned checksumming... so far we haven't found it to be much of a burden on modern hardware, but it can be turned off ('zfs set checksum=off') if you determine that your CPU cycles are better used elsewhere.
However, I'm unaware of any filesystem that is more reliable. You mentioned that "to take advantage of all reliability features that ZFS has to offer, you need multiple disks". What features are you thinking of? It's true that you need multiple disks to use mirroring or raid-z, and therefore to use self-healing. However, that is true of any other filesystem/volume manager, so I don't see it as a disadvantage for ZFS. Even using only a single disk, everything is checksummed, it is always consistent on disk, and ZFS even stores multiple copies of the metadata.
So what features would be useful to home users? Others have already mentioned snapshots...
Home users are often less equipped to deal with data loss than enterprise customers, so I think that the increased data integrity will be a win on desktops (even single-disk) too. I've seen several instances of HFS+ (and other filesystems) becoming corrupt, presumably due to crashing at the wrong moment, that would have been prevented by ZFS. Plus, there are some non-catastrophic drive failures, which would be recoverable with ZFS (even with a single disk)."
---
"If this pans out, it will be great news for Apple customers. ZFS incorporates some excellent ideas, and may fundamentally change the way people look at storage. In a way, it provides an abstraction similar to virtual memory, only for disk based storage. As such, adding storage is hardly any more difficult than adding memory. Since Sun opened the ZFS source, it may become as relevant in the coming decades as UFS/FFS have been in the past. At the very least, it is slated for inclusion in DragonFlyBSD.
As far as performance, restructuring the storage stack actually provides some excellent opportunities. The level of integration with ZFS allows for much better IO scheduling, increased parallelism by avoiding unnecessary data dependencies, as well as eliminating the RAID-5 write hole. These are not small advantages. RAID-Z in particular has huge benefits over RAID-5, in that it doesn't require a read-modify-write cycle, or the added cost and complexity of NVRAM caches.
In any case, the utility of ZFS would be more than welcome, even on a single-disk desktop or laptop. There are really two issues here; the ability to protect your data from hardware errors, and to protect it from user error. HFS+J and UFS fail miserably in both ways. With either of them, a single bad sector or other minor problem can destroy an entire filesystem. On the other hand, the ZFS on-disk format is extremely robust, and the nature of the filesystem allows for simple and elegant solutions to both of these issues.
The amount of data corruption due to buggy ATA hardware and disk write caches is far greater than with enterprise class hardware. There are also the common problems with removable media and external drives being pulled. Beyond that, heat related problems are also common with laptops. These are all basically nonexistent with ZFS, since the on-disk data is robust, always consistent, and verifiable. While the infrastructure to support removable media may not be in place at present, there is no reason why ZFS couldn't be used on everything from flash media to iPod's as well. It would certainly be a welcome replacement for FAT, that is for sure.
The other problem plaguing desktop storage is in the ability to make backups convenient and often. ZFS snapshots would greatly simplify this problem. Together with the possibility of a native encrypted filesystem, it would also be far superior to the current FileVault hack. While FileVault is a great feature, the implementation sucks, and it really complicates backup. Snapshots open up a whole range of possibilities for full and incremental backups. User directories less than 4GB could be burned to DVD with a single click, with or without FileVault."
---
"Actually, ZFS stores *all* metadata redundantly. So we can actually correct some hardware errors, even on single disk pools. And we have some neat tricks for spreading out the copies as much as possible, so if you zero out (say) the first third of your disk, we can still navigate all the metadata and at least provide access to everything that's left. If you have multiple disks, the copies will be on different disks. (BTW, we call it the "uberblock", and it has even more copies.) FYI, these copies do cost something, but since we compress all metadata, the amount of disk space used by metadata is still similar to other filesystems."
Some info and quotes from the Wiki ZFS page:
ZFS is a 128-bit file system, which means it can provide 16 billion billion times the capacity of current 64-bit systems. The limitations of ZFS are designed to be so large that they will never be encountered in any practical operation. When contemplating the capacity of this system, Bonwick stated "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans.
"if a user was creating 1,000 files a second it would take them about 9,000 years to reach the limit of the number of files."
ZFS is a 128-bit file system, which means it can provide 16 billion billion times the capacity of current 64-bit systems. The limitations of ZFS are designed to be so large that they will never be encountered in any practical operation. When contemplating the capacity of this system, Bonwick stated "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans.
"if a user was creating 1,000 files a second it would take them about 9,000 years to reach the limit of the number of files."
Case sensitive
One thing I noted from the Wikipedia page was that ZFS is case-sensitive. Would Apple switch to a case-sensitive filing system at this juncture? Would people care?
One thing I noted from the Wikipedia page was that ZFS is case-sensitive. Would Apple switch to a case-sensitive filing system at this juncture? Would people care?
It doesn't necessarily matter. Case sensitivity can still be ignored by the OS if desired, unless ZFS has reinvented the wheel there...case sensitivity is something that has to be incorporated in order for the option to exist, but doesn't necessarily have to be used.neilw said:
I'm not sure if I'm saying this well. If there is no case sensitivity at the filesystem level, you don't have the option to be case-sensitive. But after including the option, you can still go either way. Does that make sense?
better if the FS offers the option
The file system normally has APIs to do file lookups - if that API doesn't have a "case-insensitive" option, it can be much slower to find a particular file (you might have to write code to get all of the file names, and do a case-blind compare to see if it is the one you want).matticus008 said:
My thoughts too. Perhaps this is aimed primarily at virtualization/sharing with Solaris in 10.5 on Intel Macs or is meant only for OS X Server?!?nagromme said:
If it is on the roadmap for default FS, even for 10.6 we'll probably find out at WWDC.
B
matticus008 said:
It does make perfect sense to me. But personally I don't think it's worthwhile. Don't treate user as programmer, it's too much information to them(us).
If oneday you happened to type "mY doCument" you'll be freak out. yell"I just upgraded to ZHFDEIF system two days ago, where they put my files."