PDA

View Full Version : ZFS in Latest Leopard Builds




MacRumors
Dec 18, 2006, 03:40 PM
http://www.macrumors.com/images/macrumorsthreadlogo.gif (http://www.macrumors.com)

Several sites are reporting on ZFS support that is now built into the latest build of Mac OS X (Leopard) which was seeded last week.

The ZFS team at Sun was contacted (http://www.macrumors.com/pages/2006/04/20060430200158.shtml) by Apple in April about porting ZFS to Mac OS X. Arstechnica previously discussed (http://arstechnica.com/staff/fatbits.ars/2006/8/15/4995) advantages of ZFS as a file system. These include

- Efficient storage and handling of very small files.
- Logical volume management through a pooled storage model.
- Improved data integrity using checksums on all data.
- Snapshots

Mac4Ever first posted (http://mac4ever.com/news/27485/zettabyte_sur_leopard/) a screenshot of the ZFS formatting option now in Leopard:

http://images.macrumors.com/article/zfsleopard.png

More information on ZFS is available at the ZFS homepage (http://opensolaris.org/os/community/zfs/whatis/) and Wikipedia (http://en.wikipedia.org/wiki/Zfs).



failsafe1
Dec 18, 2006, 03:43 PM
Could this mean revolutionary things ahead for the next OS? Perhaps better integration with new hardware?

m-dogg
Dec 18, 2006, 03:47 PM
So will ZFS just be supported by Leopard, or will the entire file system be based off this?

I remember reading that article on Arstechnica and the idea of Time Machine with ZFS sounded pretty interesting.

jmbear
Dec 18, 2006, 03:47 PM
Cool! I love ZFS!

killmoms
Dec 18, 2006, 03:48 PM
ZFS makes me a little bit horny. Just a bit. First filesystem to do that since BFS, actually.

OH NO I'M CHEATING ON MY FIRST FILESYSTEM LOVE! :eek:

eskalation.dk
Dec 18, 2006, 03:49 PM
wtf? i hav absolutely no idea what it is, but it sounds cool... YAY!

Paranoidmarvin
Dec 18, 2006, 03:49 PM
HFS is getting tired, you can tell by the number of names they have tacked onto the end (HFS+ Journaled)!
Bring on ZFS, it makes sense with Time machine

killmoms
Dec 18, 2006, 03:51 PM
Bring on ZFS, it makes sense with Time machine

If they actually use its built-in diff/snapshot ability for Time Machine instead of the ugly hack onto HFS+ we saw as of the original dev preview, I might have to go offer my body up to any of Apple's filesystem developers.

50548
Dec 18, 2006, 03:51 PM
For us poor plebeians, what the hell does the adoption of ZFS bring anyway? Is it just another geeky filesystem, or something that might really make a difference for ordinary users?

At least HFS enabled much bigger storage and more sensible partitioning back then...could any of you clarify this issue for the non-file server admins out there? Thanks.

SeaFox
Dec 18, 2006, 03:52 PM
Awesome. The idea of being able to use ZFS in OSX was quite exciting to me when I first read the rumors before. Especially the "pooled storage" method. I assume this would be the end of one's home folder being full because the drive it resides on is full, since an additional drive added to the machine would just add to the shared pool for storage. The home folder would no longer be limited to a single device. That and the redundancy and backup features.

Ars Technica has suggested that this would be required for Time Machine to function, but isn't Time Machine already included in the developer's previews?

Peace
Dec 18, 2006, 03:53 PM
For us poor plebeians, what the hell does the adoption ZFS bring anyway? Is it just another geeky filesystem, or something that might really make a difference for ordinary users?

At least HFS enabled much bigger storage and more sensible partitioning back then...could any of you clarify this issue for the non-file server admins out there? Thanks.


The Zettabyte file system is 128-bit for one.

kainjow
Dec 18, 2006, 03:55 PM
Zis iz zery awezome! :D

50548
Dec 18, 2006, 03:55 PM
The Zettabyte file system is 128-bit for one.

Meaning..?

Westside guy
Dec 18, 2006, 03:56 PM
ZFS isn't currently bootable; but I'm not sure that matters since OS X uses a separate boot partition.

The big draw with ZFS, AFAIK, is it has the ability to manage differential file snapshots. So Time Machine would be a heck of a lot faster than if it's managed by an application within the OS.

I would bet that, with Leopard, it'll be an option but not the default. Analogous to the way case-sensitive HFS+ is currently handled. I'd be tempted to try it if it's available.

I wonder how it will handle resource forks though? I don't know if ZFS has that sort of thing available.

secretz
Dec 18, 2006, 03:56 PM
The Zettabyte file system is 128-bit for one.

And how that means what to regular users?

Edit: Slow :)

Peace
Dec 18, 2006, 03:56 PM
Meaning..?

A later release of Leopard will be 128 bit instead of 64 bit ;)

sweetandsour
Dec 18, 2006, 03:57 PM
For us poor plebeians, what the hell does the adoption of ZFS bring anyway? Is it just another geeky filesystem, or something that might really make a difference for ordinary users?

At least HFS enabled much bigger storage and more sensible partitioning back then...could any of you clarify this issue for the non-file server admins out there? Thanks.


There are at least 3 links helpfully given in the article at the begining of the thread to clear, useful, not over-technical discussion and description of zfs. You should read them.

kainjow
Dec 18, 2006, 03:58 PM
ZFS isn't currently bootable; but I'm not sure that matters since OS X uses a separate boot partition.

Check out ZFS Boot (http://opensolaris.org/os/project/zfsboot).

killmoms
Dec 18, 2006, 04:01 PM
Meaning..?

Seriously, just read the John Sicarusa Fat.Bits blog linked in the newspost. It's easy to read and explains all the advantages of ZFS.

MacVault
Dec 18, 2006, 04:07 PM
I don't really know what I'm talking about as I'm no filesystem guru by any means, but I do remember rumors in the past of Windows and/or OS X going to some kind of sql-database-driven-super-efficient-fast-revolutionary-metadata filesystem. My question is how does ZFS compare to that??? How does it compare to the [vaporware] WinFS???

DXoverDY
Dec 18, 2006, 04:08 PM
Let's run through a simple example.

You have 1 hard drive right? You are running out of space. You go buy another drive. You want to add it to the system but now you have to decide whether you want that drive to just handle storage of files, and if so which files? Well, how about this instead? You can add the drive as a pool. It then "magically" appears as if your original drive is now x gigabytes larger than it is since it is using two drives as a pool. Convenient no?

Or how about you want to backup that drive instead? Ok, add a drive to the system. Add it to the pool, tell it to mirror the drive instead. It now copies the data from one drive to the other and any changes mirror the other. If a corrupt file is on drive 1 (your original working copy) it checks the other to see if the backup is non-corrupt. If so, it opens that file, and copies the good data to the original drive as good data. It does this with Checksums of the files.

Another for the geekiness factor is RAIDZ. One drive or two can be a parity drive. Ever use PAR files? Yup very similar. You have files or a whole drive disappear? You can pretty much restore it from the parity drive if enough of the data still exists, etc.

Or how about you like having "versions" of your filesystem. You're about to update OS X to 10.5.2 and you're afraid it might break your system. So you create a ZFS snapshot. It now olds this "snapshot" of your filesystem and the files in it. You install the new update, it does indeed bork your computer. So you tell OS X and ZFS to use a previous snapshot, boom. You're back to 10.5.1 and it works.

Also the idea of using ZFS with time machine, is really really cool. Also, on the fly compression of your files. With no real performance hit. Another neat one is built in encryption. So you'll be able to have file vault at the filesystem level rather than the OS X application/OS level.

sparkleytone
Dec 18, 2006, 04:11 PM
Think about these things:

1) Shutting down your Mac Pro, opening it up and adding another hard drive. When you boot back into OS X, it tells you that you have added another hard drive and asks whether or not you'd like to add it to your "Macintosh HD" or whatever you call it. When you say yes, it automatically adds the space to your available space, and also automagically spreads your data out in the most efficient ways.

2) One of your hard drives on your Mac begins to corrupt data. Your filesystem automatically detects this, AND heals it by recreating the data on one of the other HD's in your ZFS pool.

3) Each user on your system literally has their own entire filesystem devoted to their data. No reasons to use quotas, as the actual filesystem is only a certain size that you set.

4) Each of the very small text files on your system, of which there are thousands, take up the size that they are instead of the fs block size. Lots of unused and previously unusable space saved.

5) Having multiple "snapshots" of your data, in the paradigm of Time Machine. Except get this...each snapshot is a snapshot of how the blocks look, not your files. So the size of each snapshot is only however much space it takes to record what blocks have changed.

shawnce
Dec 18, 2006, 04:13 PM
So will ZFS just be supported by Leopard, or will the entire file system be based off this?

ZFS is a file system (a very capable modern one) so the later part of your question doesn't make sense. If you mean "will Apple make ZFS the default file system for Mac OS X?" then I would say "not likely any time soon (but I would be happy to be wrong)".

aspro
Dec 18, 2006, 04:13 PM
This sounds very interesting, always good to be able to use the latest new tech.

From the Wikipedia article: "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

I wonder how long until we hit that limit :P

kresh
Dec 18, 2006, 04:15 PM
Mac4Ever first posted (http://mac4ever.com/news/27485/zettabyte_sur_leopard/) a screenshot of the ZFS formatting option now in Leopard:



I did not see a menu option for compression. I really like ZFS' 2-3x (on the fly) compression. I hope it is supported.

Flowbee
Dec 18, 2006, 04:18 PM
This is one of the reasons I love being an OS X user. Can't wait to see what Apple eventually does with it.

notjustjay
Dec 18, 2006, 04:18 PM
From the Wikipedia article: "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."


Mmm. 640K ought to be enough for anybody, right? :D

ricksbrain
Dec 18, 2006, 04:23 PM
Doesn't Time Machine ask to configure a hard drive once it's attached? Would likely be ZFS, no?

autrefois
Dec 18, 2006, 04:23 PM
I think I'm beginning to understand what this is all about. People are making it sound like there are no limits, but for the average person is it correct to say that it doesn't mean you'll have unlimited space, just that ZFS can handle files/etc. of any size while having each individual thing take up less space, am I right?

each snapshot is a snapshot of how the blocks look, not your files. So the size of each snapshot is only however much space it takes to record what blocks have changed.

So in very simplified terms, snapshots are basically like wikipedia? It just keeps tracks of changes instead of creating a whole new copy of it, correct?

Trekkie
Dec 18, 2006, 04:23 PM
This would be very exciting feature to have. The ability to have one single Macintosh HD and just keep adding drives (internal or external) to increase the space.

Sweeeet. :D

shawnce
Dec 18, 2006, 04:24 PM
Doesn't Time Machine ask to configure a hard drive once it's attached? Would likely be ZFS, no?

Don't read to much into it.

SiliconAddict
Dec 18, 2006, 04:26 PM
Somewhere out there John Siracusa just wet himself. :D

killmoms
Dec 18, 2006, 04:27 PM
Somewhere out there John Siracusa just wet himself. :D

I hope he posts pics. :D

ChrisA
Dec 18, 2006, 04:27 PM
For us poor plebeians, what the hell does the adoption of ZFS bring anyway? Is it just another geeky filesystem, or something that might really make a difference for ordinary users?

Did you read this?
http://opensolaris.org/os/community/zfs/whatis/

1) All operations are "copy on write".
data are never over written you can always go back to last month's version . The system saves space by only writing changes.

2) RAID and volume managment are integratd into the filesystem layer
File systems can span disks and you can add a disk to a file system without mess up the data.

Finally Mac OS can get rid of those disk drive icons. There can just be "storage" Much like you don't care much about how many RAM chips you have, you only care about the total amount of RAM. Disk can be like that too. Just open the box and slide in one more disk and the rest is "magic". Just like with RAM.

3) end to end checksums: These is no more thinking "maybe the file is corrupt". If anything goes wrong you will be told and conversely if you are not told you can be sure there is nothing wrong

4) there is not enough data on Earth to fil a ZFS file system. Or at least if you wrote to a disk 24x7 you would not live long enough to fill up ZFS. (although you would fill up quite a few physical drives)

5) priority based scheduling of I/O -- stuff that has to be read from the disk fast gets done ahead f stuff that can wait. ZFS may be better able to use all those CPU cores we will be seeing soon as four and eight core systems become common. ZFS can make a system seem more responsive even when it is under heavy I/O load.

6) data is compressed as it is writen to disk -- so less space is used but also because disks are slow, less data are written so it takes less time (assuming you have CPU power to "burn" which a four core system will have)

shawnce
Dec 18, 2006, 04:28 PM
This is one of the reasons I love being an OS X user. Can't wait to see what Apple eventually does with it.

I would love Apple to leverage the capabilities of ZFS (in particular zpools and related capabilities). Knowing them they will make it simple to manage, if not transparent in many cases. However I think it will be a longer road then 10.5 to get to that point.

MBP123
Dec 18, 2006, 04:32 PM
Let's run through a simple example.

You have 1 hard drive right? You are running out of space. You go buy another drive. You want to add it to the system but now you have to decide whether you want that drive to just handle storage of files, and if so which files? Well, how about this instead? You can add the drive as a pool. It then "magically" appears as if your original drive is now x gigabytes larger than it is since it is using two drives as a pool. Convenient no?

Or how about you want to backup that drive instead? Ok, add a drive to the system. Add it to the pool, tell it to mirror the drive instead. It now copies the data from one drive to the other and any changes mirror the other. If a corrupt file is on drive 1 (your original working copy) it checks the other to see if the backup is non-corrupt. If so, it opens that file, and copies the good data to the original drive as good data. It does this with Checksums of the files.

Another for the geekiness factor is RAIDZ. One drive or two can be a parity drive. Ever use PAR files? Yup very similar. You have files or a whole drive disappear? You can pretty much restore it from the parity drive if enough of the data still exists, etc.

Or how about you like having "versions" of your filesystem. You're about to update OS X to 10.5.2 and you're afraid it might break your system. So you create a ZFS snapshot. It now olds this "snapshot" of your filesystem and the files in it. You install the new update, it does indeed bork your computer. So you tell OS X and ZFS to use a previous snapshot, boom. You're back to 10.5.1 and it works.

Also the idea of using ZFS with time machine, is really really cool. Also, on the fly compression of your files. With no real performance hit. Another neat one is built in encryption. So you'll be able to have file vault at the filesystem level rather than the OS X application/OS level.

ok but say i buy an external hard drive, and "pool" it with the internal hard drive. the external hard drive craps out and dies (like most hard drives do). is all my data then gone? or is some data still there from the HD that didnt fail

thejadedmonkey
Dec 18, 2006, 04:32 PM
So in very simplified terms, snapshots are basically like wikipedia? It just keeps tracks of changes instead of creating a whole new copy of it, correct?

If I understand it correctly, Yes.

thejadedmonkey
Dec 18, 2006, 04:33 PM
What happens if you add a drive into a pool, and at a later date wish to remove it from the system? Can you just tell the computer to move all of the data off of the drive and then pull it out (after turning off the computer, of course, or not if it's SATA) with no harm or lost files?

Doctor Q
Dec 18, 2006, 04:35 PM
It's a natural fact: Over time, newer and better filesystems are developed. What's good to see is that Apple is willing to accept these changes and offer us new filesystems.

We don't want a "one size fits all" filesystem, and I mean that in both senses of the phrase: Users will benefit by being free to configure their disk space in a number of ways, and users need filesystems that are practial for variable-size files, from huge databases to the thousands of tiny files that come with some applications.

Goldfinger
Dec 18, 2006, 04:36 PM
ZFS is cool but I'm afraid that the concept of pools is a bit complicated for the average consumer. How to manage backups ? What happens when a drive fails ? Time Machine has to be intelligent enough that it doesn't place data and backup on the same physical drive. Otherwise you could lose your original data AND the backup.

But, if it's thought out well then I'm all for it.

wmmk
Dec 18, 2006, 04:36 PM
am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? If so, then wow. Simply wow.

shawnce
Dec 18, 2006, 04:37 PM
Did you read this?
http://opensolaris.org/os/community/zfs/whatis/

1) All operations are "copy on write".
data are never over written you can always go back to last month's version . The system saves space by only writing changes. Actually you are mixing up "copy on write" with snapshotting capabilities (however the former does help with the later). The COW aspect is talking about how file blocks are managed when they are changed including "file" blocks related to the file system itself. As a result the data on disk is always coherent which avoids the need for journaling and fsck.

longofest
Dec 18, 2006, 04:39 PM
Please note that as of this time, developer builds of Time Machine are using HFS+ formatted disks, and ZFS is not used. While ZFS would certainly much-enhance Time Machine, this capability has yet to appear in the application.

am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? If so, then wow. Simply wow.

Heh... While the compression algorithm may be good, you are talking about a factor of 10x + there, which means that you are going to see a noticable decrease in performance. Perhaps useful for archiving data, but not much else...

shawnce
Dec 18, 2006, 04:40 PM
ok but say i buy an external hard drive, and "pool" it with the internal hard drive. the external hard drive craps out and dies (like most hard drives do). is all my data then gone? or is some data still there from the HD that didnt fail It depends. If you are mirroring data between the two drives then no. If you are using RAIDZ (likely not applicable in this simple example of yours) then likely no, your data would still exist and be accessible (has a storage and IO cost overhead to do this). If you are using simple concatenation (not sure of ZFS terminology) then a subset of your files (or file data) would disappear.

mkrishnan
Dec 18, 2006, 04:41 PM
am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? If so, then wow. Simply wow.

Erm... no, I don't think so. Not unless the situation were extraordinarily unuasual, like you had millions upon millions of teeny tiny files and basically no files of any significant size. Or you had inordinately compressible files...i.e. ones with basically junk in them.

killmoms
Dec 18, 2006, 04:41 PM
am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? If so, then wow. Simply wow.

Yeah, you're getting it very wrong. Some data can barely be compressed at all (since it's already compressed in other ways). What pools mean is that you can add storage to your pool almost limitlessly. Your drive still only stores 100GB on it. It's software, not sorcery. ;)

shawnce
Dec 18, 2006, 04:41 PM
What happens if you add a drive into a pool, and at a later date wish to remove it from the system? Can you just tell the computer to move all of the data off of the drive and then pull it out (after turning off the computer, of course, or not if it's SATA) with no harm or lost files? Yes ZFS supports that type of operation. You can migrate data off of a storage device that you want to retire.

ChrisA
Dec 18, 2006, 04:42 PM
This would be very exciting feature to have. The ability to have one single Macintosh HD and just keep adding drives (internal or external) to increase the space.

Sweeeet. :D

Yes it would be great. Every other UNIX system works like this already. So can the mac if you want to take the time to set it up. Put you may not want to use external drives (that might get unplugged) as art of your storage pool. Best to use drives that are screwed down for that.

shawnce
Dec 18, 2006, 04:43 PM
am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? No, the supported compression is not nearly that good (and already compress files for example are sufficiently random that it couldn't compress then further).

Westside guy
Dec 18, 2006, 04:44 PM
ZFS is cool but I'm afraid that the concept of pools is a bit complicated for the average consumer. How to manage backups ? What happens when a drive fails ? Time Machine has to be intelligent enough that it doesn't place data and backup on the same physical drive. Otherwise you could lose your original data AND the backup.

But, if it's thought out well then I'm all for it.

Don't forget that ZFS was developed by Sun primarily as a next-generation filesystem for Solaris, their server OS. Some of its capabilities will not be useful to the average consumer; but that doesn't make it a negative. It's not like you HAVE to use multiple volumes.

lorductape
Dec 18, 2006, 04:45 PM
I have yet to meet someone with a zetta of hd space

Westside guy
Dec 18, 2006, 04:46 PM
Yes it would be great. Every other UNIX system works like this already. So can the mac if you want to take the time to set it up.

So can Windows - RAID has very little to do with the OS.

ChrisA
Dec 18, 2006, 04:52 PM
Actually you are mixing up "copy on write" with snapshotting capabilities (however the former does help with the later). The COW aspect is talking about how file blocks are managed when they are changed including "file" blocks related to the file system itself. As a result the data on disk is always coherent which avoids the need for journaling and fsck.

I'd say COW is the enabling technology that makes snapshotting passable. So yeas while it is the snapshots and time machine that the user see COW is what makes it passable.

What's really happened here is that we have a huge surplus of CPU power and the problem is only getting "worse". ZFS puts theis excess of CPU to work the problem of staorage. Lots of credit to Sun for dong this

shawnce
Dec 18, 2006, 04:53 PM
So can Windows - RAID has very little to do with the OS.

Also to be clear ZFS does the "integration" of storage devices at a different level then your traditional partition based volume manager. In ZFS it is managed at the file block level while traditional methods do it at the device block level.

shawnce
Dec 18, 2006, 04:54 PM
I'd say COW is the enabling technology that makes snapshotting passable. So yeas while it is the snapshots and time machine that the user see COW is what makes it passable. No COW is a more fundamental feature and snapshotting can be (often is) implemented without such a behavior.

SiliconAddict
Dec 18, 2006, 05:08 PM
So can Windows - RAID has very little to do with the OS.

I was going to say the same thing. RAID is nothing new. I've had RAID 3 on my home server and RAID 0 on my desktop for years.


I have yet to meet someone with a zetta of hd space

Umm I wonder if there is that much storage on Earth. :confused:

Goldfinger
Dec 18, 2006, 05:08 PM
Don't forget that ZFS was developed by Sun primarily as a next-generation filesystem for Solaris, their server OS. Some of its capabilities will not be useful to the average consumer; but that doesn't make it a negative. It's not like you HAVE to use multiple volumes.

Oh I know. I've known what ZFS is for a long time now. But all I'm saying is that the concept of pools could be confusing for the average consumer.
I'm not saying in any way that ZFS and it's capabilities suck and that you HAVE to use it. ZFS is awesome.

The main problem for consumers IMHO is that when they start mixing their internal drives with their external USB and FireWire drives. One day they unplug their drive and boom there goes the filesystem.

All I'm saying is that ZFS needs some clear explenation. The OS should warn you if you use it and explain how it works since the concept of pools is quite different from the current filesystem concepts.

But I'm all for ZFS, let me make that clear.:)

tkn
Dec 18, 2006, 05:09 PM
I believe that ZFS can even use network storage as part of the pool. So you could perhaps sign up for Amazon's S3 (if they had the right interface for it) and never run out of storage as your pool would just grow larger and larger as you used more space.

Basically ZFS has almost every single advanced filesystem thing you would want. I am not sure how much it differs from Reiser4 (which is still experimental), but it is near parity at least if not better.

Bonte
Dec 18, 2006, 05:10 PM
Doesn't Time Machine ask to configure a hard drive once it's attached? Would likely be ZFS, no?

Almost every feature of ZFS i read here is practically a copy of the TimeMachine presentation, i hope we get it soon. I'm having some bad memories when Spotlight was named Cherlock and indexed the whole freaking system at night. :(

killmoms
Dec 18, 2006, 05:13 PM
Almost every feature of ZFS i read here is practically a copy of the TimeMachine presentation, i hope we get it soon.

More accurately, every feature of Time Machine is (at least in the WWDC preview on HFS+) a hacky, not-as-cool/efficient attempt at implementing a small subset of end-user-visible functions that might SEEM like ZFS, but really aren't. ;)

Bonte
Dec 18, 2006, 05:16 PM
Oh I know. I've known what ZFS is for a long time now. But all I'm saying is that the concept of pools could be confusing for the average consumer.
I'm not saying in any way that ZFS and it's capabilities suck and that you HAVE to use it. ZFS is awesome.

The main problem for consumers IMHO is that when they start mixing their internal drives with their external USB and FireWire drives. One day they unplug their drive and boom there goes the filesystem.

I'm sure Apple will make it easy enough and warn the users or even exclude some features like adding a USB drive to a internal HD pool. Less is more and advanced users will find there way in, no question about that. :rolleyes:

MacRumorUser
Dec 18, 2006, 05:19 PM
this sounds uber geekingly fantastic.

Vista my arse :D

081440
Dec 18, 2006, 05:24 PM
Oh I know. I've known what ZFS is for a long time now. But all I'm saying is that the concept of pools could be confusing for the average consumer.
I'm not saying in any way that ZFS and it's capabilities suck and that you HAVE to use it. ZFS is awesome.

The main problem for consumers IMHO is that when they start mixing their internal drives with their external USB and FireWire drives. One day they unplug their drive and boom there goes the filesystem.

All I'm saying is that ZFS needs some clear explenation. The OS should warn you if you use it and explain how it works since the concept of pools is quite different from the current filesystem concepts.

But I'm all for ZFS, let me make that clear.:)


If the External storage options were to stay the same as no, I would almost call the opposite. As the "average" consumer would only have to add more drives to their system and not have to worry about where it is and how it works since the OS would just add it like RAM. This would me a great thing for the consumer.

Bonte
Dec 18, 2006, 05:24 PM
More accurately, every feature of Time Machine is (at least in the WWDC preview on HFS+) a hacky, not-as-cool/efficient attempt at implementing a small subset of end-user-visible functions that might SEEM like ZFS, but really aren't. ;)

Thats what i'm afraid of, it looks like were getting a HFS+ hack first and some years later the ZFS implementation. First Cherlock ... 5 years later Spotlight, first TimeMachine ... 5 years later ZFS. Its hard to believe Apple can implement this filesystem in osX and Timemachine in time for Leopard.

Marx55
Dec 18, 2006, 05:35 PM
AWESOME. ZFS repairs disk damage even before you ever notice it!!!

http://opensolaris.org/os/community/zfs/demos/selfheal/

iW00t
Dec 18, 2006, 05:44 PM
This sounds very interesting, always good to be able to use the latest new tech.

From the Wikipedia article: "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

I wonder how long until we hit that limit :P

Why do we need to boil the oceans?

lorductape
Dec 18, 2006, 05:46 PM
Umm I wonder if there is that much storage on Earth. :confused:

ha ha well in theory there is...

LostPacket
Dec 18, 2006, 05:47 PM
I hope Lepeord uses ZFS by default. Maybe this was one of those secret features because they weren't sure if they could do it in the alotted time. Its appearance in the preview could be a good sign.

Stridder44
Dec 18, 2006, 05:57 PM
Vista has officially been dwarfed.

SimonTheSoundMa
Dec 18, 2006, 06:02 PM
You have 1 hard drive right? You are running out of space. You go buy another drive. You want to add it to the system but now you have to decide whether you want that drive to just handle storage of files, and if so which files? Well, how about this instead? You can add the drive as a pool. It then "magically" appears as if your original drive is now x gigabytes larger than it is since it is using two drives as a pool. Convenient no?
You can do that now. It uses LVM, you can sync partitions without reboot, just use partprobe, pvcreate to make the partitions to volumes, suck them all together with vgcreate and use lvextend to make them bigger, hfsonline to make it. You can grow a drive to whatever you want.

You can also use pvmove, vgreduce, pgremove, if a drive is about to fail and you have another in sync as a mirror and bring it online.

wmmk
Dec 18, 2006, 06:04 PM
Please note that as of this time, developer builds of Time Machine are using HFS+ formatted disks, and ZFS is not used. While ZFS would certainly much-enhance Time Machine, this capability has yet to appear in the application.



Heh... While the compression algorithm may be good, you are talking about a factor of 10x + there, which means that you are going to see a noticable decrease in performance. Perhaps useful for archiving data, but not much else...

oh, oops! with people saying things like, "there's not enough data in the world to fill a ZFS pool," I assumed that data could be infinitely compressed with no hit to performance. so anyway, how much more are files in ZFS able to be compressed than in HFS+ at the same performance level?

BrianMojo
Dec 18, 2006, 06:12 PM
Vista has officially been dwarfed.

Agreed.

But moreover, if ZFS is the default for Leopard (that's a pretty big if, but not out of the question -- God bless "top secret" features) then I would imagine that it would automatically create a pool out of any internal drives but that external drives, by default, would not be considered part of that pool.

This is the kind of thing that Apple excels at: because the internal hard drives are inside the box, we should think of them as one -- together they are the storage space of the machine. But anything outside of the box, logically, seems as though it should be a separate part unless you specifically tell the computer otherwise. Thus, there could be a checkbox under "Get Info" or something similar to add that drive to the pool. I can't imagine that Apple would make it much more complicated than that.

Erasmus
Dec 18, 2006, 06:14 PM
So... 128 bit, eh?
Let's see here. If we had a 2^128 bit hard drive, we would be set for storage for a long time. Let's see here:

A Terabit is 2^40 bits, a Terabyte is 2^43 bytes. Sooo...
If we had one bit for every terabyte of storage, and then one bit for every terabyte of storage in that hard drive, we would still have half a terabyte left. That's twice what my computer has now, and about eight times what I'm actually using.

So, that ends up being roughly 6 * 10 ^ 26 times more data than is on my computer at the moment. Or, 600,000,000,000,000,000,000,000,000 times the data on my disk, or 340,000,000,000,000,000,000,000,000,000,000,000,000 bits of data.

In terms of how much data is in the world, it depends. Do you mean in hard drives around the world, or how much disk space it would take to back up the Earth?

Interesting question... I'll get back to you on that, with some funky made up physics and maths.

Just thought I should put it in perspective.

tcmcam
Dec 18, 2006, 06:15 PM
First of all, yes ZFS is very cool.

BUT, it isn't going to be great for a portable MacBook or an iMac. It's going to be great for an XServe or a MacPro with a lot of attached storage (read: many drives).

The power in ZFS comes in when you start to add drives in pairs. Each mirrored pair is spanned with the other mirrored pairs. This provides more fault tolerance than RAID 5 solutions and is obviously more flexible in terms of growth compaired with today's RAID-1 solutions.

Until they start putting multiple hard drives in a MacBookPro, ZFS is primarily a Workstation or Server feature.

And yes, that is still a fantastic addtion to OS X.

joshysquashy
Dec 18, 2006, 06:18 PM
am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? If so, then wow. Simply wow.

i think what you are getting confused with is the limits of the file system:

Fat32 can not store a file larger than 4GB of data. The max size of a disk can be up to 8 Terabytes.

NTFS is 16 Terabytes per file and 256 total size.

HFS+ is 16 Exibytes (lots more)

ZFS is practicaly infinite, that means you can format any sized drive, and store files as large as they will ever, ever get in the history of everything. there never will need to be an increase because the energy required to fill this file system up would be the same as needed to boil the oceans.

hope that answers your question, someone correct me if im wrong.

shawnce
Dec 18, 2006, 06:20 PM
oh, oops! with people saying things like, "there's not enough data in the world to fill a ZFS pool," I assumed that data could be infinitely compressed with no hit to performance. so anyway, how much more are files in ZFS able to be compressed than in HFS+ at the same performance level?

File compressibility is not file system specific. The only advantage ZFS has is that 1) is has support for compression built-in, and 2) it has better file block allocation allowing small files to waste less space (compared to a file system that uses fixed allocation blocks).

m-dogg
Dec 18, 2006, 06:22 PM
ZFS is a file system (a very capable modern one) so the later part of your question doesn't make sense. If you mean "will Apple make ZFS the default file system for Mac OS X?" then I would say "not likely any time soon (but I would be happy to be wrong)".

Yup - That's what I meant...if it will be the default for all users or if it will just be an option for those that understand it and choose to utilize it.

balamw
Dec 18, 2006, 06:31 PM
This is the kind of thing that Apple excels at: because the internal hard drives are inside the box, we should think of them as one -- together they are the storage space of the machine.
You mean I shouldn't think of the first internal hard drive in the system as the C: drive? The next one as D:, etc... [Head explodes].

Drive letters in Windows annoy me to no end.

B

Erasmus
Dec 18, 2006, 06:31 PM
By the way, you couldn't store the Earth in a 2^128 bit hard drive. Maybe you could store a person, but only with some serious compression algorithms.

So, in conclusion, the Earth has more data in it than a ZFS HD can deal with.

Just thought you should know.

I'll save you from the wierd logic and very large numbers. :D

maxterpiece
Dec 18, 2006, 06:39 PM
This whole pooling feature seems to be the thing that has people most excited about ZFS. What happens if I have two drives inside my computer - one faster one for large files and one slower one for everything else. Now i want to know and assign where each of my files go and this feature is useless to me.

twoodcc
Dec 18, 2006, 07:04 PM
more good news about Leopard! i just hope it's out soon!

GregA
Dec 18, 2006, 07:13 PM
Let me start by saying I know VERY LITTLE about ZFS :)

This whole pooling feature seems to be the thing that has people most excited about ZFS. What happens if I have two drives inside my computer - one faster one for large files and one slower one for everything else. Now i want to know and assign where each of my files go and this feature is useless to me.Very good question. I certainly want files I access all the time on the faster partition. If it does that automatically that's great - I'm not sure whether manually putting it on a slower drive would make much difference????

Also, does this (or can this) relate in any way to the new hard disks which have built in flash memory - to speed up the most common read requests? (eg at OS startup).

I believe that ZFS can even use network storage as part of the pool. So you could perhaps sign up for Amazon's S3 (if they had the right interface for it) and never run out of storage as your pool would just grow larger and larger as you used more space.That'd be very interesting (though I'm not in a position to use it anymore).

I have no idea what extent this goes to. I'd love to be able to "give away" all the free space on each Mac in a network, into a central network pool. And from that have a "virtual OSX file-server" drive for every machine. Build in redundancy would be required, of course, for when a machine dies or is turned off.

Or (... a little more reliably...) if I have a central network file-server, instead of using a RAID for redundancy, how about using the excess disk space on each workstation in the network?

Does ZFS allow that kind of concept?

shawnce
Dec 18, 2006, 07:13 PM
By the way, you couldn't store the Earth in a 2^128 bit hard drive. Maybe you could store a person, but only with some serious compression algorithms. Well based on one proof of the ability of matter to store data... a fully populated ZFS file system would require around 136 billion kilograms of matter... which is about 43,970,588,235,294 times smaller then the mass of the earth but of course far larger then the mass of a person. :D

This assumes perfectly efficient storage of information by that block of matter... which isn't feasibly possible... so you likely need several orders of magnitude more matter to store a fully populated ZFS file system.

tseitz89
Dec 18, 2006, 07:14 PM
For us poor plebeians, what the hell does the adoption of ZFS bring anyway? Is it just another geeky filesystem, or something that might really make a difference for ordinary users?

At least HFS enabled much bigger storage and more sensible partitioning back then...could any of you clarify this issue for the non-file server admins out there? Thanks.


No ZFS can hold more data "if 1,000 files were created every second, it would take about 9,000 years to reach the limit" (source wikipedia)

dllavaneras
Dec 18, 2006, 07:15 PM
So if I have a 40 Gb HD, add a new 250 Gb HD to the pool that would bring my total HD space to 290 Gb. Now, if I have 20 free Gb in my 40 Gb HD, and 30 free Gb in my 250 Gb drive, can I copy a 40 Gb file without there being a problem? What if a file (a really big one) spans multiple hard drives and one drive is lost? is there a way to fix that? Obviously someone with the need for a file that big is pretty careful about backing up, but that's another story...

tseitz89
Dec 18, 2006, 07:17 PM
PEople do know that ZFS can't be used as root file system which means no boot capability for Mac os X right? but Apple does speed up things when they want it for them so hopefully they can get the ZFS Boot project team working an that

shawnce
Dec 18, 2006, 07:28 PM
Some interesting blog posts by ZFS folks...

128-bit storage: are you high? (http://blogs.sun.com/bonwick/date/20040925)
ZFS: The Last Word in Filesystems (http://blogs.sun.com/bonwick/date/20051031)
RAID-Z (http://blogs.sun.com/bonwick/date/20051118)
ZFS End-to-End Data Integrity (http://blogs.sun.com/bonwick/date/20051209)
SEEK_HOLE and SEEK_DATA for sparse files (http://blogs.sun.com/bonwick/date/20051212)
Smokin' Mirrors (http://blogs.sun.com/bonwick/date/20060502)
You say zeta, I say zetta (http://blogs.sun.com/bonwick/date/20060504)
ZFS Block Allocation (http://blogs.sun.com/bonwick/date/20061104)

ChrisA
Dec 18, 2006, 07:39 PM
So if I have a 40 Gb HD, add a new 250 Gb HD to the pool that would bring my total HD space to 290 Gb. Now, if I have 20 free Gb in my 40 Gb HD, and 30 free Gb in my 250 Gb drive, can I copy a 40 Gb file without there being a problem? What if a file (a really big one) spans multiple hard drives and one drive is lost? is there a way to fix that? Obviously someone with the need for a file that big is pretty careful about backing up, but that's another story...

No that won't work. for two reasons

The boot device can't be ZFS
The operating system can't be stored on ZFS

Currently, ZFS is for data

But on to your question -- First off you don't need to have a large file to span disks. even a smaller file will span disks, parts of it will be distributed over the disks in the pool. So really ALL files are distributed over all the drives. This makes it fast as we can do reads in parallel.

What happens when a drive fails? Typically you get a notification and the failed drive is taken off line by the system. Reads can continue because data are stored redundantly. If you are lucky you have a hot spare installed and the system will spin up that drive and reconstruct the lost data on that drive in the background. If you don't have a hot spare, go buy one. You can configure various levels of redundancy so that you can survive one, two or more drive failures with out going down. In a large data centers they might keep a half dozen drives plugged in as spares, so they could loose tha many drives before needing to call the repair guy -- This is not new. We have a big array of disks downstairs. It is a 6 foot tall rack with about 80 drives. The technology is about 20 years old now.

What's new is that ZFS generalizes the concept so well and integrates it into the file system layer.

Going back to spanning drives -- if the file didn't span two or more drives then it would be lost if the drive it was on failed. right? So they are ALL spanned and written redundantly. even very small files

scott523
Dec 18, 2006, 07:51 PM
Does ZFS have the auto defragment disk feature like the HFS+ has?

ChrisA
Dec 18, 2006, 07:55 PM
The main problem for consumers IMHO is that when they start mixing their internal drives with their external USB and FireWire drives. One day they unplug their drive and boom there goes the filesystem.

This is where Apple will need to add value over what Sun has given us. Apple will need to make a nice interface. Currently Apple allows you to build a RAID 5 system with a set of external FW drives. This is not such a great idea either.

I'dlke to see Apple ofer some better desktop storage options. They will need it if they want to sell those iTV things. Where to put 500 hours of video?

Peace
Dec 18, 2006, 08:55 PM
Does ZFS have the auto defragment disk feature like the HFS+ has?

No need for defrag using the Zettabyte File System (http://www.sun.com/emrkt/campaign_docs/expertexchange/knowledge/solaris_zfs_perf.html#32)

odaiwai
Dec 18, 2006, 08:58 PM
I am not sure how much it differs from Reiser4 (which is still experimental), but it is near parity at least if not better.

Well, one major difference is that the ZFS authors are unlikely to be serving 25 to life anytime soon.
(http://en.wikipedia.org/wiki/Hans_Reiser)

SeaFox
Dec 18, 2006, 09:10 PM
What happens if you add a drive into a pool, and at a later date wish to remove it from the system? Can you just tell the computer to move all of the data off of the drive and then pull it out (after turning off the computer, of course, or not if it's SATA) with no harm or lost files?

It would, but keep in mind for that to work you would have to have enough space on the remaining drives to accommodate all the data you are moving. So if I had two HD's and wanted to swap one out and add another, if both of those drives are full (which is probably why you are wanting to change HD's) you are going to have a problem. The solution would be to crack open the case, install your new drive as a third drive, then after adding it tell the file system you want to remove the one older drive so it can swap the data around.

So you would have to have a machine with at least one more internal bay than the number of drives you wish have in the machine. Except, I suppose you could use an external drive as a temporary holding place. So then you only have to crack the case open once to add the new drive and remove the old one. Hey, I wonder if it would be possible to use the new drive as the temporary drive. So you put the new drive in an external case, hook up, add to storage pool and migrate data off the old drive. Then shut down the machine and move the new drive from the external case to the internal bay after removing the old drive, and everything still matches up in the file system!

Rocketman
Dec 18, 2006, 09:14 PM
ZFS is the basis of time machine.

Rocketman

N9YTY
Dec 18, 2006, 09:19 PM
First of all, yes ZFS is very cool. BUT, it isn't going to be great for a portable MacBook or an iMac. It's going to be great for an XServe or a MacPro with a lot of attached storage (read: many drives).
With the smaller capacities available for laptop drives, ZFS and it's ability to compress on the fly AND not to waste space with minimum cluster sizes is going to be HUGE for portable users. Not for MacBooks? Indeed, they will benefit greatly!

shawnce
Dec 18, 2006, 09:33 PM
ZFS is the basis of time machine. O'rly? (no not really)

mozmac
Dec 18, 2006, 09:40 PM
Leopard is looking to be a much bigger update than I anticipated. It's going to have a lot more things than Tiger had. Tiger seemed more like a "features" update, whereas Leopard is overhauling a whole bunch of stuff, while still giving us some cool features. Every major update has done that, except for Tiger to an extent. So, I guess you could compare Tiger that middle movie in the trilogy that doesn't do as much but is necessary to get you from one to three. **cough, cough** Pirates of the Caribbean **cough, cough**

asphalt-proof
Dec 18, 2006, 09:44 PM
Mmm. 640K ought to be enough for anybody, right? :D

Ohhh I so wanted to use that!!! As soon as I read that quote I was going to write that exact phrase. :p

The rest of that wiki is very interesting as well.

asphalt-proof
Dec 18, 2006, 09:50 PM
Why do we need to boil the oceans?

on demand boiled shrimp and lobster. Need I say more? Not to mention getting rid of those pesky whales and their incessant singing.

SeaFox
Dec 18, 2006, 09:52 PM
ZFS is the basis of time machine.

Rocketman
Yes, that certainly explains why Time Machine exists in the current developer's builds that are running on HFS+ :rolleyes:

ZFS would make a great basis for an implementation of Time Machine (read as: where we are going) but the program itself will not require a ZFS file system, it will be compatible with HFS+ for current users' sake.

I'm hoping for an HFS/HFS+ type transistion: ZFS will be in OS 10.5, you can choose to reformat your hard drive and start using ZFS for your boot partition, or keep using HFS+ for now. If you do change to ZFS you'll get better Time Machine performance, the pooled storage, ect.

Then at some point in the future (10.6, 10.7?) they will drop the HFS+ and you'll have to use a ZFS file system to upgrade.

Everyone keeps talking about how ZFS booting is not ready yet. Silly users, ZFS is Open-Source, is it not possible Apple has been working on booting ability on their own and have solved the issue for 10.5?

One thing that will be a problem in the beginning is there will be no DiskWarrior, ect to repair drives with. But given the features of ZFS, I wonder what use there would be for diagnostic programs to begin with. :D

Analog Kid
Dec 18, 2006, 10:00 PM
Ars Technica has suggested that this would be required for Time Machine to function, but isn't Time Machine already included in the developer's previews?
I think this has been implied earlier on the thread, but never stated outright: Ars suggested that Time Machine was a whole lot like ZFS, but probably wasn't ZFS. Right now, Time Machine likely uses a series of hard links in the BSD layer or something similar to keep the snapshots. Not a bad approach (it's how we do backups at work) but not very efficient for big files.
From the Wikipedia article: "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

Great, data storage as a primary cause of global warming... "Americans are 5% of the worlds population but they generate 35% of the worlds data" is just around the corner now...
1) All operations are "copy on write".
data are never over written you can always go back to last month's version . The system saves space by only writing changes.
Well, you can get last month's version assuming nothing else took that block for new storage...
Finally Mac OS can get rid of those disk drive icons. There can just be "storage" Much like you don't care much about how many RAM chips you have, you only care about the total amount of RAM. Disk can be like that too. Just open the box and slide in one more disk and the rest is "magic". Just like with RAM.
Nah. People have a lot of reasons for wanting separate data stores. I don't wanna plug in my digital camera and have it become "part of the pool". I like my firewire drives separate so I can power them down and shut them up. There will always be a place for human storage management, it's just that the pool approach is good for large data stores.
4) there is not enough data on Earth to fil a ZFS file system. Or at least if you wrote to a disk 24x7 you would not live long enough to fill up ZFS. (although you would fill up quite a few physical drives)
Statements like this are just asking to be proven silly... One ZFS file system can store the DNA sequences of every cell in every creature on the planet? The state of every electron in ever atom on the planet? The energy, frequency, and vector of every photon passing by the planet? 3e38 is big, but not infinite...
You can do that now. It uses LVM, you can sync partitions without reboot, just use partprobe, pvcreate to make the partitions to volumes, suck them all together with vgcreate and use lvextend to make them bigger, hfsonline to make it. You can grow a drive to whatever you want.

You can also use pvmove, vgreduce, pgremove, if a drive is about to fail and you have another in sync as a mirror and bring it online.
Has this been wrapped in a tool anywhere? Seems like there's plenty of people who could use this now, but wouldn't want to touch the command line...
But moreover, if ZFS is the default for Leopard (that's a pretty big if, but not out of the question -- God bless "top secret" features) then I would imagine that it would automatically create a pool out of any internal drives but that external drives, by default, would not be considered part of that pool.

This is the kind of thing that Apple excels at: because the internal hard drives are inside the box, we should think of them as one -- together they are the storage space of the machine. But anything outside of the box, logically, seems as though it should be a separate part unless you specifically tell the computer otherwise. Thus, there could be a checkbox under "Get Info" or something similar to add that drive to the pool. I can't imagine that Apple would make it much more complicated than that. This sounds like the right model to me. My external drives are individual units-- or at least one unit per enclosure. I don't want that assimilated into the Borg. Anything inside the case is considered "primary storage" and should flow together and cooperate.

Digitalclips
Dec 18, 2006, 10:09 PM
I wonder where Microsoft are with their R&D (i.e. copy and / or rip off Apple) are with this?

Analog Kid
Dec 18, 2006, 10:29 PM
Leopard is looking to be a much bigger update than I anticipated. It's going to have a lot more things than Tiger had. Tiger seemed more like a "features" update, whereas Leopard is overhauling a whole bunch of stuff, while still giving us some cool features. Every major update has done that, except for Tiger to an extent. So, I guess you could compare Tiger that middle movie in the trilogy that doesn't do as much but is necessary to get you from one to three. **cough, cough** Pirates of the Caribbean **cough, cough**
Tiger also overhauled a whole bunch of stuff, little of it was indicated with little GUI lights though so it's only noticed as improvements in performance and stability.
I wonder where Microsoft are with their R&D (i.e. copy and / or rip off Apple) are with this?
Uh... You did notice that ZFS is a Sun designed file system, right?

iGuy
Dec 18, 2006, 10:34 PM
am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? If so, then wow. Simply wow.

However, if you add another nine, 100GB HDs, you could create a 1TB logical volume.

You can do this already in OSX 10.4 Tiger using Disk Utility. But the difference with ZFS is that this 'pooling' of hard disks to create larger logical volumes is handled at the filesystem level, not the OS level.

I think I got that right?

There are reasons that you may want to do this and reasons that you may not.

So far in this thread I haven't seen much reference to the size and type of files being stored. For example you want to handle large binary image/video files differently than you handle large numbers of small files.

Most of the discussion so far assumes that most average users store a large number of smaller files. And this is a reasonable assumption.

Just be careful not to jump to conclusions or make broad assumptions. When Apple does release an OS that supports ZFS be sure to take the time to think about what storage configuration is going to work best for you most of the time. Nothing is perfect or absolute.

~iGuy

AppliedVisual
Dec 18, 2006, 10:41 PM
Tiger also overhauled a whole bunch of stuff, little of it was indicated with little GUI lights though so it's only noticed as improvements in performance and stability.

Uh... You did notice that ZFS is a Sun designed file system, right?

And ZFS is available for Windows too... At a price.

vertgo
Dec 18, 2006, 10:59 PM
HFS+ seems to be a little aging, and now there's finally some path for Apple to go to a next generation (or two gens later) file system. I hope they migrate native filesystem to this, eventually. Then we can have permanently versioned, self healing files. I would want ZRaid standard on my mac pros

ChrisA
Dec 18, 2006, 11:43 PM
I wonder where Microsoft are with their R&D (i.e. copy and / or rip off Apple) are with this?


ZFS was written by Sun and is part of Solaris. Sun made Solaris Open source and give it away to everyone. There is no need to "zip off" what is available for free. Apple picked up two important Open Source parts of Solaris: Dtrace and ZFS. These are some of the best technical ideas to come out along in the UNIX world in a long time. We should thank Sun for this. Some day maybe Apple will contribute something back

I doubt ZFS could be made to work under Windows. Put the source is available so anyone could give it a try

Westside guy
Dec 18, 2006, 11:47 PM
Oh I know. I've known what ZFS is for a long time now. But all I'm saying is that the concept of pools could be confusing for the average consumer.
I'm not saying in any way that ZFS and it's capabilities suck and that you HAVE to use it. ZFS is awesome.

The main problem for consumers IMHO is that when they start mixing their internal drives with their external USB and FireWire drives. One day they unplug their drive and boom there goes the filesystem.

All I'm saying is that ZFS needs some clear explenation. The OS should warn you if you use it and explain how it works since the concept of pools is quite different from the current filesystem concepts.

But I'm all for ZFS, let me make that clear.:)

Hehe, sorry. I misunderstood where you were coming from, so to speak. And I've got to admit I've only read about ZFS, not used it (sounds like it might be a while before it's available on Linux, and even then it'll be clunky if they're going through FUSE).

I would guess that, given Apple's usual way of doing things, those potentially problematic parts will be made unavailable to the average user. Of course some of us will whine and complain about the crippled functionality in that case. :D

x704
Dec 19, 2006, 12:05 AM
I'll tell you what Apple, when you can make my computer boot from ZFS, I will buy the upgrade (at student discount of course).
I would imagine by the time they intergrate ZFS into their OS's, there will be lots of other reasons to upgrade. Like a system wide vector/nerbs windowing system, much like CAD. Then <control><scroolwheel> will look ausom!

Westside guy
Dec 19, 2006, 12:11 AM
Everyone keeps talking about how ZFS booting is not ready yet. Silly users, ZFS is Open-Source, is it not possible Apple has been working on booting ability on their own and have solved the issue for 10.5?

It's not bootable for Solaris yet either, so it might be a bigger problem than it sounds like.

However to be annoyingly repetitive - since OS X boots off a separate partition, there's no reason your main "disk" couldn't be ZFS even if ZFS isn't bootable. Just make the boot partition HFS+ or UFS. Different partitions on the same disk can have different file systems (anyone who's run Bootcamp has done this).

ChrisA
Dec 19, 2006, 12:16 AM
Statements like this are just asking to be proven silly... One ZFS file system can store the DNA sequences of every cell in every creature on the planet? The state of every electron in ever atom on the planet? The energy, frequency, and vector of every photon passing by the planet? 3e38 is big, but not infinite...


All of the world's printed data (today) would not come close to filling even a 2^64 bytes. 2^128 is larger than 2^64 by a factor of 2^64. I think it is safe to say all of the world's recored data could fit. This is different than storing the state of the Earth. I doubt the state of the Earth could be stored on a device only the size of the Earth, it would need to be larger.

But who cares. The point is that this thing will not need to be redesigned on 20 years. It can last the rest of our lives at least. After that I don't care.

Westside guy
Dec 19, 2006, 12:45 AM
Check out ZFS Boot (http://opensolaris.org/os/project/zfsboot).

Hey that's cool - doubly so because it plugs into Grub. I am probably dreaming, but if Intel Macs could be booted from Grub it would be trivial to make them multi-boot Linux, Windows, etc.

iGuy
Dec 19, 2006, 12:46 AM
All of the world's printed data (today) would not come close to filling even a 2^64 bytes. 2^128 is larger than 2^64 by a factor of 2^64. I think it is safe to say all of the world's recored data could fit. This is different than storing the state of the Earth. I doubt the state of the Earth could be stored on a device only the size of the Earth, it would need to be larger.


You've overlooked ZFS's compression. Clearly the entire quantum state of the Earth could easily be compressed in to a mass the size of Mars. :D

~iGuy

Analog Kid
Dec 19, 2006, 01:28 AM
It's not bootable for Solaris yet either, so it might be a bigger problem than it sounds like.

However to be annoyingly repetitive - since OS X boots off a separate partition, there's no reason your main "disk" couldn't be ZFS even if ZFS isn't bootable. Just make the boot partition HFS+ or UFS. Different partitions on the same disk can have different file systems (anyone who's run Bootcamp has done this).
I'm not sure what prevents booting into ZFS, but surely you can store program files under ZFS, right? If so, then only a small boot kernel would be needed to get the files system established and load the rest of the OS from there. Sounds like a good use for flash...

Of course it sounds like ZFS boot is just around the corner, so time will probably solve the problem just as well.

A question that was asked earlier was how ZFS compares to the elusive WinFS, and it never really got an answer... To restate the question differently: is ZFS the future of files systems with Spotlight and the like continuing to rely on index files, or is the future more along the lines of a file system comprising similar storage flexibility but with a database front-end?

ZFS sounds reasonably good for a large, homogenous data set, but for most desktop users the data set isn't homogenous and the file tree isn't organized. Metadata indexing is a great help in the later case.

Evangelion
Dec 19, 2006, 02:11 AM
So can Windows - RAID has very little to do with the OS.

We are not talking about RAID here. I think the word you are looking for is "Logical Volume Management (http://en.wikipedia.org/wiki/Logical_volume_management)"

Evangelion
Dec 19, 2006, 02:20 AM
First of all, yes ZFS is very cool.

BUT, it isn't going to be great for a portable MacBook or an iMac.

Laptops or iMac are not going to benefit from self-healing filesystem? A filesystem that supports snapshotting? A 100% bulletproof filesystem? Tell me: why should laptops and iMacs use HFS+ instead of ZFS? Are there any downsides in ZFS that HFS+ does not have? While there are lots of features that are ideal for hi-end systems, there are lots of features that will benefit smaller systems as well.

ZFS would be very cool for a laptop that has Flash-RAM alongside the hard-drive. With ZFS, HD and the Flash would seem like one block of storage, instead of "this thing here is the HD, this thing here is the Flash".

The power in ZFS comes in when you start to add drives in pairs. Each mirrored pair is spanned with the other mirrored pairs.

You are talking about RAID here. You don't nbeed ZFS for RAID.

Evangelion
Dec 19, 2006, 02:22 AM
Nah. People have a lot of reasons for wanting separate data stores. I don't wanna plug in my digital camera and have it become "part of the pool".

That would not happen, with or without ZFS. You could have a single pool of storage, but ad-hoc removable storage (like cameras, USB-sticks etc.) would still be their own thing.

This sounds like the right model to me. My external drives are individual units-- or at least one unit per enclosure. I don't want that assimilated into the Borg.

Then don't, sheesh. Why do you think that just because something COULD be done, it will be absolutely, positively crammed down your throat whether you want it or not?

Evangelion
Dec 19, 2006, 02:31 AM
As to having ZFS in Leopard.... I really, really, REALLY want to see that happen, and it seems that we will get it. But I don't think that it will be the default filesystem. I also think that you can't use it in the root-filesystem. That would simply be too big of a change. What I could see happening is that the version after Leopard (Lion? Lynx?) could have ZFS as the new default. Or maybe they are holding out until OS 11, the Mac OS with the Solaris-kernel ;).

Analog Kid
Dec 19, 2006, 02:34 AM
That would not happen, with or without ZFS. You could have a single pool of storage, but ad-hoc removable storage (like cameras, USB-sticks etc.) would still be their own thing.

Then don't, sheesh. Why do you think that just because something COULD be done, it will be absolutely, positively crammed down your throat whether you want it or not?
Before getting huffy, read what I was responding to... ChrisA's comment was: "Finally Mac OS can get rid of those disk drive icons. There can just be 'storage'". My point was that I believe that's going too far. Sounds like you agree...

Evangelion
Dec 19, 2006, 02:54 AM
Before getting huffy, read what I was responding to... ChrisA's comment was: "Finally Mac OS can get rid of those disk drive icons. There can just be 'storage'". My point was that I believe that's going too far. Sounds like you agree...

Cameras, iPods and the like are not disk-drive icons. So we can (and should) get rid of those hard-drive icons, while still keeping cameras and iPods as separate.

Analog Kid
Dec 19, 2006, 03:01 AM
Cameras, iPods and the like are not disk-drive icons. So we can (and should) get rid of those hard-drive icons, while still keeping cameras and iPods as separate.
My camera attaches as mass-storage. How is the system to know it's not a USB drive?

As long as we want independent storage devices, we'll want icons to distinguish them.

dnedved
Dec 19, 2006, 03:15 AM
This article left me with more questions than answers. Why does the author get LVM confused with the filesystem? Why does he spend so much time talking about spotlight performance, but there's no mention of a database driven filesystem like BeOS had a decade ago?

Are writers from USAToday switching over to write about filesystems now?!?

Evangelion
Dec 19, 2006, 03:16 AM
My camera attaches as mass-storage. How is the system to know it's not a USB drive?

trust me, the system will know. it might seem like mass-storage to the user, but the system knows what kind of hardware is connected to it. It really isn't that hard.

In Linux I can look at the data the OS has about various devices connected to it. And there's more than enough information in there to separate between different types of storage. And I don't think that things would be any worse in OS X.

As long as we want independent storage devices, we'll want icons to distinguish them.

And you would have them. And you could have them for hard-drives as well, if you so wish. So what's the problem here? ZFS would offer us a new and simplified way of managing our storage. Instead of having to think about different hard-drives, we would just have "storage". We wouldn't have to worry about having enough free space in different hard-drives, since the physical location of the files would be irrelevant. And yes, you could have separate storage-areas if you so wish.

Evangelion
Dec 19, 2006, 03:20 AM
This article left me with more questions than answers. Why does the author get LVM confused with the filesystem?

Because ZFS offers built-in LVM. So you get LWM as an added bonus with ZFS. Or as Sun says it:

Unlike traditional file systems that require a separate volume manager, Solaris ZFS introduces the integration of volume management functions. The traditional combination of file system and volume manager maintains one-to-one mapping between the file system and its associated volumes. Solaris ZFS breaks out of this limitation with the storage pool model. When capacity is no longer required by one file system in the pool, it becomes available to others.

Analog Kid
Dec 19, 2006, 03:38 AM
trust me, the system will know. it might seem like mass-storage to the user, but the system knows what kind of hardware is connected to it. It really isn't that hard.

In Linux I can look at the data the OS has about various devices connected to it. And there's more than enough information in there to separate between different types of storage. And I don't think that things would be any worse in OS X.



And you would have them. And you could have them for hard-drives as well, if you so wish. So what's the problem here? ZFS would offer us a new and simplified way of managing our storage. Instead of having to think about different hard-drives, we would just have "storage". We wouldn't have to worry about having enough free space in different hard-drives, since the physical location of the files would be irrelevant. And yes, you could have separate storage-areas if you so wish.
Most systems recognize cameras by the /DCIM directory at the root of the files system. This is also how the iPod is recognized as source for images. It's kind of moot though. I don't think you're proposing that each unique type of hardware that purports to be mass-storage should be uniquely identified.

Regardless, I see we're creating an argument where none exists... When you say "we can get rid of the hard drive icon" you apparently don't mean remove hard drive icons from the system.

The best solution I can envision is what was described earlier with different handling of inside the box and outside the box drives-- or just treat it like Disk Utility handles RAID drives and drag it into the pool.

mccoy88f
Dec 19, 2006, 03:54 AM
I think that today is possible to include ZFS in Leopard... i explain why

Porting of ZFS for FreeBSD is near complete:

Pawel Jakub Dawidek is manteiner of this porting, in a recent post he write:
"I've almost all file system functions working.
I started to run some heavy file system regression tests. They work."
this from http://www.opensolaris.org/jive/thread.jspa?threadID=12766&tstart=0

We also knew that zfs can now boot from x86 ZFS machine
http://opensolaris.org/os/project/zfsboot/
Using EFI booting for boot from ZFS i thinks i more simple...

ZFS respect POSIX standard that explain that appz don't need to edit to func with ZFS.

What does you think?

Evangelion
Dec 19, 2006, 04:10 AM
Most systems recognize cameras by the /DCIM directory at the root of the files system. This is also how the iPod is recognized as source for images. It's kind of moot though. I don't think you're proposing that each unique type of hardware that purports to be mass-storage should be uniquely identified.

Each piece of hardware IS uniquely identified. At least in Linux each piece of hardware is differentiated by their type, manufacturer and model. And each piece of hardware has an individual identification-string, so they are all individually identified. If I wanted to, I could tell the system that "when a device with this particular identification-string is attached to the system, I want you to do this". Naturally most users don't do that, since the system does it all automagically. But it could be done. I have written such rules back in the day :).

I'm not really sure what you are trying to say here. that the system does not know what pieces of hardware are connected to it? Trust me, it does know. If it doesn't, then it's broken IMO.

Regardless, I see we're creating an argument where none exists... When you say "we can get rid of the hard drive icon" you apparently don't mean remove hard drive icons from the system.

Of course there should be some kind of icon to access the filesystem. But there would be no need to separate each physical hard-drive in to separate icons, since they could all be part of one storage-system. There could be such separate icons if the user wants to, but it wouldn't be mandatory anymore.

I don't really see that much difference between the way OS X handles hard-drive volumes when compared to Windows. In Windows I have C:, D: and so forth. In OS X I have "Serenity" (the HD in my Mac Mini), "Media" (my first external HD) and "Data" (my second external hard-drive). It would be the same regardless of whether those drives were internal or external. The names are different, but there's not much difference between Windows and OS X.

With ZFS we would just have "Storage". And if that storage starts to get full, you could add another HD to the system, which would then be added to the Storage. If that happens today, I would have to buy a bigger HD, move the data to the new HD, or split it between the two hard-drives.

The best solution I can envision is what was described earlier with different handling of inside the box and outside the box drives

You could do that is you wanted to. But ZFS COULD handle those as one coherent whole. Hell, it could use network-drives as part of the storage-pool.

Analog Kid
Dec 19, 2006, 04:45 AM
Each piece of hardware IS uniquely identified. At least in Linux each piece of hardware is differentiated by their type, manufacturer and model. And each piece of hardware has an individual identification-string, so they are all individually identified. If I wanted to, I could tell the system that "when a device with this particular identification-string is attached to the system, I want you to do this". Naturally most users don't do that, since the system does it all automagically. But it could be done. I have written such rules back in the day :).

I'm not really sure what you are trying to say here. that the system does not know what pieces of hardware are connected to it? Trust me, it does know. If it doesn't, then it's broken IMO.
I haven't done Linux system hacking myself, but I've worked with Windows enough to know what happens if you try to make decisions on too fine grained of a level. Generally that leads to needing drivers for every specific implementation, whereas the OS X model is to rely on the standards to abstract device classes.

I've never had to install a device driver for a camera, and I know it's not because OS X recognizes each and every camera. They all mount as "disks" (evidenced by the disk icon in Finder). Higher level system services rely on the fact that standard camera media places the /DCIM directory at the root and launch an application if I've set it up to do so, but cameras are treated as a special case in this regard.

If I pull the media from my camera and put it into a card reader, the Mac still treats it as an image source because it's a disk with the key directory, not because it recognizes each unique type of memory card. My card reader didn't require special drivers even though it's an off brand that Apple certainly didn't write a driver for itself.

If I connect a piece of custom hardware we've developed, it opens as mass storage with no special directory to key off of. We didn't design it to be Mac compatible, but it is because it conforms to the Mass Storage standard spec for USB. All of the files I've collected on the device are there for transfer. This is good general behavior and the Mac isn't broken for not knowing I've attached some unknown piece of hardware.

Of course there should be some kind of icon to access the filesystem. But there would be no need to separate each physical hard-drive in to separate icons, since they could all be part of one storage-system. There could be such separate icons if the user wants to, but it wouldn't be mandatory anymore.

...

With ZFS we would just have "Storage". And if that storage starts to get full, you could add another HD to the system, which would then be added to the Storage. If that happens today, I would have to buy a bigger HD, move the data to the new HD, or split it between the two hard-drives.

You could do that is you wanted to. But ZFS COULD handle those as one coherent whole. Hell, it could use network-drives as part of the storage-pool.
Yes. I agree with you. As I said, the pool could be handled in the same way Disk Utility handles RAID drives-- one icon for the entire RAID. Drives aren't subsumed into the RAID just by being attached, however-- I need to do that intentionally. Yes, I know you can't just add another drive to an existing RAID and that the pool has all kinds of additional advantages, I'm only using the RAID example as a, well... example.

The only point I've been trying to make is that I don't want ZFS making these decisions for me, and setting it up to do so would be dangerous. Nothing deep or terribly meaningful here...

xUKHCx
Dec 19, 2006, 04:46 AM
I don't really see that much difference between the way OS X handles hard-drive volumes when compared to Windows. In Windows I have C:, D: and so forth. In OS X I have "Serenity" (the HD in my Mac Mini), "Media" (my first external HD) and "Data" (my second external hard-drive). It would be the same regardless of whether those drives were internal or external. The names are different, but there's not much difference between Windows and OS X.


There is a big difference and it is extremely annoying. If you have links to a file that is on the external drive. In Windows when you attach drive "Serenity" there is no guarantee what letter that drive will be assigned. However on a mac it just doesnt matter.

Which does annoy the hell out of me, for example on my USB stick i have a few portable apps, Firefox etc. etc, they create their own folders so i hide them inside a hidden folder. I have links to the applications so i can run them easily. However depending on what other drive is attached these links will not work. Which is shocking because the links are on the drive linking to other files on the drive, this is a pretty serious problem in my view.

Bonte
Dec 19, 2006, 04:59 AM
Until they start putting multiple hard drives in a MacBookPro, ZFS is primarily a Workstation or Server feature.

And yes, that is still a fantastic addtion to OS X.

It also compresses data in realtime and saves HD space with a large number of small files (for example osX itself has thousands of ultra small txt files), i'm sure this will be beneficial for small HD's in laptops, mini's and iTV's.

Evangelion
Dec 19, 2006, 05:05 AM
I haven't done Linux system hacking myself, but I've worked with Windows enough to know what happens if you try to make decisions on too fine grained of a level.

We are talking about a difference between a hard-drive and a camera here. I don't think that's "too fine grained" ;).

Generally that leads to needing drivers for every specific implementation

um, I'm not talking about drivers. Just because each piece of hardware is uniquely identified does not mean that each of them needs specific drivers. It just means that the OS knows what is connected to the system, and where. In Linux you could even see where a USB-stick is connected. In case of Apple Keyboard, it would basically tell that "USB stick is connected to Port 2 on Apple Keyboard, which in turn is connected to Port 3 on the computer". And the system would recognize one USB-stick from another, but it wouldn't need specific drivers for each.

The unique identification I'm talking about simply means that the system can tell different pieces of hardware from each other, that's all. And it can do that even if they are identical make and model. There's no reason why OS X couldn't do that. And that unique identifier would not mean that they would need specific drivers. It could just mean that the printer on the left side of the computer is different printer than the identical printer on the right side of the computer.

whereas the OS X model is to rely on the standards to abstract device classes.

As does Linux. But that doesn't mean that each piece of hardware couldn't be uniquely identified. I really have the feeling here that you have misunderstood what I have been saying here.

I've never had to install a device driver for a camera, and I know it's not because OS X recognizes each and every camera. They all mount as "disks" (evidenced by the disk icon in Finder).

yes they do, but do you REALLY think that when you plug in your camera, the only thing OS X knows is that "someone plugged in a disk"? No, it knows a lot more. For starters, when I plug in my camera, OS X somehow knows that it needs to start iPhoto. It doesn't start iPhoto when I plug in my USB-stick, so OS X can clearly tell the two apart.

The system knows quite a bit about the hardware that is plugged in. It just doesn't pass that information to the user because the user does not need to know it. I would think that OS X knows as much as Linux does. Difference is that Linux has tools that can extract that information and present it to the user if he wants it. I have looked at the info Linux has, and it's more than enough to differentiate between different devices. And thanks to that differentiation, there is no risk that your digital camera would get added to your logical volume

Analog Kid
Dec 19, 2006, 05:37 AM
We are talking about a difference between a hard-drive and a camera here. I don't think that's "too fine grained" ;).
It would be if you used my other example of custom hardware...

The unique identification I'm talking about simply means that the system can tell different pieces of hardware from each other, that's all. And it can do that even if they are identical make and model. There's no reason why OS X couldn't do that. And that unique identifier would not mean that they would need specific drivers. It could just mean that the printer on the left side of the computer is different printer than the identical printer on the right side of the computer.
OS X can. Go to Apple->About this Mac->More Info and you'll see a very similar set of information.

As does Linux. But that doesn't mean that each piece of hardware couldn't be uniquely identified. I really have the feeling here that you have misunderstood what I have been saying here.I must be... And we've gotten into this whole side discussion about cameras which were merely an example of my point because most people have one. I'm not saying OS X can't tell one piece of hardware from another, I'm saying I don't want my file system to make it's own decisions about what to add to a storage pool.

I'm saying there are many, many types of devices that act as disks but that shouldn't be rolled in with all the other storage on my system. The system can't possibly be smart enough to know what I intend to with what I just attached.

yes they do, but do you REALLY think that when you plug in your camera, the only thing OS X knows is that "someone plugged in a disk"? No, it knows a lot more. For starters, when I plug in my camera, OS X somehow knows that it needs to start iPhoto. It doesn't start iPhoto when I plug in my USB-stick, so OS X can clearly tell the two apart.

It's too late for me to try this experiment myself, but if you're curious let me suggest it: take your USB stick, format it as FAT, copy the DCIM directory from your camera to the root of the USB stick. Remove the stick and reinsert it. I'm guessing iPhoto recognizes it as an image source. Maybe I'm wrong, and maybe OS X is recognizing that my USB drive is really an xD card wrapped into a USB stick, but it doesn't change the fact that adding a step between insertion of a drive and adding it to the pool is a smarter, safer thing to do.

Uncle.

Evangelion
Dec 19, 2006, 05:45 AM
OS X can. Go to Apple->About this Mac->More Info and you'll see a very similar set of information.

And I did say that I expected as much. So what exactly is the problem here? OS X already does what I have been talking about, and we (obviously) do not have the problems you mentioned.

I'm not saying OS X can't tell one piece of hardware from another, I'm saying I don't want my file system to make it's own decisions about what to add to a storage pool.

And you could do just that. And I think that even if the OS handled all this automagically, it would be smart enough to NOT add cameras and iPods to the storage-pool.

I'm saying there are many, many types of devices that act as disks but that shouldn't be rolled in with all the other storage on my system.

And they wouldn't be.

The system can't possibly be smart enough to know what I intend to with what I just attached.

But it would be smart enough to not add your iPod to your storage-pool ;). Thats the point. And, of course, you could still decide what to do with each individual component. If you want your Firewire-HD to be added to the pool, you could do so. Or you might want to keep it separate, And you could do so.

I'm not really seeing any issues here.

4np
Dec 19, 2006, 05:51 AM
Could this mean revolutionary things ahead for the next OS? Perhaps better integration with new hardware?

Leopard's Time Machine is a front end to ZFS's snapshots...

Terranova
Dec 19, 2006, 06:49 AM
You can do that now. It uses LVM, you can sync partitions without reboot, just use partprobe, pvcreate to make the partitions to volumes, suck them all together with vgcreate and use lvextend to make them bigger, hfsonline to make it. You can grow a drive to whatever you want.

You can also use pvmove, vgreduce, pgremove, if a drive is about to fail and you have another in sync as a mirror and bring it online.

Great! sounds easy! I'll try it after West Wing tonight.

Morris
Dec 19, 2006, 07:18 AM
As said by others, ZFS can have tremendous benefits, even for regular users with just one harddrive.

Granted, most of the benefits ZFS has over HFS+ are things you don't need until you start using more storage. Still, among other things, home users can greatly benefit from the added reliability and handling of the fast growing amount of very small files.

Regarding reliability, picture this. A friend of mine had very weird things going on with his Powerbook. Seemingly random kernel panics, crashes of software, difficulty booting. A re-install of Tiger would help but only for two weeks at most. After removing part of his memory to rule out memory failures it turned out to be his drive silently 'losing bits'. Sometimes the losses were in parts of no importance to the OS, like pictures or music. However, whenever a few bits or bytes in important files got messed up the trouble began.
This silent deterioration of your data is impossible with ZFS. At worst you'd be immediately warned and have time to create backups and get another drive. At best (when using more than one drive in some kind of pool) data will be automatically kept healthy.

Regarding small files, think of Spotlight. With more and more apps getting developed with Spotlight in mind, the number of small files increases significantly because Spotlight likes every chunk of data to have it's own file. Mail.app switched from a few big files storing all your e-mail to one file per e-mail when Tiger came out. In my situation that meant over 30.000 tiny files instead of a few big ones!
Most file systems have trouble with small files in relation to efficiency resulting in them taking up much more space than needed or slowing down. Not with ZFS.

Hhmm... I should look for a job in marketing... any tech marketing jobs in London? :D

wrldwzrd89
Dec 19, 2006, 07:48 AM
ZFS sounds so much like the wave of the future for Mac OS X. I agree with all the other points that have been raised, namely:


Letting the user decide what goes in the ZFS pool
Vista has officially been dwarfed
Time Machine will benefit hugely from ZFS
Will Apple make ZFS bootable?


I'd love to see ZFS offered as an option in Leopard - even if it's not the default, I'd use it just for the coolness factor :D

xUKHCx
Dec 19, 2006, 08:19 AM
I'd love to see ZFS offered as an option in Leopard - even if it's not the default, I'd use it just for the coolness factor :D

Same here:D

jholzner
Dec 19, 2006, 08:35 AM
Leopard's Time Machine is a front end to ZFS's snapshots...

Wow, this rumor dies hard huh? Tme Machine has nothing to do with ZFS in anyway. I thinks it's been mentioned in this thread even.

http://storagemojo.com/?p=213

Digitalclips
Dec 19, 2006, 08:59 AM
This sounds very interesting, always good to be able to use the latest new tech.

From the Wikipedia article: "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

I wonder how long until we hit that limit :P

I think they say it would take more energy to polulate it than it would to boil the oceans, not that you have to actualy boil the oceans to populate it.

Digitalclips
Dec 19, 2006, 09:03 AM
ZFS sounds so much like the wave of the future for Mac OS X. I agree with all the other points that have been raised, namely:


Letting the user decide what goes in the ZFS pool
Vista has officially been dwarfed
Time Machine will benefit hugely from ZFS
Will Apple make ZFS bootable?


I'd love to see ZFS offered as an option in Leopard - even if it's not the default, I'd use it just for the coolness factor :D

And in no time at all we will all accept is as standard on Macs and talk fondly of the 'old HFS days', meanwhile Vista is based on what? It isn't still underpinned by DOS from the 1970's is it?

Trekkie
Dec 19, 2006, 09:08 AM
You are talking about RAID here. You don't nbeed ZFS for RAID.

So if I have two 250GB Drives in my ZFS files system, and 400GB of data, and one 250GB drive goes the way of the dodo I don't need RAID?

That's quite the magical file system.

Rocketman
Dec 19, 2006, 09:20 AM
Yes, that certainly explains why Time Machine exists in the current developer's builds that are running on HFS+ :rolleyes:

ZFS would make a great basis for an implementation of Time Machine (read as: where we are going) but the program itself will not require a ZFS file system, it will be compatible with HFS+ for current users' sake.

I'm hoping for an HFS/HFS+ type transistion: ZFS will be in OS 10.5, you can choose to reformat your hard drive and start using ZFS for your boot partition, or keep using HFS+ for now. If you do change to ZFS you'll get better Time Machine performance, the pooled storage, ect.

Then at some point in the future (10.6, 10.7?) they will drop the HFS+ and you'll have to use a ZFS file system to upgrade.

Everyone keeps talking about how ZFS booting is not ready yet. Silly users, ZFS is Open-Source, is it not possible Apple has been working on booting ability on their own and have solved the issue for 10.5?

One thing that will be a problem in the beginning is there will be no DiskWarrior, ect to repair drives with. But given the features of ZFS, I wonder what use there would be for diagnostic programs to begin with. :D

I hear you.

Apple licensed ZFS from whoever, and then started adding features of that system to both HFS+ version of the OS and linking the OS to use ZFS directly. Perhaps there are patented aspects of ZFS Apple had to license to use within HFS+ for commercial resale?

Perhaps in 10.5 TM will use a live size-changing ZFS partition to implement TM features. And I ask, why NOT have two drives in a laptop? The second one can be reallly tiny.

As I have been saying for years now: RAID everywhere.

Rocketman

wrldwzrd89
Dec 19, 2006, 09:31 AM
And in no time at all we will all accept is as standard on Macs and talk fondly of the 'old HFS days', meanwhile Vista is based on what? It isn't still underpinned by DOS from the 1970's is it?
Based on what I hear from Microsoft, there are still 2 vestiges of DOS left behind in today's Windows (Vista):


cmd.exe - the command line
Drive letters


Cmd.exe was supposed to go away with the new UNIX-like command line called Monad, which was pulled at the last minute for undisclosed reasons.

As for drive letters, rumors placed their demise with Windows Vista as well, but this too was pulled, presumably for the sake of backward compatibility when Vista was restarted and based on the Windows Server 2003 codebase early in its development.

Westside guy
Dec 19, 2006, 09:59 AM
We are not talking about RAID here. I think the word you are looking for is "Logical Volume Management (http://en.wikipedia.org/wiki/Logical_volume_management)"

If that's was was being referred to, then the post I was responding to was incorrect. LVM needs to be set up on Linux or BSD (or whatever) - it's not how disk management is done by default on other Unixes. And there are Windows-based drive-spanning solutions as well. There have also been drive-vendor-specific offerings that do this (clunkily).

Cameras, iPods and the like are not disk-drive icons. So we can (and should) get rid of those hard-drive icons, while still keeping cameras and iPods as separate.

Not to mention that it's unlikely cameras would be anything other than FAT16/FAT32.

I originally was going to say "and iPods", but then I realized that might not be a given on the Mac end.

If I wanted to, I could tell the system that "when a device with this particular identification-string is attached to the system, I want you to do this". Naturally most users don't do that, since the system does it all automagically. But it could be done. I have written such rules back in the day :).

Hehe, it wasn't very long ago that you HAD to do exactly this under Linux for a lot of devices (which I'm guessing is exactly what you're referring to).

MacVault
Dec 19, 2006, 10:21 AM
This article left me with more questions than answers. Why does the author get LVM confused with the filesystem? Why does he spend so much time talking about spotlight performance, but there's no mention of a database driven filesystem like BeOS had a decade ago?

Are writers from USAToday switching over to write about filesystems now?!?

Yea, ever since this ZFS in Leopard news surfaced I've been wondering how it compares to the database-driven filesystem of BeOS that was talked about in the past few years as being the holy grail of filesystems. Would someone enlighten us as to whether ZFS offers the same technologies and advantages as the BeOS filesystem??? Oh, and what about the WinFS (vaporware concept? How does ZFS compare to these "database-metadata-driven" filesystems?

shawnce
Dec 19, 2006, 10:29 AM
I hear you.

Apple licensed ZFS from whoever, and then started adding features of that system to both HFS+ version of the OS and linking the OS to use ZFS directly. Perhaps there are patented aspects of ZFS Apple had to license to use within HFS+ for commercial resale? Lets not make stuff up.... HFS+ has had the capability that Time Machine leverages since HFS+ was integrated with Mac OS X. Nothing was borrowed from ZFS for this, in fact a feature that ZFS doesn't yet have was added to HFS+ in 10.5. Also ZFS is available via the CDDL open source license (thanks to Sun for making it available).

MacVault
Dec 19, 2006, 10:58 AM
http://images.macrumors.com/article/zfsleopard.png



This image showing ZFS in what looks like Disk Utility appears to be slightly transparent. If you look closely there are some bullet points. I just opened Disk Utility (not on Leopard) and don't see any transparency in it. What's the explanation for this??? Is Leopard's Disk Utility transparent? Why do I see text in the background? Or is this just some bad photoshop job?

Morris
Dec 19, 2006, 11:06 AM
This image showing ZFS in what looks like Disk Utility appears to be slightly transparent. If you look closely there are some bullet points. I just opened Disk Utility (not on Leopard) and don't see any transparency in it. What's the explanation for this??? Is Leopard's Disk Utility transparent? Why do I see text in the background? Or is this just some bad photoshop job?

When I look closely I can read: "To repair disk permission problems..." and "To repair yo...... choose Utilities > Disk Utilities". That seems to me genuine content to appear on a preceding screen. There you'd select to create a volume and then this screen would pop up. Apparently slightly transparent.

You'd have to be a very cunning PS con man to decide to make up a completely new window and options and to show it with just 10% transparency in the background.

Peace
Dec 19, 2006, 11:07 AM
This image showing ZFS in what looks like Disk Utility appears to be slightly transparent. If you look closely there are some bullet points. I just opened Disk Utility (not on Leopard) and don't see any transparency in it. What's the explanation for this??? Is Leopard's Disk Utility transparent? Why do I see text in the background? Or is this just some bad photoshop job?


It's real.

shawnce
Dec 19, 2006, 11:11 AM
This image showing ZFS in what looks like Disk Utility appears to be slightly transparent. If you look closely there are some bullet points. I just opened Disk Utility (not on Leopard) and don't see any transparency in it. What's the explanation for this??? Is Leopard's Disk Utility transparent? Why do I see text in the background? Or is this just some bad photoshop job? It looks like a standard sheet (hence sightly transparent) that you see in disk utility when you go to create a disk image. Go try it again in Tiger.

steelfist
Dec 19, 2006, 11:48 AM
although it's good, what matters even more is if microsoft allows apple to access and write ntfs and their new filesystem (dunno if they have or not)

Goldfinger
Dec 19, 2006, 12:44 PM
although it's good, what matters even more is if microsoft allows apple to access and write ntfs and their new filesystem (dunno if they have or not)

The problem with this is that NTFS is closed source. And they'd have to reverse engineer it or licensce it.

AppliedVisual
Dec 19, 2006, 12:56 PM
The problem with this is that NTFS is closed source. And they'd have to reverse engineer it or licensce it.

Yep... NTFS read AND WRITE would sure be useful. But like you say, Microsoft has done little to allow such things on other platforms. They have made available basic licensing so that other platforms can READ NTFS volumes, but that's about it.

SPUY767
Dec 19, 2006, 01:37 PM
am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? If so, then wow. Simply wow.

Basically, OS X has hundreds of thousands of files that only have a few hundred bytes of info or less in them, very tiny. The smnallest allocation block for HFS+ is 4k, correct me if I'm wrong. Now, instead of that 152Byte file taking up 30x the space that it needs to, it may only take up 200Bytes. Make sense, the problem was woprse with old Mac file systems, system 7 had a minimum allocation block of 32k, so your 15 byte text doc saying, "hello there bob" took up 2000x the space that the file actually contained. The real interesting thing is, that the old mac OS used to tell us how much space the file took up on the drive, i.e. the Get Info window would never show a file less than 32k, but OS X tells us only how large the data contained in the file is, not how large the block of data holding the file on the hard drive is.

Goldfinger
Dec 19, 2006, 05:17 PM
the Get Info window would never show a file less than 32k, but OS X tells us only how large the data contained in the file is, not how large the block of data holding the file on the hard drive is.

Yes it does. U just made an RTF with the text "bla" in it. Get Info tells me:
4KB on disc (341 bytes).

charkshark
Dec 19, 2006, 05:38 PM
ZFS is quite feasible for an OS as ahead of it's time as (I believe) leopard will be. It is a very capable file system for the future, and have no problem accepting as the future file-system for Mac OS.

SPUY767
Dec 19, 2006, 06:31 PM
Yes it does. U just made an RTF with the text "bla" in it. Get Info tells me:
4KB on disc (341 bytes).

I was typing fast and misspoke when what I meant to say was only that Old Mac OS would ONLY show the disk size.

SeaFox
Dec 19, 2006, 06:47 PM
I was typing fast and misspoke when what I meant to say was only that Old Mac OS would ONLY show the disk size.

No, that's still incorrect. I'm not in front of my Quadra right now, but I remember the Get Info window having a size, and a "size on disk".

daveL
Dec 19, 2006, 06:57 PM
I think that today is possible to include ZFS in Leopard... i explain why

Porting of ZFS for FreeBSD is near complete:

Pawel Jakub Dawidek is manteiner of this porting, in a recent post he write:
"I've almost all file system functions working.
I started to run some heavy file system regression tests. They work."
this from http://www.opensolaris.org/jive/thread.jspa?threadID=12766&tstart=0

We also knew that zfs can now boot from x86 ZFS machine
http://opensolaris.org/os/project/zfsboot/
Using EFI booting for boot from ZFS i thinks i more simple...

ZFS respect POSIX standard that explain that appz don't need to edit to func with ZFS.

What does you think?
The ZFS Boot project hasn't been updated since the 30th of August. You'll also notice that it's a "hidden" project. The last time it was updated was build 46; the current build is 54 - soon to be 55. Not to mention you have to jump through a bunch of hoops to get build 46 to actually boot off of ZFS. It appears to me that the project has been put on the back burner.

On top of that, there's not much similarity between the boot process on Solaris and the boot process on OS X. I think Apple would have to put some work into ZFS for OS X to be able to boot off of it. However, another approach is to have a small HFS+ boot partition, but install the actual OS in a separate ZFS partition.

Just my 2 cents.

Manic Mouse
Dec 20, 2006, 06:36 AM
The ZFS Boot project hasn't been updated since the 30th of August. You'll also notice that it's a "hidden" project. The last time it was updated was build 46; the current build is 54 - soon to be 55. Not to mention you have to jump through a bunch of hoops to get build 46 to actually boot off of ZFS. It appears to me that the project has been put on the back burner.

On top of that, there's not much similarity between the boot process on Solaris and the boot process on OS X. I think Apple would have to put some work into ZFS for OS X to be able to boot off of it. However, another approach is to have a small HFS+ boot partition, but install the actual OS in a separate ZFS partition.

Just my 2 cents.

Or have that small boot partition as part of the flash RAM in the new hybrid HDDs.

wrldwzrd89
Dec 20, 2006, 06:55 AM
Or have that small boot partition as part of the flash RAM in the new hybrid HDDs.
That would be interesting, but isn't the flash portion of these HDDs meant to be used as an additional cache and not as storage space? If that's true I don't see how your idea could be implemented.

sjk
Dec 20, 2006, 03:32 PM
I was going nuts while reading this thread on a non-networked PDA and not having a chance to respond until now, especially about this one particular issue:

Not to mention that it's unlikely cameras would be anything other than FAT16/FAT32.
That's the closest anyone's come to what I've been waiting to say

I originally was going to say "and iPods", but then I realized that might not be a given on the Mac end.
That the iPod may be "on the Mac end" is probably irrelevant.

Unless I missed it what others posts about this seem to be overlooking is that it's the media/storage format, not the device type, that makes the most difference whether or not it's a candidate for being added to a ZFS storage pool.

If you attach a camera, iPod, etc. that's already formatted with FAT32, HFS+, or any other filesystem than ZFS there's no reason for the system to consider it for a ZFS pool(!). And it may trigger an app to open or some other behavior depending on the device and/or format, which I remember being mentioned in this thread but I'm too lazy to find the post(s).

If you attach a device with uninitialized/unformatted media the system might prompt for how to handle it (e.g. give Time Machine permission to use it) or do something automatically.

That simple explanation makes the most sense to me though it's certainly possible I've overlooked something (obvious or not). I really don't see how the presence of ZFS on OS X would radically change how the system currently handles multiple filesystems and storage formats. If anyone knows better please enlighten me/us. :)

bob5820
Dec 22, 2006, 08:04 AM
All of the world's printed data (today) would not come close to filling even a 2^64 bytes. 2^128 is larger than 2^64 by a factor of 2^64. I think it is safe to say all of the world's recored data could fit. This is different than storing the state of the Earth. I doubt the state of the Earth could be stored on a device only the size of the Earth, it would need to be larger.

Of course you could save about 2^32 by boiling off the oceans first :D

weldon
Dec 28, 2006, 04:57 PM
I love that ZFS is opensource. How great would it be if someone were to write rock-solid ZFS support for Windows XP and Vista. Then we could have shared volumes between OS X and Windows (Boot Camp or virtualization) that supported multimedia files (>4GB) w/o hacky software that doesn't work at times.

tyr2
Jan 3, 2007, 11:19 AM
Just came across this (http://themachackers.com/2006/12/19/zfs-on-mac-os-x-105-a-closer-look/) article showing zfs from the command line on Leopard.

All the stuff like 'zpool create' etc.. appear to be there. Just seems to have to dig a bit deeper than 'disk utility' to get to the good stuff.