ZFS in Latest Leopard Builds

[Analog Kid]

Ars Technica has suggested that this would be required for Time Machine to function, but isn't Time Machine already included in the developer's previews?

I think this has been implied earlier on the thread, but never stated outright: Ars suggested that Time Machine was a whole lot like ZFS, but probably wasn't ZFS. Right now, Time Machine likely uses a series of hard links in the BSD layer or something similar to keep the snapshots. Not a bad approach (it's how we do backups at work) but not very efficient for big files.

From the Wikipedia article: "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

Great, data storage as a primary cause of global warming... "Americans are 5% of the worlds population but they generate 35% of the worlds data" is just around the corner now...

1) All operations are "copy on write".
data are never over written you can always go back to last month's version . The system saves space by only writing changes.

Well, you can get last month's version assuming nothing else took that block for new storage...

Finally Mac OS can get rid of those disk drive icons. There can just be "storage" Much like you don't care much about how many RAM chips you have, you only care about the total amount of RAM. Disk can be like that too. Just open the box and slide in one more disk and the rest is "magic". Just like with RAM.

Nah. People have a lot of reasons for wanting separate data stores. I don't wanna plug in my digital camera and have it become "part of the pool". I like my firewire drives separate so I can power them down and shut them up. There will always be a place for human storage management, it's just that the pool approach is good for large data stores.

4) there is not enough data on Earth to fil a ZFS file system. Or at least if you wrote to a disk 24x7 you would not live long enough to fill up ZFS. (although you would fill up quite a few physical drives)

Statements like this are just asking to be proven silly... One ZFS file system can store the DNA sequences of every cell in every creature on the planet? The state of every electron in ever atom on the planet? The energy, frequency, and vector of every photon passing by the planet? 3e38 is big, but not infinite...

You can do that now. It uses LVM, you can sync partitions without reboot, just use partprobe, pvcreate to make the partitions to volumes, suck them all together with vgcreate and use lvextend to make them bigger, hfsonline to make it. You can grow a drive to whatever you want.

You can also use pvmove, vgreduce, pgremove, if a drive is about to fail and you have another in sync as a mirror and bring it online.

Has this been wrapped in a tool anywhere? Seems like there's plenty of people who could use this now, but wouldn't want to touch the command line...

But moreover, if ZFS is the default for Leopard (that's a pretty big if, but not out of the question -- God bless "top secret" features) then I would imagine that it would automatically create a pool out of any internal drives but that external drives, by default, would not be considered part of that pool.

This is the kind of thing that Apple excels at: because the internal hard drives are inside the box, we should think of them as one -- together they are the storage space of the machine. But anything outside of the box, logically, seems as though it should be a separate part unless you specifically tell the computer otherwise. Thus, there could be a checkbox under "Get Info" or something similar to add that drive to the pool. I can't imagine that Apple would make it much more complicated than that.

This sounds like the right model to me. My external drives are individual units-- or at least one unit per enclosure. I don't want that assimilated into the Borg. Anything inside the case is considered "primary storage" and should flow together and cooperate.

 

[Digitalclips]

I wonder where Microsoft are with their R&D (i.e. copy and / or rip off Apple) are with this?

 

[Analog Kid]

Leopard is looking to be a much bigger update than I anticipated. It's going to have a lot more things than Tiger had. Tiger seemed more like a "features" update, whereas Leopard is overhauling a whole bunch of stuff, while still giving us some cool features. Every major update has done that, except for Tiger to an extent. So, I guess you could compare Tiger that middle movie in the trilogy that doesn't do as much but is necessary to get you from one to three. **cough, cough** Pirates of the Caribbean **cough, cough**

Tiger also overhauled a whole bunch of stuff, little of it was indicated with little GUI lights though so it's only noticed as improvements in performance and stability.

I wonder where Microsoft are with their R&D (i.e. copy and / or rip off Apple) are with this?

Uh... You did notice that ZFS is a Sun designed file system, right?

 

[iGuy]

not quite

am i getting this wrong, or could I format my 100GB HD to ZFS and have it store as much data as a 1TB+ HFS drive? If so, then wow. Simply wow.

However, if you add another nine, 100GB HDs, you could create a 1TB logical volume.

You can do this already in OSX 10.4 Tiger using Disk Utility. But the difference with ZFS is that this 'pooling' of hard disks to create larger logical volumes is handled at the filesystem level, not the OS level.

I think I got that right?

There are reasons that you may want to do this and reasons that you may not.

So far in this thread I haven't seen much reference to the size and type of files being stored. For example you want to handle large binary image/video files differently than you handle large numbers of small files.

Most of the discussion so far assumes that most average users store a large number of smaller files. And this is a reasonable assumption.

Just be careful not to jump to conclusions or make broad assumptions. When Apple does release an OS that supports ZFS be sure to take the time to think about what storage configuration is going to work best for you most of the time. Nothing is perfect or absolute.

~iGuy

 

[AppliedVisual]

Tiger also overhauled a whole bunch of stuff, little of it was indicated with little GUI lights though so it's only noticed as improvements in performance and stability.

Uh... You did notice that ZFS is a Sun designed file system, right?

And ZFS is available for Windows too... At a price.

 

[vertgo]

finally

HFS+ seems to be a little aging, and now there's finally some path for Apple to go to a next generation (or two gens later) file system. I hope they migrate native filesystem to this, eventually. Then we can have permanently versioned, self healing files. I would want ZRaid standard on my mac pros

 

[ChrisA]

I wonder where Microsoft are with their R&D (i.e. copy and / or rip off Apple) are with this?

ZFS was written by Sun and is part of Solaris. Sun made Solaris Open source and give it away to everyone. There is no need to "zip off" what is available for free. Apple picked up two important Open Source parts of Solaris: Dtrace and ZFS. These are some of the best technical ideas to come out along in the UNIX world in a long time. We should thank Sun for this. Some day maybe Apple will contribute something back

I doubt ZFS could be made to work under Windows. Put the source is available so anyone could give it a try

 

[Westside guy]

Oh I know. I've known what ZFS is for a long time now. But all I'm saying is that the concept of pools could be confusing for the average consumer.
I'm not saying in any way that ZFS and it's capabilities suck and that you HAVE to use it. ZFS is awesome.

The main problem for consumers IMHO is that when they start mixing their internal drives with their external USB and FireWire drives. One day they unplug their drive and boom there goes the filesystem.

All I'm saying is that ZFS needs some clear explenation. The OS should warn you if you use it and explain how it works since the concept of pools is quite different from the current filesystem concepts.

But I'm all for ZFS, let me make that clear.

Hehe, sorry. I misunderstood where you were coming from, so to speak. And I've got to admit I've only read about ZFS, not used it (sounds like it might be a while before it's available on Linux, and even then it'll be clunky if they're going through FUSE).

I would guess that, given Apple's usual way of doing things, those potentially problematic parts will be made unavailable to the average user. Of course some of us will whine and complain about the crippled functionality in that case.

 

[x704]

I'll tell you what Apple, when you can make my computer boot from ZFS, I will buy the upgrade (at student discount of course).
I would imagine by the time they intergrate ZFS into their OS's, there will be lots of other reasons to upgrade. Like a system wide vector/nerbs windowing system, much like CAD. Then <control><scroolwheel> will look ausom!

 

[Westside guy]

Everyone keeps talking about how ZFS booting is not ready yet. Silly users, ZFS is Open-Source, is it not possible Apple has been working on booting ability on their own and have solved the issue for 10.5?

It's not bootable for Solaris yet either, so it might be a bigger problem than it sounds like.

However to be annoyingly repetitive - since OS X boots off a separate partition, there's no reason your main "disk" couldn't be ZFS even if ZFS isn't bootable. Just make the boot partition HFS+ or UFS. Different partitions on the same disk can have different file systems (anyone who's run Bootcamp has done this).

 

[ChrisA]

Statements like this are just asking to be proven silly... One ZFS file system can store the DNA sequences of every cell in every creature on the planet? The state of every electron in ever atom on the planet? The energy, frequency, and vector of every photon passing by the planet? 3e38 is big, but not infinite...

All of the world's printed data (today) would not come close to filling even a 2^64 bytes. 2^128 is larger than 2^64 by a factor of 2^64. I think it is safe to say all of the world's recored data could fit. This is different than storing the state of the Earth. I doubt the state of the Earth could be stored on a device only the size of the Earth, it would need to be larger.

But who cares. The point is that this thing will not need to be redesigned on 20 years. It can last the rest of our lives at least. After that I don't care.

 

[Westside guy]

Check out ZFS Boot.

Hey that's cool - doubly so because it plugs into Grub. I am probably dreaming, but if Intel Macs could be booted from Grub it would be trivial to make them multi-boot Linux, Windows, etc.

 

[iGuy]

size does matter

All of the world's printed data (today) would not come close to filling even a 2^64 bytes. 2^128 is larger than 2^64 by a factor of 2^64. I think it is safe to say all of the world's recored data could fit. This is different than storing the state of the Earth. I doubt the state of the Earth could be stored on a device only the size of the Earth, it would need to be larger.

You've overlooked ZFS's compression. Clearly the entire quantum state of the Earth could easily be compressed in to a mass the size of Mars.

~iGuy

 

[Analog Kid]

It's not bootable for Solaris yet either, so it might be a bigger problem than it sounds like.

However to be annoyingly repetitive - since OS X boots off a separate partition, there's no reason your main "disk" couldn't be ZFS even if ZFS isn't bootable. Just make the boot partition HFS+ or UFS. Different partitions on the same disk can have different file systems (anyone who's run Bootcamp has done this).

I'm not sure what prevents booting into ZFS, but surely you can store program files under ZFS, right? If so, then only a small boot kernel would be needed to get the files system established and load the rest of the OS from there. Sounds like a good use for flash...

Of course it sounds like ZFS boot is just around the corner, so time will probably solve the problem just as well.

A question that was asked earlier was how ZFS compares to the elusive WinFS, and it never really got an answer... To restate the question differently: is ZFS the future of files systems with Spotlight and the like continuing to rely on index files, or is the future more along the lines of a file system comprising similar storage flexibility but with a database front-end?

ZFS sounds reasonably good for a large, homogenous data set, but for most desktop users the data set isn't homogenous and the file tree isn't organized. Metadata indexing is a great help in the later case.

 

[Evangelion]

So can Windows - RAID has very little to do with the OS.

We are not talking about RAID here. I think the word you are looking for is "Logical Volume Management"

 

[Evangelion]

First of all, yes ZFS is very cool.

BUT, it isn't going to be great for a portable MacBook or an iMac.

Laptops or iMac are not going to benefit from self-healing filesystem? A filesystem that supports snapshotting? A 100% bulletproof filesystem? Tell me: why should laptops and iMacs use HFS+ instead of ZFS? Are there any downsides in ZFS that HFS+ does not have? While there are lots of features that are ideal for hi-end systems, there are lots of features that will benefit smaller systems as well.

ZFS would be very cool for a laptop that has Flash-RAM alongside the hard-drive. With ZFS, HD and the Flash would seem like one block of storage, instead of "this thing here is the HD, this thing here is the Flash".

The power in ZFS comes in when you start to add drives in pairs. Each mirrored pair is spanned with the other mirrored pairs.

You are talking about RAID here. You don't nbeed ZFS for RAID.

 

[Evangelion]

Nah. People have a lot of reasons for wanting separate data stores. I don't wanna plug in my digital camera and have it become "part of the pool".

That would not happen, with or without ZFS. You could have a single pool of storage, but ad-hoc removable storage (like cameras, USB-sticks etc.) would still be their own thing.

This sounds like the right model to me. My external drives are individual units-- or at least one unit per enclosure. I don't want that assimilated into the Borg.

Then don't, sheesh. Why do you think that just because something COULD be done, it will be absolutely, positively crammed down your throat whether you want it or not?

 

[Evangelion]

As to having ZFS in Leopard.... I really, really, REALLY want to see that happen, and it seems that we will get it. But I don't think that it will be the default filesystem. I also think that you can't use it in the root-filesystem. That would simply be too big of a change. What I could see happening is that the version after Leopard (Lion? Lynx?) could have ZFS as the new default. Or maybe they are holding out until OS 11, the Mac OS with the Solaris-kernel .

 

[Analog Kid]

That would not happen, with or without ZFS. You could have a single pool of storage, but ad-hoc removable storage (like cameras, USB-sticks etc.) would still be their own thing.

Then don't, sheesh. Why do you think that just because something COULD be done, it will be absolutely, positively crammed down your throat whether you want it or not?

Before getting huffy, read what I was responding to... ChrisA's comment was: "Finally Mac OS can get rid of those disk drive icons. There can just be 'storage'". My point was that I believe that's going too far. Sounds like you agree...

 

[Evangelion]

Before getting huffy, read what I was responding to... ChrisA's comment was: "Finally Mac OS can get rid of those disk drive icons. There can just be 'storage'". My point was that I believe that's going too far. Sounds like you agree...

Cameras, iPods and the like are not disk-drive icons. So we can (and should) get rid of those hard-drive icons, while still keeping cameras and iPods as separate.

 

[Analog Kid]

Cameras, iPods and the like are not disk-drive icons. So we can (and should) get rid of those hard-drive icons, while still keeping cameras and iPods as separate.

My camera attaches as mass-storage. How is the system to know it's not a USB drive?

As long as we want independent storage devices, we'll want icons to distinguish them.

 

[dnedved]

Confusing tripe article

This article left me with more questions than answers. Why does the author get LVM confused with the filesystem? Why does he spend so much time talking about spotlight performance, but there's no mention of a database driven filesystem like BeOS had a decade ago?

Are writers from USAToday switching over to write about filesystems now?!?

 

[Evangelion]

My camera attaches as mass-storage. How is the system to know it's not a USB drive?

trust me, the system will know. it might seem like mass-storage to the user, but the system knows what kind of hardware is connected to it. It really isn't that hard.

In Linux I can look at the data the OS has about various devices connected to it. And there's more than enough information in there to separate between different types of storage. And I don't think that things would be any worse in OS X.

As long as we want independent storage devices, we'll want icons to distinguish them.

And you would have them. And you could have them for hard-drives as well, if you so wish. So what's the problem here? ZFS would offer us a new and simplified way of managing our storage. Instead of having to think about different hard-drives, we would just have "storage". We wouldn't have to worry about having enough free space in different hard-drives, since the physical location of the files would be irrelevant. And yes, you could have separate storage-areas if you so wish.

 

[Evangelion]

This article left me with more questions than answers. Why does the author get LVM confused with the filesystem?

Because ZFS offers built-in LVM. So you get LWM as an added bonus with ZFS. Or as Sun says it:

Unlike traditional file systems that require a separate volume manager, Solaris ZFS introduces the integration of volume management functions. The traditional combination of file system and volume manager maintains one-to-one mapping between the file system and its associated volumes. Solaris ZFS breaks out of this limitation with the storage pool model. When capacity is no longer required by one file system in the pool, it becomes available to others.

 

[Analog Kid]

trust me, the system will know. it might seem like mass-storage to the user, but the system knows what kind of hardware is connected to it. It really isn't that hard.

In Linux I can look at the data the OS has about various devices connected to it. And there's more than enough information in there to separate between different types of storage. And I don't think that things would be any worse in OS X.



And you would have them. And you could have them for hard-drives as well, if you so wish. So what's the problem here? ZFS would offer us a new and simplified way of managing our storage. Instead of having to think about different hard-drives, we would just have "storage". We wouldn't have to worry about having enough free space in different hard-drives, since the physical location of the files would be irrelevant. And yes, you could have separate storage-areas if you so wish.

Most systems recognize cameras by the /DCIM directory at the root of the files system. This is also how the iPod is recognized as source for images. It's kind of moot though. I don't think you're proposing that each unique type of hardware that purports to be mass-storage should be uniquely identified.

Regardless, I see we're creating an argument where none exists... When you say "we can get rid of the hard drive icon" you apparently don't mean remove hard drive icons from the system.

The best solution I can envision is what was described earlier with different handling of inside the box and outside the box drives-- or just treat it like Disk Utility handles RAID drives and drag it into the pool.