ZFS in Latest Leopard Builds

[mccoy88f]

I think that today is possible to include ZFS in Leopard... i explain why

Porting of ZFS for FreeBSD is near complete:

Pawel Jakub Dawidek is manteiner of this porting, in a recent post he write:
"I've almost all file system functions working.
I started to run some heavy file system regression tests. They work."
this from http://www.opensolaris.org/jive/thread.jspa?threadID=12766&tstart=0

We also knew that zfs can now boot from x86 ZFS machine
http://opensolaris.org/os/project/zfsboot/
Using EFI booting for boot from ZFS i thinks i more simple...

ZFS respect POSIX standard that explain that appz don't need to edit to func with ZFS.

What does you think?

 

[Evangelion]

Most systems recognize cameras by the /DCIM directory at the root of the files system. This is also how the iPod is recognized as source for images. It's kind of moot though. I don't think you're proposing that each unique type of hardware that purports to be mass-storage should be uniquely identified.

Each piece of hardware IS uniquely identified. At least in Linux each piece of hardware is differentiated by their type, manufacturer and model. And each piece of hardware has an individual identification-string, so they are all individually identified. If I wanted to, I could tell the system that "when a device with this particular identification-string is attached to the system, I want you to do this". Naturally most users don't do that, since the system does it all automagically. But it could be done. I have written such rules back in the day .

I'm not really sure what you are trying to say here. that the system does not know what pieces of hardware are connected to it? Trust me, it does know. If it doesn't, then it's broken IMO.

Regardless, I see we're creating an argument where none exists... When you say "we can get rid of the hard drive icon" you apparently don't mean remove hard drive icons from the system.

Of course there should be some kind of icon to access the filesystem. But there would be no need to separate each physical hard-drive in to separate icons, since they could all be part of one storage-system. There could be such separate icons if the user wants to, but it wouldn't be mandatory anymore.

I don't really see that much difference between the way OS X handles hard-drive volumes when compared to Windows. In Windows I have C:, D: and so forth. In OS X I have "Serenity" (the HD in my Mac Mini), "Media" (my first external HD) and "Data" (my second external hard-drive). It would be the same regardless of whether those drives were internal or external. The names are different, but there's not much difference between Windows and OS X.

With ZFS we would just have "Storage". And if that storage starts to get full, you could add another HD to the system, which would then be added to the Storage. If that happens today, I would have to buy a bigger HD, move the data to the new HD, or split it between the two hard-drives.

The best solution I can envision is what was described earlier with different handling of inside the box and outside the box drives

You could do that is you wanted to. But ZFS COULD handle those as one coherent whole. Hell, it could use network-drives as part of the storage-pool.

 

[Analog Kid]

Each piece of hardware IS uniquely identified. At least in Linux each piece of hardware is differentiated by their type, manufacturer and model. And each piece of hardware has an individual identification-string, so they are all individually identified. If I wanted to, I could tell the system that "when a device with this particular identification-string is attached to the system, I want you to do this". Naturally most users don't do that, since the system does it all automagically. But it could be done. I have written such rules back in the day .

I'm not really sure what you are trying to say here. that the system does not know what pieces of hardware are connected to it? Trust me, it does know. If it doesn't, then it's broken IMO.

I haven't done Linux system hacking myself, but I've worked with Windows enough to know what happens if you try to make decisions on too fine grained of a level. Generally that leads to needing drivers for every specific implementation, whereas the OS X model is to rely on the standards to abstract device classes.

I've never had to install a device driver for a camera, and I know it's not because OS X recognizes each and every camera. They all mount as "disks" (evidenced by the disk icon in Finder). Higher level system services rely on the fact that standard camera media places the /DCIM directory at the root and launch an application if I've set it up to do so, but cameras are treated as a special case in this regard.

If I pull the media from my camera and put it into a card reader, the Mac still treats it as an image source because it's a disk with the key directory, not because it recognizes each unique type of memory card. My card reader didn't require special drivers even though it's an off brand that Apple certainly didn't write a driver for itself.

If I connect a piece of custom hardware we've developed, it opens as mass storage with no special directory to key off of. We didn't design it to be Mac compatible, but it is because it conforms to the Mass Storage standard spec for USB. All of the files I've collected on the device are there for transfer. This is good general behavior and the Mac isn't broken for not knowing I've attached some unknown piece of hardware.

Of course there should be some kind of icon to access the filesystem. But there would be no need to separate each physical hard-drive in to separate icons, since they could all be part of one storage-system. There could be such separate icons if the user wants to, but it wouldn't be mandatory anymore.

...

With ZFS we would just have "Storage". And if that storage starts to get full, you could add another HD to the system, which would then be added to the Storage. If that happens today, I would have to buy a bigger HD, move the data to the new HD, or split it between the two hard-drives.

You could do that is you wanted to. But ZFS COULD handle those as one coherent whole. Hell, it could use network-drives as part of the storage-pool.

Yes. I agree with you. As I said, the pool could be handled in the same way Disk Utility handles RAID drives-- one icon for the entire RAID. Drives aren't subsumed into the RAID just by being attached, however-- I need to do that intentionally. Yes, I know you can't just add another drive to an existing RAID and that the pool has all kinds of additional advantages, I'm only using the RAID example as a, well... example.

The only point I've been trying to make is that I don't want ZFS making these decisions for me, and setting it up to do so would be dangerous. Nothing deep or terribly meaningful here...

 

[xUKHCx]

I don't really see that much difference between the way OS X handles hard-drive volumes when compared to Windows. In Windows I have C:, D: and so forth. In OS X I have "Serenity" (the HD in my Mac Mini), "Media" (my first external HD) and "Data" (my second external hard-drive). It would be the same regardless of whether those drives were internal or external. The names are different, but there's not much difference between Windows and OS X.

There is a big difference and it is extremely annoying. If you have links to a file that is on the external drive. In Windows when you attach drive "Serenity" there is no guarantee what letter that drive will be assigned. However on a mac it just doesnt matter.

Which does annoy the hell out of me, for example on my USB stick i have a few portable apps, Firefox etc. etc, they create their own folders so i hide them inside a hidden folder. I have links to the applications so i can run them easily. However depending on what other drive is attached these links will not work. Which is shocking because the links are on the drive linking to other files on the drive, this is a pretty serious problem in my view.

 

[Bonte]

Until they start putting multiple hard drives in a MacBookPro, ZFS is primarily a Workstation or Server feature.

And yes, that is still a fantastic addtion to OS X.

It also compresses data in realtime and saves HD space with a large number of small files (for example osX itself has thousands of ultra small txt files), i'm sure this will be beneficial for small HD's in laptops, mini's and iTV's.

 

[Evangelion]

I haven't done Linux system hacking myself, but I've worked with Windows enough to know what happens if you try to make decisions on too fine grained of a level.

We are talking about a difference between a hard-drive and a camera here. I don't think that's "too fine grained" .

Generally that leads to needing drivers for every specific implementation

um, I'm not talking about drivers. Just because each piece of hardware is uniquely identified does not mean that each of them needs specific drivers. It just means that the OS knows what is connected to the system, and where. In Linux you could even see where a USB-stick is connected. In case of Apple Keyboard, it would basically tell that "USB stick is connected to Port 2 on Apple Keyboard, which in turn is connected to Port 3 on the computer". And the system would recognize one USB-stick from another, but it wouldn't need specific drivers for each.

The unique identification I'm talking about simply means that the system can tell different pieces of hardware from each other, that's all. And it can do that even if they are identical make and model. There's no reason why OS X couldn't do that. And that unique identifier would not mean that they would need specific drivers. It could just mean that the printer on the left side of the computer is different printer than the identical printer on the right side of the computer.

whereas the OS X model is to rely on the standards to abstract device classes.

As does Linux. But that doesn't mean that each piece of hardware couldn't be uniquely identified. I really have the feeling here that you have misunderstood what I have been saying here.

I've never had to install a device driver for a camera, and I know it's not because OS X recognizes each and every camera. They all mount as "disks" (evidenced by the disk icon in Finder).

yes they do, but do you REALLY think that when you plug in your camera, the only thing OS X knows is that "someone plugged in a disk"? No, it knows a lot more. For starters, when I plug in my camera, OS X somehow knows that it needs to start iPhoto. It doesn't start iPhoto when I plug in my USB-stick, so OS X can clearly tell the two apart.

The system knows quite a bit about the hardware that is plugged in. It just doesn't pass that information to the user because the user does not need to know it. I would think that OS X knows as much as Linux does. Difference is that Linux has tools that can extract that information and present it to the user if he wants it. I have looked at the info Linux has, and it's more than enough to differentiate between different devices. And thanks to that differentiation, there is no risk that your digital camera would get added to your logical volume

 

[Analog Kid]

We are talking about a difference between a hard-drive and a camera here. I don't think that's "too fine grained" .

It would be if you used my other example of custom hardware...

The unique identification I'm talking about simply means that the system can tell different pieces of hardware from each other, that's all. And it can do that even if they are identical make and model. There's no reason why OS X couldn't do that. And that unique identifier would not mean that they would need specific drivers. It could just mean that the printer on the left side of the computer is different printer than the identical printer on the right side of the computer.

OS X can. Go to Apple->About this Mac->More Info and you'll see a very similar set of information.

As does Linux. But that doesn't mean that each piece of hardware couldn't be uniquely identified. I really have the feeling here that you have misunderstood what I have been saying here.

I must be... And we've gotten into this whole side discussion about cameras which were merely an example of my point because most people have one. I'm not saying OS X can't tell one piece of hardware from another, I'm saying I don't want my file system to make it's own decisions about what to add to a storage pool.

I'm saying there are many, many types of devices that act as disks but that shouldn't be rolled in with all the other storage on my system. The system can't possibly be smart enough to know what I intend to with what I just attached.

yes they do, but do you REALLY think that when you plug in your camera, the only thing OS X knows is that "someone plugged in a disk"? No, it knows a lot more. For starters, when I plug in my camera, OS X somehow knows that it needs to start iPhoto. It doesn't start iPhoto when I plug in my USB-stick, so OS X can clearly tell the two apart.

It's too late for me to try this experiment myself, but if you're curious let me suggest it: take your USB stick, format it as FAT, copy the DCIM directory from your camera to the root of the USB stick. Remove the stick and reinsert it. I'm guessing iPhoto recognizes it as an image source. Maybe I'm wrong, and maybe OS X is recognizing that my USB drive is really an xD card wrapped into a USB stick, but it doesn't change the fact that adding a step between insertion of a drive and adding it to the pool is a smarter, safer thing to do.

Uncle.

 

[Evangelion]

OS X can. Go to Apple->About this Mac->More Info and you'll see a very similar set of information.

And I did say that I expected as much. So what exactly is the problem here? OS X already does what I have been talking about, and we (obviously) do not have the problems you mentioned.

I'm not saying OS X can't tell one piece of hardware from another, I'm saying I don't want my file system to make it's own decisions about what to add to a storage pool.

And you could do just that. And I think that even if the OS handled all this automagically, it would be smart enough to NOT add cameras and iPods to the storage-pool.

I'm saying there are many, many types of devices that act as disks but that shouldn't be rolled in with all the other storage on my system.

And they wouldn't be.

The system can't possibly be smart enough to know what I intend to with what I just attached.

But it would be smart enough to not add your iPod to your storage-pool . Thats the point. And, of course, you could still decide what to do with each individual component. If you want your Firewire-HD to be added to the pool, you could do so. Or you might want to keep it separate, And you could do so.

I'm not really seeing any issues here.

 

[4np]

Could this mean revolutionary things ahead for the next OS? Perhaps better integration with new hardware?

Leopard's Time Machine is a front end to ZFS's snapshots...

 

[Terranova]

You can do that now. It uses LVM, you can sync partitions without reboot, just use partprobe, pvcreate to make the partitions to volumes, suck them all together with vgcreate and use lvextend to make them bigger, hfsonline to make it. You can grow a drive to whatever you want.

You can also use pvmove, vgreduce, pgremove, if a drive is about to fail and you have another in sync as a mirror and bring it online.

Great! sounds easy! I'll try it after West Wing tonight.

 

[Morris]

As said by others, ZFS can have tremendous benefits, even for regular users with just one harddrive.

Granted, most of the benefits ZFS has over HFS+ are things you don't need until you start using more storage. Still, among other things, home users can greatly benefit from the added reliability and handling of the fast growing amount of very small files.

Regarding reliability, picture this. A friend of mine had very weird things going on with his Powerbook. Seemingly random kernel panics, crashes of software, difficulty booting. A re-install of Tiger would help but only for two weeks at most. After removing part of his memory to rule out memory failures it turned out to be his drive silently 'losing bits'. Sometimes the losses were in parts of no importance to the OS, like pictures or music. However, whenever a few bits or bytes in important files got messed up the trouble began.
This silent deterioration of your data is impossible with ZFS. At worst you'd be immediately warned and have time to create backups and get another drive. At best (when using more than one drive in some kind of pool) data will be automatically kept healthy.

Regarding small files, think of Spotlight. With more and more apps getting developed with Spotlight in mind, the number of small files increases significantly because Spotlight likes every chunk of data to have it's own file. Mail.app switched from a few big files storing all your e-mail to one file per e-mail when Tiger came out. In my situation that meant over 30.000 tiny files instead of a few big ones!
Most file systems have trouble with small files in relation to efficiency resulting in them taking up much more space than needed or slowing down. Not with ZFS.

Hhmm... I should look for a job in marketing... any tech marketing jobs in London?

 

[wrldwzrd89]

ZFS sounds so much like the wave of the future for Mac OS X. I agree with all the other points that have been raised, namely:

• Letting the user decide what goes in the ZFS pool
• Vista has officially been dwarfed
• Time Machine will benefit hugely from ZFS
• Will Apple make ZFS bootable?

I'd love to see ZFS offered as an option in Leopard - even if it's not the default, I'd use it just for the coolness factor

 

[xUKHCx]

I'd love to see ZFS offered as an option in Leopard - even if it's not the default, I'd use it just for the coolness factor

Same here

 

[jholzner]

Leopard's Time Machine is a front end to ZFS's snapshots...

Wow, this rumor dies hard huh? Tme Machine has nothing to do with ZFS in anyway. I thinks it's been mentioned in this thread even.

http://storagemojo.com/?p=213

 

[Digitalclips]

This sounds very interesting, always good to be able to use the latest new tech.

From the Wikipedia article: "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans."

I wonder how long until we hit that limit

I think they say it would take more energy to polulate it than it would to boil the oceans, not that you have to actualy boil the oceans to populate it.

 

[Digitalclips]

ZFS sounds so much like the wave of the future for Mac OS X. I agree with all the other points that have been raised, namely:

• Letting the user decide what goes in the ZFS pool
• Vista has officially been dwarfed
• Time Machine will benefit hugely from ZFS
• Will Apple make ZFS bootable?

I'd love to see ZFS offered as an option in Leopard - even if it's not the default, I'd use it just for the coolness factor

And in no time at all we will all accept is as standard on Macs and talk fondly of the 'old HFS days', meanwhile Vista is based on what? It isn't still underpinned by DOS from the 1970's is it?

 

[Trekkie]

You are talking about RAID here. You don't nbeed ZFS for RAID.

So if I have two 250GB Drives in my ZFS files system, and 400GB of data, and one 250GB drive goes the way of the dodo I don't need RAID?

That's quite the magical file system.

 

[Rocketman]

Yes, that certainly explains why Time Machine exists in the current developer's builds that are running on HFS+

ZFS would make a great basis for an implementation of Time Machine (read as: where we are going) but the program itself will not require a ZFS file system, it will be compatible with HFS+ for current users' sake.

I'm hoping for an HFS/HFS+ type transistion: ZFS will be in OS 10.5, you can choose to reformat your hard drive and start using ZFS for your boot partition, or keep using HFS+ for now. If you do change to ZFS you'll get better Time Machine performance, the pooled storage, ect.

Then at some point in the future (10.6, 10.7?) they will drop the HFS+ and you'll have to use a ZFS file system to upgrade.

Everyone keeps talking about how ZFS booting is not ready yet. Silly users, ZFS is Open-Source, is it not possible Apple has been working on booting ability on their own and have solved the issue for 10.5?

One thing that will be a problem in the beginning is there will be no DiskWarrior, ect to repair drives with. But given the features of ZFS, I wonder what use there would be for diagnostic programs to begin with.

I hear you.

Apple licensed ZFS from whoever, and then started adding features of that system to both HFS+ version of the OS and linking the OS to use ZFS directly. Perhaps there are patented aspects of ZFS Apple had to license to use within HFS+ for commercial resale?

Perhaps in 10.5 TM will use a live size-changing ZFS partition to implement TM features. And I ask, why NOT have two drives in a laptop? The second one can be reallly tiny.

As I have been saying for years now: RAID everywhere.

Rocketman

 

[wrldwzrd89]

And in no time at all we will all accept is as standard on Macs and talk fondly of the 'old HFS days', meanwhile Vista is based on what? It isn't still underpinned by DOS from the 1970's is it?

Based on what I hear from Microsoft, there are still 2 vestiges of DOS left behind in today's Windows (Vista):

1. cmd.exe - the command line
2. Drive letters

Cmd.exe was supposed to go away with the new UNIX-like command line called Monad, which was pulled at the last minute for undisclosed reasons.

As for drive letters, rumors placed their demise with Windows Vista as well, but this too was pulled, presumably for the sake of backward compatibility when Vista was restarted and based on the Windows Server 2003 codebase early in its development.

 

[Westside guy]

We are not talking about RAID here. I think the word you are looking for is "Logical Volume Management"

If that's was was being referred to, then the post I was responding to was incorrect. LVM needs to be set up on Linux or BSD (or whatever) - it's not how disk management is done by default on other Unixes. And there are Windows-based drive-spanning solutions as well. There have also been drive-vendor-specific offerings that do this (clunkily).

Cameras, iPods and the like are not disk-drive icons. So we can (and should) get rid of those hard-drive icons, while still keeping cameras and iPods as separate.

Not to mention that it's unlikely cameras would be anything other than FAT16/FAT32.

I originally was going to say "and iPods", but then I realized that might not be a given on the Mac end.

If I wanted to, I could tell the system that "when a device with this particular identification-string is attached to the system, I want you to do this". Naturally most users don't do that, since the system does it all automagically. But it could be done. I have written such rules back in the day .

Hehe, it wasn't very long ago that you HAD to do exactly this under Linux for a lot of devices (which I'm guessing is exactly what you're referring to).

 

[MacVault]

This article left me with more questions than answers. Why does the author get LVM confused with the filesystem? Why does he spend so much time talking about spotlight performance, but there's no mention of a database driven filesystem like BeOS had a decade ago?

Are writers from USAToday switching over to write about filesystems now?!?

Yea, ever since this ZFS in Leopard news surfaced I've been wondering how it compares to the database-driven filesystem of BeOS that was talked about in the past few years as being the holy grail of filesystems. Would someone enlighten us as to whether ZFS offers the same technologies and advantages as the BeOS filesystem??? Oh, and what about the WinFS (vaporware concept? How does ZFS compare to these "database-metadata-driven" filesystems?

 

[shawnce]

I hear you.

Apple licensed ZFS from whoever, and then started adding features of that system to both HFS+ version of the OS and linking the OS to use ZFS directly. Perhaps there are patented aspects of ZFS Apple had to license to use within HFS+ for commercial resale?

Lets not make stuff up.... HFS+ has had the capability that Time Machine leverages since HFS+ was integrated with Mac OS X. Nothing was borrowed from ZFS for this, in fact a feature that ZFS doesn't yet have was added to HFS+ in 10.5. Also ZFS is available via the CDDL open source license (thanks to Sun for making it available).

 

[MacVault]

This image showing ZFS in what looks like Disk Utility appears to be slightly transparent. If you look closely there are some bullet points. I just opened Disk Utility (not on Leopard) and don't see any transparency in it. What's the explanation for this??? Is Leopard's Disk Utility transparent? Why do I see text in the background? Or is this just some bad photoshop job?

 

[Morris]

This image showing ZFS in what looks like Disk Utility appears to be slightly transparent. If you look closely there are some bullet points. I just opened Disk Utility (not on Leopard) and don't see any transparency in it. What's the explanation for this??? Is Leopard's Disk Utility transparent? Why do I see text in the background? Or is this just some bad photoshop job?

When I look closely I can read: "To repair disk permission problems..." and "To repair yo...... choose Utilities > Disk Utilities". That seems to me genuine content to appear on a preceding screen. There you'd select to create a volume and then this screen would pop up. Apparently slightly transparent.

You'd have to be a very cunning PS con man to decide to make up a completely new window and options and to show it with just 10% transparency in the background.

 

[Peace]

This image showing ZFS in what looks like Disk Utility appears to be slightly transparent. If you look closely there are some bullet points. I just opened Disk Utility (not on Leopard) and don't see any transparency in it. What's the explanation for this??? Is Leopard's Disk Utility transparent? Why do I see text in the background? Or is this just some bad photoshop job?

It's real.