Hacking Fusion to work on Mac Pro?

[dyn]

Originally Posted by goMac

Apple says 10.8.2.

That only applies to target disk mode. If you read the other parts of the document you'll see something about Disk Utility which clearly states that the current version of Disk Utility will not work because you need an adapted version. The support document has been updated on 31-10-2012 with more information. We now also know that the commandline version (dsutil) is able to setup Fusion Drives (that's what Jollyjinx used). The Fusion Drive support/implementation still is partial. With 10.8.3 there probably will be full support/implementation for it in OS X. We may not even need "hacking" (which I'm hoping).

[Loa]

Originally Posted by Concorde Rules

Well, clearly your setup works because you have 10% the library size of me!!

If I had a huge RAW library folder, I would do it manually like I used to when I had a 80GB SSD: import on SSD, move to RAW folder on HD when editing is done. Takes 2 seconds to move the folder from the SSD to the folder that has a link in my Finder windows. I'll personally take a few seconds of "manual work" over not knowing which of my older data is on the SSD.

Also, here's a heads up on some potential issues with large data and Fusion drive. It's only hypothetical, but I'm curious to see how it will turn out in practice.

http://www.zdnet.com/mac-fusion-driv...re-7000006661/

Loa

[avemestr]

Originally Posted by gnasher729

And last it calculate how often the file is read.

Please tell, how that is calculated the first time a file enters the system? The system can only count, and not initially anticipate, how often a file is read.

With the 4GB buffer and then spill-over, what will probably happen when one import 10GB of RAW pictures, is that 4 will end up on SSD and 6 on HDD.

Then you start working on them, and the system moves the files to the SSD again. Automatic, yes, but slower than if you were able to specify the SSD as target for all files from the start.

You then discard a quarter of the pictures, but the remaining 7.5GB are files you have been working heavily on for a few days. So Fusion decides to keep them on SSD for some time.

A customer from a few weeks back calls you, and ask you to re-visit some files you worked on a month ago. Those files are obviously on the HDD now. So you open them, and starts working on them. Unfortunately your SSD doesn't have enough room to have them moved over automatically, because the 7.5GB you just worked on seems much more "active" in the eyes of Fusion. Perhaps a day later, your old files is also moved to the SSD, after some different files have been moved to the HDD.

So now you have 10GB of space on the SSD taken up by pictures you're worked on the last couple of days, that you know you won't be revisiting in the near future.

Or, the alternative: Import the pictures to the SSD. Work on them. Move to HDD. If a customer calls, you decide if the specific request mandates a temporary move to SSD.

Fusion isn't magic. It can't tell "oh, gnasher just sat down in front of the computer, I better get those mp3s ready" or "oh, it's Concorde Rules, I know he'll want his RAW files ready. Oh, and of course I know he's going to work on the not 2 weeks old, not 4 weeks old, but the 3 weeks old RAW files today".

[dyn]

Originally Posted by Loa

Also, here's a heads up on some potential issues with large data and Fusion drive. It's only hypothetical, but I'm curious to see how it will turn out in practice.

http://www.zdnet.com/mac-fusion-driv...re-7000006661/

There are 2 problems with that story:

1. The assumption that FD moves entire files which Jollyjinx has proven to be untrue (it moves parts of files).
2. The assumption that FD moves files when you are doing work on that machine which is again proven to be not true by Jollyjinx (FD moves it when the system is idle, just like how Garbage Collection on SSDs work).

The last one isn't that far off though because even in idle time the system has to move parts of the file. What will happen when in that time you start working on the machine again? Since it moves only parts of files I don't think it will be that much of a problem.

There will probably be one bigger disadvantage with this system as opposed to running in ssd-only mode: the amount of writes will be much higher, especially if you have a mixed workload. It will continuously move files from and to the ssd. Somehow I think Apple knows this and has devised something that will take this into account (like only moving parts of files, these can be as small as optimal for an ssd).

Other things I'm wondering about: how will it be able to do GC when it is also moving files at idle time? Does this mean we now really need TRIM because there isn't much idle time for GC to do its magic? How will FD prevent the ssd from filling up completely in order to keep the performance up to par?

[deconstruct60]

Originally Posted by avemestr

Please tell, how that is calculated the first time a file enters the system? The system can only count, and not initially anticipate, how often a file is read.

That is likely why they went with it is both a split-speed logic volume and a cache approach instead of just one or the other.

With the 4GB buffer and then spill-over, what will probably happen when one import 10GB of RAW pictures, is that 4 will end up on SSD and 6 on HDD.

Not necessarily. If there is space on the SSD then files probably exit the write cache to that device first. Unless HFS+ has some bias against placing large files there to avoid fragmentation. Generally though it will be easier to place large files in a more wide open space like a large HDD.

Fusion isn't magic. It can't tell "oh, gnasher just sat down in front of the computer, I better get those mp3s ready" or "oh, it's Concorde Rules, I know he'll want his RAW files ready. Oh, and of course I know he's going to work on the not 2 weeks old, not 4 weeks old, but the 3 weeks old RAW files today".

Fusion is faster than a HDD all by itself. That is all it really has to be.
Apple's graphs on Fusion drive a relative a 1TB HHD drive.

" Aperture (Photo import) 3.5x
Copy File (Duplicate a 4GB folder) 3.5x
Boot System 1.7x "
http://www.apple.com/imac/performance/

They have conveniently selected write operations around the same size as the write buffer. :-) However, it isn't really a question of how much slower a 1TB SSD Fusion is. The question of whether are buying something close to a 500-1,000GB SDD performance at a far lower price is no. But it isn't being even being pitched that way. That's where many of these threads try to push it but it never was there in the first place.

That said. Even just splitting the OS/App workload from the serial sequential load of large files will cut down on the random accesses that the HDD is doing.

----------

Originally Posted by dyn

There are 2 problems with that story:

1. The assumption that FD moves entire files which Jollyjinx has proven to be untrue (it moves parts of files).
2. The assumption that FD moves files when you are doing work on that machine which is again proven to be not true by Jollyjinx (FD moves it when the system is idle, just like how Garbage Collection on SSDs work).

You are talking about the blog linked to the front page story?

http://jollyjinx.tumblr.com/post/346...ince-apple-has

That demos neither one of those.

[goMac]

Originally Posted by dyn

That only applies to target disk mode. If you read the other parts of the document you'll see something about Disk Utility which clearly states that the current version of Disk Utility will not work because you need an adapted version. The support document has been updated on 31-10-2012 with more information. We now also know that the commandline version (dsutil) is able to setup Fusion Drives (that's what Jollyjinx used). The Fusion Drive support/implementation still is partial. With 10.8.3 there probably will be full support/implementation for it in OS X. We may not even need "hacking" (which I'm hoping).

It looks like the Fusion Drive support that Jollyjinx used is actually complete.

As far as I can see it's entirely working. The new version of Disk Utility probably just includes new support for doing what the command line tools can already do. Disk Utility has always just been a wrapper over the command line tools.

There also isn't any hacking going on here. These command line tools are the official Apple tools for setting this up.

Jollyjinx doesn't seem to think Fusion Drive has specific files that it doesn't move out of the SSD cache. Although I'd assume if it did, there is just a command line option to pin those files to the SSD.

There are a people in this thread making kind of bizarre assumptions about how caching works. A write cache is only for holding files while the system decides what to do with them. They could end up on the SSD or the HD. Either way, keep in mind that at the same time things are being written into the write cache things are being moved to their final destination. This is much like the write cache when you burn a CD.

[avemestr]

Originally Posted by deconstruct60

Fusion is faster than a HDD all by itself. That is all it really has to be.

[...]
But it isn't being even being pitched that way. That's where many of these threads try to push it but it never was there in the first place.

I agree. Fusion is an improvement in iMac. It's faster than a traditional HDD for a lot of use cases.

But I read this thread as being about whether or not Fusion would be feasible and a good solution for Mac Pro users, and here I don't see much benefit.

[dyn]

Originally Posted by deconstruct60

You are talking about the blog linked to the front page story?

Yes.

That demos neither one of those.

That's because you didn't read the follow up on his blog Check out the following: More on BYO Fusion drive. And while you're at it, there's another follow up" Fusion Drive - loose ends.

Originally Posted by goMac

It looks like the Fusion Drive support that Jollyjinx used is actually complete.

As far as I can see it's entirely working. The new version of Disk Utility probably just includes new support for doing what the command line tools can already do. Disk Utility has always just been a wrapper over the command line tools.

The only thing he showed is that he could set it up and that it does what it says it does. What he didn't demo are the stability and how it works with things like Filevault2. In other words: it looks like there is full support in 10.8.2 but we are not sure. If you value Disk Utility a lot then clearly this implementation isn't complete as the current Disk Utility version doesn't work with FD.

There also isn't any hacking going on here. These command line tools are the official Apple tools for setting this up.

We've seen that with TRIM support as well. Then Apple came along and limited the support to their own SSDs. That's where the problem is, they might do that: limit FD to their own disks. Also, these tools are NOT the official Apple tools for setting this up. The functionality is stumbled upon (you could say researched) by Jollyjinx. Apple didn't tell anyone about it. If you read their support document you can see they are pushing Disk Utility and not the cli tools. Not very strange if you look at the target audience.

Jollyjinx doesn't seem to think Fusion Drive has specific files that it doesn't move out of the SSD cache. Although I'd assume if it did, there is just a command line option to pin those files to the SSD.

From what Jollyjinx is reporting this clearly is tiering and not caching because it moves files from 1 medium to the other and back. It is most likely that it does this by counting how many times you access something because you can't predict the future (you can make educated guesses). We just don't know how it does this and thus we do not know if there is any kind of mechanism to pin certain files/folders to the SSD/HDD. This might be one of the things that require Apple SSDs/HDDs. At the moment we just don't know.

[goMac]

Originally Posted by dyn

The only thing he showed is that he could set it up and that it does what it says it does. What he didn't demo are the stability and how it works with things like Filevault2. In other words: it looks like there is full support in 10.8.2 but we are not sure. If you value Disk Utility a lot then clearly this implementation isn't complete as the current Disk Utility version doesn't work with FD.

The version of Disk Utility doesn't matter. It may not support Fusion, but that doesn't mean Fusion isn't functional.

Edit: if you look at what he's doing, he's using the command line version of Disk Utility. This functionality is in Disk Utility, just in Apple's command line version.

Originally Posted by dyn

Also, these tools are NOT the official Apple tools for setting this up. The functionality is stumbled upon (you could say researched) by Jollyjinx. Apple didn't tell anyone about it.

Yes they did. Apple wrote the tools. Apple shipped them. Apple documented them. He's working from Apple documentation.

He even cites Apple documentation on his blog.

Just because it's not in Disk Utility doesn't mean Apple didn't make it or it doesn't exist.

Originally Posted by dyn

From what Jollyjinx is reporting this clearly is tiering and not caching because it moves files from 1 medium to the other and back. It is most likely that it does this by counting how many times you access something because you can't predict the future (you can make educated guesses). We just don't know how it does this and thus we do not know if there is any kind of mechanism to pin certain files/folders to the SSD/HDD. This might be one of the things that require Apple SSDs/HDDs. At the moment we just don't know.

Fusion Drive is tiering (not that there is much of a distinction), so I'm not sure what your point is. Apples documentation for Fusion says exactly what you just described. It moves recently/most used files to the SSD.

[dyn]

Originally Posted by goMac

The version of Disk Utility doesn't matter. It may not support Fusion, but that doesn't mean Fusion isn't functional.

I never said that

Edit: if you look at what he's doing, he's using the command line version of Disk Utility. This functionality is in Disk Utility, just in Apple's command line version.

Not very wise to say it like that, clearly there is a difference between diskutil (the cli version) and Disk Utility (the GUI version) which is probably also why they don't have the same name ("but there is a space in Disk Utility which doesn't work in the cli!" as if Disk Utility is the only name for a tool like this ).

Yes they did. Apple wrote the tools. Apple shipped them. Apple documented them.

It doesn't matter if Apple wrote the tools and shipped them because there is this thing called "unsupported feature" (aka "undocumented feature"). It might be there now but it does not mean it will be there in the future. Biiiiiiig difference. Apple also did not document this feature at all.

He's working from Apple documentation.

He even cites Apple documentation on his blog.

The documentation he is using and citing from is the support document that was posted here earlier. That document has been modified on 31-10-2012 and does not mention the cli tools at all. The only thing that it mentions is Disk Utility which is the GUI tool.

He didn't create a Fusion Drive though. He created 1 logical volume with 2 physical disks and then put a 500GB HFS+ partition on it (he later on changed it to a ZFS partition). It is the OS that automatically created the Fusion Drive because it somehow recognised one was an ssd (probably via SMART, maybe via something similar as what Windows 7 does: measuring throughput) and the other an hdd. Creating volumes was already possible when CoreStorage was introduced in Lion. There is nothing new there. What is new, is that the OS itself creates the Fusion Drive when you put an ssd and hdd in the same logical volume. It is not Disk Utility nor diskutil that creates them. These tools only create logical volumes and such. Apparently Disk Utility hasn't got a full LV implementation. I'm guessing that Apple only adjusted Disk Utility so it can display Filevault2 disks and not any other kind of LV configuration.

Just because it's not in Disk Utility doesn't mean Apple didn't make it or it doesn't exist.

Just because Apple wrote and distributes the tools and created a support document on Fusion Drive doesn't mean that this is a supported and documented feature. You can backup TM to a network share on non-OS X devices but this isn't documented nor is it supported.

Fusion Drive is tiering (not that there is much of a distinction), so I'm not sure what your point is. Apples documentation for Fusion says exactly what you just described. It moves recently/most used files to the SSD.

There is quite a big distinction: cache is a copy, tiering is a proper move of data. That's not my actual point. I was aiming at the system that decides when something should be copied to ssd and when to hdd. It does not copy applications and the OS X base system, it keeps it on the ssd. Now how does it do all these things? Apples support document says absolutely nothing about these things and they probably won't ever because this is something that any company would like to keep secret since it is very valuable information for competitors. The same with things like GC in an ssd.

[goMac]

Originally Posted by dyn

Not very wise to say it like that, clearly there is a difference between diskutil (the cli version) and Disk Utility (the GUI version) which is probably also why they don't have the same name ("but there is a space in Disk Utility which doesn't work in the cli!" as if Disk Utility is the only name for a tool like this ).

There isn't. The GUI version of Disk Utility is just a covering of the cli version. They're the same thing.

Originally Posted by dyn

It doesn't matter if Apple wrote the tools and shipped them because there is this thing called "unsupported feature" (aka "undocumented feature"). It might be there now but it does not mean it will be there in the future. Biiiiiiig difference. Apple also did not document this feature at all.

They documented it with the Disk Utility CLI.

Originally Posted by dyn

The documentation he is using and citing from is the support document that was posted here earlier. That document has been modified on 31-10-2012 and does not mention the cli tools at all. The only thing that it mentions is Disk Utility which is the GUI tool.

Not the right document.

Originally Posted by dyn

He didn't create a Fusion Drive though. He created 1 logical volume with 2 physical disks and then put a 500GB HFS+ partition on it (he later on changed it to a ZFS partition).

That's exactly what a Fusion Drive is. A CoreStorage single logical volume consisting of an SSD and a hard drive.

Originally Posted by dyn

Just because Apple wrote and distributes the tools and created a support document on Fusion Drive doesn't mean that this is a supported and documented feature. You can backup TM to a network share on non-OS X devices but this isn't documented nor is it supported.

They released it and documented it in CoreStorage. It's supported.

Originally Posted by dyn

There is quite a big distinction: cache is a copy, tiering is a proper move of data. That's not my actual point. I was aiming at the system that decides when something should be copied to ssd and when to hdd. It does not copy applications and the OS X base system, it keeps it on the ssd. Now how does it do all these things? Apples support document says absolutely nothing about these things and they probably won't ever because this is something that any company would like to keep secret since it is very valuable information for competitors. The same with things like GC in an ssd.

Well, first, caching is not necessarily copying. But letting that point go...

Fusion Drive as described by Apple is "tiering", the blog post shows something that exactly matches the functionality of Fusion Drive. Data is kept exclusively on the SSD until it needs to be sent bad to the drive.

In addition, he's talked about how the base system and applications are likely kept on the SSD. The first things installed on the drive live on the SSD. So when applications and OS are installed at the factory, they're live on the SSD. If the user never uses them, they are eventually pushed to the HD.

[derbothaus]

When you press buttons in Disk Utility you are leveraging diskutil. The gui is a shortcut. They are not separate only in what is hosting the command that kicks it off. Pretty much the entire functionality + some can be done in a shell for all of OS X. diskutil is so much better than the GUI.app. I say plus because that is where new tech emerges before Apple paints a GUI and calls it something flashy like "Fusion". The shell is usually more stable as well. Just nerdy. So tired of the word Fusion. Vmware, Extensis, now this... I am on a Fusion branding strike.

[dyn]

Originally Posted by goMac

There isn't. The GUI version of Disk Utility is just a covering of the cli version. They're the same thing.

Ah so you are now implying that Apple has put up false information in their own support article? Do think before you respond... No, both diskutil and Disk Utility are different applications because they differ in functionality.

They documented it with the Disk Utility CLI.

If you do a man diskutil and search for fusion it comes up with nothing... What they documented is the various parameters for diskutil. One of them is simply creating an LVG.

Not the right document.

It is the only document he is linking and referring to...

That's exactly what a Fusion Drive is. A CoreStorage single logical volume consisting of an SSD and a hard drive.

Then you clearly have no understanding of what Fusion Drive is. Having an LVG of an ssd and hdd is only part of the story. Having an LVG does nothing more than pairing 2 disk and making it look like 1. There has to be a mechanism that calculates how much a block of data is being used and when to move it from one disk to another. It requires far more than just an LVG.

They released it and documented it in CoreStorage. It's supported.

There is no reference to Fusion Drive there.

Well, first, caching is not necessarily copying. But letting that point go...

Wise thing as the only definition one will be able to find is that of copying data.

In addition, he's talked about how the base system and applications are likely kept on the SSD. The first things installed on the drive live on the SSD. So when applications and OS are installed at the factory, they're live on the SSD. If the user never uses them, they are eventually pushed to the HD.

Except for the fact that this isn't anywhere on his blog... He is not testing with apps or the OS, he's merely testing with directories and files he created himself.

Since you haven't read the blog and you consistently keep refusing to actually read it combined with the fact that you have no clear knowledge about the matter I'm not even going to bother responding to you any further. The blogs, documents, applicaties and manuals of the applications speak for themselves. They do not support your statements above at all.

[derbothaus]

Originally Posted by dyn

No, both diskutil and Disk Utility are different applications because they differ in functionality.

How so? What can't you do in diskutil that you can do in the GUI? I couldn't find 1 thing unless Icon view and window option is counted.
It was my understanding they differ only because Apple didn't want to make certain functions available for everyone in the app while other things are rudimentary implementations of new stuff. Like any SW it is cut and branched but is one string of check ins.

[the kid 05]

Originally Posted by JesterJJZ

Yes, because you really want to trust your data to a hacked split volume solution that just came out.

Its been out a while now.

http://en.wikipedia.org/wiki/Hybrid_drive

[derbothaus]

Originally Posted by the kid 05

Its been out a while now.

http://en.wikipedia.org/wiki/Hybrid_drive

Not the same thing. Read this thread.
A bit more if you have not been watching.
http://arstechnica.com/apple/2012/11...ons-confirmed/

[derbothaus]

They ripped one apart:
http://arstechnica.com/apple/2012/11...-fusion-drive/

[Concorde Rules]

Originally Posted by derbothaus

They ripped one apart:
http://arstechnica.com/apple/2012/11...-fusion-drive/

Well that works just how I want it to.

Excellent!!

[viktorcode]

I created a fusion drive on my Mac Pro 2008. Disks used: 240 GB SSD and 750 GB HDD. So far it works well.

One thing I did differently is I never reinstalled OS on fusion drive, opting for cloning to an external drive instead, fusing internal drives from there, and cloning the OS back. Judging by the first time launch speed OS X did land on SSD during cloning.

For those of you who are wondering is it necessary to have special Disk Utility from newest Macs for compatibility, the answer is no. Current Disk Utility in 10.8.2 supports all you need for fusion (I'm talking about its command line variant, diskutil), and I presume the only special thing in newer Disk Utility is that it may support creating fusion drives from GUI.

[lssmit02]

One aspect of a Fusion Drive that I am tempted by is that it allows you to take full advantage of your SSD, without having to manually move things around. The system fills up the SSD (leaving 4GB free), ensuring that I am more likely to have all of the data that I access frequently located on the SSD. Right now, I keep my music, images and videos on an HDD, and the system, apps and other documents on my SSD. This means that many GBs of data is on the slower drive, because it would take up too much space on my SSD. While I could split up my images into multiple Aperture files, keeping only the most recent pictures in an image file on the SSD, this does make using Aperture more inconvenient (and slower) when I want to access older images. Since the Fusion Drive moves chunks of data, not files, this means that if I start a project using older images, they will be moved to the SSD by the system, improving performance with those images that I do happen to access more frequently. This would be true of my music, etc. too. Plus, all of the older documents that I don't access often will be shifted to the HDD, freeing up even more space on the SSD. In short, I get to take full advantage of the money spent on the SSD.

[goMac]

Strongly considering moving from RAID to Fusion. The Samsung SSDs are so cheap right now. The only thing holding me back is that I don't think only one drive from my existing RAID + SSD would have enough room for my stuff, so I'd have to invest in a larger hard drive as well, which might just push me over the line for what I want to put into my 08 Mac Pro that I'd like to replace next year.

[Concorde Rules]

I tried it out on my Mac Pro this week and it was never transferring the files to the HDD so it would write ~500MB to the SSD then the rest to the HDD.

So until someone works out how to set fusion to keep more space free, I think I'm out