Coming Soon: Mac Server Clusters

[soilchmst]

Server clustering is coming for OSX Server. Apple appears to be making a complete server alternative, with xServe, xRaid, and soon, server clustering in the easiest to use server software on the market :cool:

http://www.thinksecret.com/news/osxclustering.html

[bidge]

i was just talking about this with a friend the other day, he said that OS X is the only UNIX based OS that doesn't do this? Is this bull or is it true?

[sparkleytone]

now this is something i hope is true. apple could really pull a fast one on everyone if they can come up with a classic apple solution to clustering. the ease of use and capabilities in an implementation of this could mean wonders.

[vniow]

Mmmmmm.......Final Cut Pro rendering farms.:D
I wonder if this would only be for the server OSX, hell, I'd love to get two Macs together to do major media editing.:)

[bidge]

Or if they used rendesvouz so that they automatically discovered each other you could go into an Apple store and have all of the computers clustered.

Awesome rendering capabilities. You could be working on your powerbook while your dualie's rendering.

Cool

[oldMac]

Software exists to cluster almost any type of Unix, NT or even Mac OS 9 servers.

The hard part is:

1) Making sure that the software you intend to cluster runs on your hardware.
- It's just like any other software, it has to be compiled specifically for the hardware you intend to run it on.
- It's much easier to port Unix software to the Mac now that we've got OS X, so there is a lot more clustering potential for the Mac OS now.

2) Setting up clustering can be a pain
- Generally this process isn't pretty in existing solutions
- Apple could utilize Rendevous to make this a lot easier
- Even just throwing a nice GUI on the setup process would be a huge help

It would be interesting to see if Apple provides a way to automatically distribute the software that you intend to cluster, as well. Imagine an interface that looks like this...

1) You open the "Clustering" control panel
2) Rendevous automatically discovers the local machines available for clustering
3) You can select/deselect the machines available from the list that's displayed
4) You drag your application into the control panel
5) Click "Start"


Since Apple built the XServe for clustering, it really makes sense that they would be getting ready to bundle nice software for it. This is completely supported by including Rendevous in OS 10.2

[nuckinfutz]

Originally posted by bidge
i was just talking about this with a friend the other day, he said that OS X is the only UNIX based OS that doesn't do this? Is this bull or is it true?

Ahhh I don't think he's thinking this through. This is about Clustering at the "Kernel" level. I doubt that this is a standard item of all Unices.

[MisterMe]

Originally posted by bidge
i was just talking about this with a friend the other day, he said that OS X is the only UNIX based OS that doesn't do this? Is this bull or is it true?
Your friend is talking through his hat. UNIX is clustered using a tool called MPI, available free of charge from the University of Michigan, IIRC. If all versions of UNIX included clusering as a standard feature, then MPI would never have been developed. BTW, there is a MacOS 9 version of MPI called MacMPI.

[Chisholm]

It seems to me the xServe was released a little prematurely. The very nature of its design (case wise) seems like clustering would be almost an assumed component of the OS.

"Harness all of the power (almost, well kinda' if you look at it from this angle under this lighting, etc.) of Unix"

[nuckinfutz]

Originally posted by Chisholm
It seems to me the xServe was released a little prematurely. The very nature of its design (case wise) seems like clustering would be almost an assumed component of the OS.

"Harness all of the power (almost, well kinda' if you look at it from this angle under this lighting, etc.) of Unix"

Not really. There's Firewire although I doubt it'd be used for clustering. Xserves have PCI and that would probably be your link to PCI Fibre Channel based connections.

[arn]

Originally posted by Chisholm
It seems to me the xServe was released a little prematurely. The very nature of its design (case wise) seems like clustering would be almost an assumed component of the OS.

It can cluster... there is software for it. It's simply not built into the kernel.

arn

[shadowfax0]

Everyone go look up 'Project Appleseed' in Google, RIGHT NOW. I made a cluster between my G4 tower and a TiBook to render a Mathematica problem...and it still took a week! Anyhow, I suggest everyone go look fo rit, becuase just a few powermacs can equal the power of a Cray T3E.

[shadowfax0]

Also, this is a bit of a side note, but I think it applies. All you MacRumors folks who live in England, go to a magazine shop, and get the mag 3D World, they have an article about making a 'render farm' by utilizing Maya, Lightwave, or Cinema 4D, very helpful tutorial, and I know alot you folks are looking for it!

[P-Worm]

Originally posted by shadowfax0
Everyone go look up 'Project Appleseed' in Google, RIGHT NOW. I made a cluster between my G4 tower and a TiBook to render a Mathematica problem...and it still took a week! Anyhow, I suggest everyone go look fo rit, becuase just a few powermacs can equal the power of a Cray T3E.

Does this work for OS X yet? I'm excited about the technology, but I want to make myself less OS 9 dependent, not more.

P-Worm

[Rocketman]

Originally posted by edvniow
Mmmmmm.......Final Cut Pro rendering farms.:D
I wonder if this would only be for the server OSX, hell, I'd love to get two Macs together to do major media editing.:)

That's what IS possible now.

http://www.apple.com/scitech/research/hiperformance/cluster/

http://www.apple.com/scitech/stories/appleseed/

Rocketman

[shadowfax0]

AppleSeed is Mac OS X all the way, it has a Mac OS 9 version, so if you have nodes running 9, they can help too :)

[IJ Reilly]

Originally posted by shadowfax0
AppleSeed is Mac OS X all the way, it has a Mac OS 9 version, so if you have nodes running 9, they can help too :)

Then you really ought to update the Project Appleseed FAQ -- it says no support for OSX yet, and I can't find OSX mentioned anywhere else on the site.

One other gotcha worth advertising: an application needs to be written (or re-written) to take advantage of parallel computing.

[MacHack]

What exactly is in that controller thingy.
It seems to be directly connecting: 1) Firewire, 2) RAM, 3) Processors. Hmmm...
Gee, if only that could help to facilitate linking something that could take advantage of a really fast link to the processor with a 2GB DDR buffer. Hmmm...
I wonder why it showed up in the xServe?
Now that it is in the desktops, I wonder if they, too, could take advantage?
Boy, with faster Firewire and much faster bus speeds coming.... okay, that's enough.

[MacHack]

for reference

[Telomar]

This has been in Apple's sights for a very long time now. Back in January/February it was being aimed at the next major release after the current one. Whether or not they are still aiming it for then I can't say.

[Catfish_Man]

...this type of processing (massively parallel, especially clusters) is exactly what Mach was designed for (according to "Building Cocoa Applications" by the O'Reilly group). The kernel level support is all there (you can schedule tasks against "processor groups", which, iirc, can be on a networked machine), Rendezvous is there (for auto-discovery of clusterable machines), all we need is two things: A little check box in the sharing preference, and a thing that checks each task being processed to see if it's worth sending to another machine (really small tasks would probably take MORE time if you sent them to another machine because the network latency would be higher than the processing time). You could even include it in the home OSX. Advertise it as each computer on a network taking advantage of the unused resources of the others (the server version would probably be optimized differently as you wouldn't have to worry about stealing CPU time from programs on the other machine). I really hope they're thinking along these lines... mmmm...home autoclustering networks....

<crosses fingers>

disclaimer: the info about Mach is not guaranteed to be at all accurate, it's a moderately vague memory

[shadowfax0]

AppleSeed is OS X supported; I've used it in OS X natively. The program you're looking for is Pooch. You can even get it on versiontracker.com I believe. Hope that helps :) Also, a little disclaimer: This is no 'magic program' that no matter what application you drop on it it's going to spread out over multiple computers, it's not designed like that. The program itself must be WRITTEN to be clustered; with my Mathematica problem, I had to write some part of it in C, but Mathematica is meant for things like this, so that helped out. But things like SETI won't work unless someone wants to roll up their sleeves and take a stab at it (which wouldn't be a bad idea!) Anyhow, good luck, program away! :)

[sparkleytone]

i would assume this would only work on GiG-E configurations of Apple Hardware, with Gig-E being required to functoin. You really wouldnt want any kind of latency issues going on here.

[MacHack]

Wouldn't Firewire work better that Gigabit Ethernet for a cluster in the same rack? I would think so. Maybe that could be an option in the setup? Firewire or Ethernet.
What does a fully loaded (42u) rack go for anyway? I need to start budgeting.

[Telomar]

Gigabit ethernet so no not really.

A full 42U rack would set you back probably around $300k. Assuming all you wanted was standard Xserves.

[rkcrane]

For large scale clustering setups there are more problems other than simpling getting the kernal to handle clustered systems. There are bookkeeping and billing requirements needed to handle a large number of clients whose cluster requirements will be varied and variable from hours to weeks.

The system has to keep track of cpu time, main memory and temp and long term storage for billing purposes.

The only company that advertizes a complete suite of middleware that runs on many os platforms including OSX on Mac is

http://www.platform.com.


The type of clustering that they support is a complete Grid system where the distributed computer can be cooperatively virtually assembled from single cpus to local clusters to wide area distributed systems crossing national boundaries, yet maintain secure computation resourcing. This is the only company that I have been able to find that supports the whole suite of real world requirements.

Of course, the software is not free.

I have no personal stake in the company and have never used the software. What I know is just what I have researched and read about in the past year.