Virginia Tech Supercomputer Cluster Info Session

[jettredmont]

Re: Re: Not running OSX

Originally posted by eddyg
And how is "a special build of Darwin" not OS X. OS X is the entire package, including kernel, not just the GUI.

So if they are running Darwin, then they are running OS X.

Cheers, Edward.

And you can download the Linux kernel, compile it, and suddenly be running Red Hat!

While OS X is based on the Darwin Kernel, the Darwin Kernel is not OS X. Although, one can imagine such vagueries dissapearing in a powerpoint presentation to a bunch of college kids.

Also, in keeping with the Darwin builds only, VT allows the possibility of tweaking the OS itself to run more optimally on their servers. It would be extremely hard to tweak the Darwin-under-Panther OS without NDAs and serious help from Apple.

Finally, as others have said, I can think of nothing that OS X adds on top of Darwin that a cluster would need. You're obviously not going to be doing Quartz compositing on these beasties!

 

[ffakr]

Re: Re: Re: Not running OSX

Originally posted by jettredmont
And you can download the Linux kernel, compile it, and suddenly be running Red Hat!

Well, no, you'd be running a Linux kernel. Red Had is just a collection of GNU software packages with a Linux Kernel. If you downloaded RedHat, you'd be running RedHat.

Finally, as others have said, I can think of nothing that OS X adds on top of Darwin that a cluster would need. You're obviously not going to be doing Quartz compositing on these beasties!

How about the entire rapid development environment. Xcode would be pretty sweet in a 1100 node cluster with a 20Gbit network fabric. It would bring new meaning to distributed compiles.
Running OS X not only gets you the development tools, but it also get's you access to all of the whole RAD. I'm sure that professors would rather be running code than writting it.

Also, there is no reason why you have to run quartz and aqua on OS X. It's trivial to boot it to a command line an it's trivial to run X-Window in place of Quartz/Aqua.

 

[nilspace]

Re: Re: Not running OSX

Originally posted by ffakr
The Admin should explain to you that Darwin is not an open-source Kernel. Mach is the kernel in Darwin, which is an Open source OS, which is based off of FreeBSD, which makes up the core of Mac OS X.

Essentially (over simplified):
FreeBSD + Mach + Apple's Mods = Darwin
Darwin + Quartz + Aqua + OpenStep = OS X

You're right. My fault for not specifying this explicitly. But really Darwin is a branch of the Mach kernel. And are you sure that Darwin is based on FreeBSD and not just a forward implementation?

Of course, I remember working on the Mach on a NeXT box (which I still have lying around somewhere).

 

[cbfro]

Just in case some of you don't realize this already, VT ordered these computers right after WWDC. Putting them in with the same bunch that ordered that day. They haven't even recieved the machines yet, and won't get any until Saturday. For all we know, they actually placed there order before those of you complaining of delays, and people that ordered after they did could have already gotten their G5.

 

[Myuuchan]

Re: Re: Re: Not running OSX

Originally posted by jettredmont

While OS X is based on the Darwin Kernel, the Darwin Kernel is not OS X. Although, one can imagine such vagueries dissapearing in a powerpoint presentation to a bunch of college kids.

Uh. That presentation was geared towards faculty, staff, and grad research students in ECE/CS. Total attendence? 30 TOPS, mostly faculty/staff.

 

[uberman42]

You Say...

You say Darwin, I say OS X, you say Mach, I say Darwin; OS X, Darwin, Mach, Darwin, OS X, lets call the whole thing off...or read the FAQ... (http://developer.apple.com/darwin/projects/darwin/faq.html)

 

[Rocketman]

Okay, so this whole thread has me thinking like a programmig shop or a rendershop might.

If this 1000 (100 spares) node compute farm (cluster format) for $5.2m is highly practical either with OSX proper, or with YDL, or with Darwin, or with a recompile of Darwin, according to my in-house geekzoids needs, then it follows that perhaps hundreds of small programming, rendering, and other compute shops/companies need a 4 or 8 CPU version of this infiniband/rack/G5 thing like eraly next week.

So what is the PO item listing for say an 8 2x2 G% CPU cluster farm?
Looks to me like something like this:
8 Apple 2x2 G5 CPU's (1 is master)
1 Infiniband router
8 infiniband cables
24 1GB memory sticks
8 Infiniband PCI cards
2 Racks
1 UPS (specify)
1 Liquid cooler system (specify)
8 1000BT back channel cables
1 1000BT router/switch
8 FW800 cables
1 FW800 router
1 NAS box with infiniband, FW800 and dual 1000BT

And the cost for this mess?

Rocketman

 

[AidenShaw]

GigE good for big chunks, IB for tiny chunks

Originally posted by Rocketman
then it follows that perhaps hundreds of small programming, rendering, and other compute shops/companies need a 4 or 8 CPU version of this infiniband/rack/G5 thing like eraly next week

Few would need InfiniBand....

For example, look at SETI@home, Folding, Grid.Org or any of those massively parallel architectures. They work just fine with dial-up modem connections, maybe a little faster with DSL but not much.

The reason is that the time spent transferring the "work units" is a very small fraction of the time spent analyzing it - a couple minutes at 56K compared to at least several hours to process.

Fast low-latency transports tip the equation upside down - you can move quite large amounts of data for relatively short processing times. Instead of a cumbersome "work unit" batch approach, you can use MPI, OpenMP and other parallel programming tools to do very fine-grained distributed processing.

For example, a program loop could be written to send each iteration of the loop to a different CPU to run in parallel. To do this, you need a good network fabric.

For rendering and other more coarse-grained tasks, plain old GigE will be just as good.

 

[ffakr]

Re: You Say...

Originally posted by uberman42
You say Darwin, I say OS X, you say Mach, I say Darwin; OS X, Darwin, Mach, Darwin, OS X, lets call the whole thing off...or read the FAQ... (http://developer.apple.com/darwin/projects/darwin/faq.html)

well, the thing is...
Darwin is an Operating System.
Mach is a Kernel.

Mach is one piece of software whose main function is to control the hardware (and interface it to other software)

Darwin is much more than just a Kernel. It is the sum of a Kernel, drivers, and lots of software.

Each of these terms are supersets containing the component(s) of the previous term.

Mach (kernel) -> Darwin (core OS) -> OS X (Apple's BSD based OS)

I hate to be uptight and picky, but it's inaccurate to refer to Darwin as a kernel. It confuses people.

That's like taking:
sparkplug -> drivetrain -> Car
... and mixing up all the terms. If you are talking about an engine or the whole drivetrain and you use the term sparkplug... well you've really confusing matters.
It gets worse when you are talking about the differences between the performance of a Honda S2000 and a Nissan Z350 and you keep comparing the drivetrain of the Honda to the sparkplugs of the Nissan.

<-- not trying to be a jerk.. just trying to be accurate.

 

[WM.]

Originally posted by Rocketman

8 Apple 2x2 G5 CPU's (1 is master)
24 1GB memory sticks

Looks like an odd number of DIMMs per box...can't do that...

WM

 

[neilt]

Re: GigE good for big chunks, IB for tiny chunks

Originally posted by AidenShaw
Few would need InfiniBand....
…

This is a quote from an email from the admin of #65 on the Top500.

"The interconnects play a HUGE role in achieving those sorts of numbers. The latency with Gig E is much more than myrinet or any other highspeed interconnect._ I can achieve about 500 G/Flop with the 128 myrinet nodes. It takes over twice that number of Gig/E nodes to produce similar numbers. "

Our cluster has 512 nodes. Only 128 of those 512 have Myrinet (similar, yet slower than infiniband)

So, even though GigE is pleny fast, it really does make a diffence when you are talking about these big clusters. The name of the game is speed. The jobs that we run now take hours and days instead of weeks and months.

neilt

 

[AidenShaw]

Re: Re: GigE good for big chunks, IB for tiny chunks

Originally posted by neilt
So, even though GigE is pleny fast, it really does make a diffence when you are talking about these big clusters.

I agree completely - MPI programming for parallel LINPACKD numbers on thousand node clusters is critically dependent on very fast, very low latency network fabrics.

I was replying to someone who suggested using InfiniBand for an 8 node renderfarm.

That would be a waste of money, don't you agree? With far smaller clusters, and far coarser parallelism - the network is not as critical. And besides, every box in this class on the market already includes GigE on the mobo.... Heck, even my new Dell laptop has GigE on the mobo.

 

[neilt]

Re: Re: Re: GigE good for big chunks, IB for tiny chunks

Originally posted by AidenShaw

I was replying to someone who suggested using InfiniBand for an 8 node renderfarm.

Ah..... sorry about that. Yes you are right, i didn't read the post you were replying too correctly.

neilt

 

[Bifrost]

parallelism and high-speed interconnect

I just wanted to comment on two of the issues being discussed in this thread.

First, there were some questions about whether or not the cluster needed to be homogeneous. I doubt that it HAS to be. But you may not gain any advantage by adding faster machines to the cluster. If the programs that the cluster runs are desigend to break the total job into equal-sized chunks (such as the AltiVec Fractal Carbon program available from www.daugerresearch.com) then a cluster composed of, let's say, 5 fast and 5 slow machines would have a performance equal to a cluster of 10 of the slow machines. On the other hand, if the program is written so that it constantly feeds new jobs to any available processor then you do gain some advantage by adding faster machines to the cluster. Faster processors will complete more jobs in the same time compared to slower processors. Each processor does the most it can to help complete the overall task.

The second issue is the need for a high-speed interconnect. This is also dependent on how the programming is done. If the parallelism is coarse-grained and doesn't require a lot of data to be transmitted then Ethernet is fine. But if the parallelism is fine-scale and each step depends on the result of a previous step then the data needs to fly fast and furious and you better have a high-speed interconnect.

For my (very small) cluster I use a variety of machines and Fast Ethernet because my program feeds jobs as processors become available but each job takes about an hour to complete and requires little data to be transmitted. On the other hand a program that breaks a problem into equal chunks and must transmit a lot of data would be best run on a homogeneous cluster with InfiniBand.

My guess is that VT wants a cluster that can perform well for any type of software. They will likely add faster machines when it becomes economically feasible to do so (i.e. when a 4 GHz dual that costs what the current 2GHz dual does now is out), knowing that some programs can utilize that extra speed and some will effectively see it as just another 2GHz machine. And using InfiniBand gives the researchers using the machine the opportunity to use fine-scale parallelism if they need to. Smart move, really, if you've got the $5.2 million to spend. My cluster was assembled for about $12,000 so it's not quite as nice......

 

[websterphreaky]

Apple says, screw you Apple G5 buyers

Have you noticed this rather disturbing trend the last half decade from Apple. PR before Apple Loyalist? PR before "first in line buyers", "OS X will completely support all G3 Macs"

I certainly hope that all the cupertino morning prayers Apple Kool-Aid Drinkers thoroughly enjoy another Apple screwing.

What's that ol saying . . . oh yeah; bend over, grab your ankles and don't forget to smile.

 

[jadedchameleon]

Re: Apple says, screw you Apple G5 buyers

Originally posted by websterphreaky
Have you noticed this rather disturbing trend the last half decade from Apple. PR before Apple Loyalist? PR before "first in line buyers", "OS X will completely support all G3 Macs"

I certainly hope that all the cupertino morning prayers Apple Kool-Aid Drinkers thoroughly enjoy another Apple screwing.

What's that ol saying . . . oh yeah; bend over, grab your ankles and don't forget to smile.

Oh come on. You have absolutely NO idea (nor do ANY of us) exactly when this order was placed. They may have placed it as early as anyone. Besides that fact, almost every company in the world has the idea of "priority customers". You go try to get support from Microsoft or many other major software firms sometime--it's amazing how much more they are willing to accomodate you when you're a major developer instead of joe-hacks-on-the-weekend.

It's not a friggin democracy people, and it's not any better on the other side of the fence so stop trying to color it that way.

 

[Myuuchan]

In interesting news, the first 126 were unpacked (and probably racked) today...

... in an hour and a half.

Woot. Go Hokies.

 

[Johnny Mnemonic]

Details

As a pro Mac tech in a closeby state (MD), I have to admit that I am obsessed by the details of the development of this cluster.

For instance:

What are the details of this speciazlied cooling arrangement?
What are the detials of the custom built racks? What make them special?
Is each G5 shipping with a keyboard, mouse, and CDs? Do they have the Infiniband PCI pre-installed?

What is going to be done with all that extra HD space? That's a lot of pr0n!

Of perhaps wider interest:
I saw a post from an Mellanox developer on the Darwin Dev mailing list about mid-July. That would lead me to believe that their drivers aren't ready yet.

Are they?
When they are, who will own them--Mellanox, Apple, or VT?
Will they be available for public use? Let's say I wanted to build my own cluster...(and I might, but maybe at 1/10th the size)

Can you tell us more about Deja Vu? How will that work? I saw an interesting post on /. about the use of non-ECC RAM, and what that will mean for errors in a cluster of this size. How will Deaj Vu protect against that kind of issue--namely errors in off-the-shelf equipment?

What is the expected fail rate of these units? How will it be handled? Repair/replace?

Why is Darwin the first choice? Why is Linux the fall-back? Why isn't the cluster understood well enough to know whether or not Darwin will meet the needs already (ie what if the fallback doesn't work, also?)

What kind of monitoring of each node is going to be done?

Let me invite you to post these details not only here, but to a new apple list for clusters: http://lists.apple.com/mailman/listinfo/clusters. Particulary the info on IB--there's interest there about using these drivers, if they're going to be available.

And a shout-out to neilt--you don't know me, but I'm guessing you work for a little outfit called t-gen. Like strangers in the night...

 

[neilt]

Re: Details

Originally posted by Johnny Mnemonic
As a pro Mac tech in a closeby state (MD), I have to admit that I am obsessed by the details of the development of this cluster.

And a shout-out to neilt--you don't know me, but I'm guessing you work for a little outfit called t-gen. Like strangers in the night...

Yep. it's a little outfit with a huge amount of processing power!
I used to work for NHGRI at the NIH as the lead mac tech, now I run technical support for TGen.

where are you in DC? I used to live at the Constitution and 14th on Capitol Hill

neilt

 

[Myuuchan]

I'm afraid I can't answer any questions about the physical details about the cluster until wednesday - i.e. the morning after I spend time in assembly.

 

[themadchemist]

Re: PSC

Originally posted by Booga
The only acronym for PSC I know, and one that fits with supercomputing, is the Pittsburgh Supercomputing Center, in Pittsburgh, PA. It's one of the original NSF supercomputer centers, operated from Carnegie Mellon University, and one of the reasons the internet's predecessor was created. I believe PSC upgraded through the years from a Cray X-MP to a Y-MP to a T3D, and who knows what they have now.

It's kind of funny and makes a certain amount of sense they'd be involved. CMU must have been one of the biggest customers for NeXT computers back in the day, and has always maintained decent Mac clusters as well as their UNIX and Windows offerings for their students.

oh, Carnegie Mellon is quite pro Macintosh. Two years ago, when all the universities were sending me their brochures, I was considering applying to Carnegie Mellon. I was looking over their application, and there was one part that dealt with submissions of digital media (I think it was for art students, but I was reading it anyway, don't ask me why). Well, it said that they would ONLY accept submissions that could be read on a Mac.

I draw from that that the school has a pretty Macintosh streak in it. It may just be the Artsy people, but I'm not sure. I had the feeling that they were kind of Mac-oriented in general.

 

[niter]

Originally posted by iPC
More than half, from what I have heard. At best, this is just a nice excuse. It is good PR for Apple, but it is a lame way of trying to appease all those customers that ordered these machines when they were announced back in June. You would think Apple wants to keep the Mac faithful that buy the newest stuff happy...

I will admit that I am a complete newbie when it comes to poking around in Apple info, but I do know Tech. Even if Apple did not have a committment to work with Educational needs, I think it would be justified for Tech to commandeer a significant amount of any new Apple products based on the fact that Tech supplies most of their computing facilities with brand new Apple products. In that respect, Tech is a VERY loyal customer. If you do not believe me, swing by the Math Emporium where 500 brand new iMacs reside. That computing lab always has the newest Apple computers available. Then you can visit the library and the numerous other computer labs on campus to see the vast array of Apple products. Then look at the Apple computers Tech provides to its professors. I really wonder how many students Tech has made into loyal Apple users. All I know is that if I want to find the newest and latest when it comes to an Apple desktop, all I have to do is walk into one of Tech's computer labs.

Putting all that aside, so what if Apple sells Tech a bunch of Macs for PR? Good for Apple. Their name is getting splashed all over the place and piquing more interest in their products. Education causes, whether for Tech or another university are very noble and I say kudos to Apple for putting education and research first.

 

[nilspace]

Re: Details

Originally posted by Johnny Mnemonic

For instance:

What are the details of this speciazlied cooling arrangement?
What are the detials of the custom built racks? What make them special?
Is each G5 shipping with a keyboard, mouse, and CDs? Do they have the Infiniband PCI pre-installed?

Well, I've heard some of the answers to these q's from the SysAdmin:

-> The custom racks are meant to hold 5 G5's in their cases.
-> Each system *is* shipping with keyboard, mouse, etc. (even the styrofoam packaging) all of which is being stored at a warehouse.

 

[Spock]

I want to see pictures. I bet in about 5 years we will see these on eBay. Or whatever is going in 5 years