UMaine Building 256 Node G5 Supercomputer

[MacRumors]

In a press release (http://www.umaine.edu/news/071904/ArmySupercomputer.htm), the University of Maine announced its intentions to build a 256-node cluster of G5 Xserves (http://www.apple.com/xserve/) at the Target Technology Center in Orono. Progress on building the machine can be watched in their online gallery (http://www.clusters.umaine.edu/gallery/xserve), as well as a webcam (http://www.clusters.umaine.edu/) mounted on the server room door.

Aided with funding from the US Army, this $680,000 cluster will remain running 24/7 and will be available for use by both UMaine researchers and businesses. Dubbed "Baby MACH 5", this machine is meant to be a running test for the US Army's installation of it's 1566 node cluster (http://www.macrumors.com/pages/2004/06/20040621202310.shtml) dubbed MACH 5.

256 node clusters seem to be a magic number, with the recent release of the UCLA Plasma Physics Xserve cluster (http://www.macrumors.com/pages/2004/07/20040705194056.shtml) also having 256 nodes.

[mmmbop]

Apple: Supercomputers for the rest of us.

Who isn't building a G5 supercomputer these days?

[animationkid]

is it just me or are super computers made of G5s the official "IN" thing at the moment

[GFLPraxis]

Great! That means we get to wait even longer for our G5's to ship! ;)

On the bright side, this also means that more people are adopting Apple computers...increased sales may mean decreased prices eventually :)

[Laslo Panaflex]

Mach 5?

"Go Speed Racer, Go Speed Racer, Go Speed Racer, Go!"

[Freg3000]

For the record: This is a different machine than the Army one we heard about a couple weeks ago?

I think it is but I want to make sure.

[musicpyrite]

Just another step for Apple.

[Mudbug]

For the record: This is a different machine than the Army one we heard about a couple weeks ago?

I think it is but I want to make sure.

It is indeed a different machine. the army is partially funding this one to get a good look at the setup/maintenance before sticking their own together.

[itsa]

Great! That means we get to wait even longer for our G5's to ship! ;)

On the bright side, this also means that more people are adopting Apple computers...increased sales may mean decreased prices eventually :)

Easy now!!! ;)

[Mudbug]

What happened to the huge air intakes (http://www.clusters.umaine.edu/gallery/album05/dsc00465)?

these are cluster nodes - the front is different on them, not needing the same layout as the full server version.

[Skraemer123]

The cluster nodes don't seem to have the huge air intakes look at apples website. I ownder why the otherones need them then?

[Soire]

Can I just say that it's all the right places that are running with these Apple supercomputers?

If you know what I mean, not to be a snob or anything, but from what I've heard the places that are using extensive Apple products are:

-Virginia Tech
-US Army
-UCLA
-UMaine
-Princeton

I think apples would go quite well at these places. They've got that Appley feel and are down to earth, no BS places. Just as long as Harvard doesn't start using macs, I think Apple has attracted some upstanding customers. "Just my $.02" as ya'll say.

[the_mole1314]

Watch the web cam, it's LIVE. They're installing the units RIGHT now! :D

[michaelrjohnson]

This is definitly a positive thing for Apple. No doubt.

[Buzzfish]

As a UMaine Computer science Alum, this is some of the coolest news I've heard in a long time.

Wish i was back up there now!

[slipper]

so wait, are these the 2.5ghz xserves?

[jackieonasses]

so wait, are these the 2.5ghz xserves?
i dont see those at the apple store do you?

[pgwalsh]

Just as long as Harvard doesn't start using macs, I think Apple has attracted some upstanding customers. What do you have against Harvard?


It appears from the pictures that each node has one U space between the lower and uper node. I suppose this is for cooling, but I wonder if this was Apples suggestion or something they decided to do...

Has anyone seen the updated pictures of the Virginia Tech supercomputer?

[sonyrules]

Man, What a deal. Apple is just going off the charts with the Xserve. This keeps up, IBM is going to be BUSY BUSY BUSY!

Just wait till 2.5 Xserves come out. OH and someone need to call cnet, i guess they underestimated the xserve also.

GO APPLE, WHERE MY G5 POWERBOOK?? *L* had to say it

[nagromme]

That's a lot of Mac clusters lately. Big Mac/System X, Mach 5, Baby Mach 5, UCLA Plasma... and some others smaller than 256 I think?

Xserve Cluster Nodes lack the big intakes--but they probably don't have much airflow/heat problem since they are missing lots of internals... like any drives at all. (EDIT: one HD of course... no optical drive, no additional HD bays.)

The extra in-between space in the racks probably just means they have space to spare :)

[yamabushi]

A 2TFlop+ cluster like this one is going to be is pretty powerful. It would easily be in the top 500 supercomputer list. In fact, it should probably make the top 100 this fall if they decide to post benchmark results.

[MacsRgr8]

WOW.....

It's getting better and better.....
who would have thought all these G5 "super computers" a couple of years ago.

A G5 super computer is getting to be more "in" than an iPod :D

[Soire]

What do you have against Harvard?





I don't like Harvard much is all.

Princeton yes, Harvard no.

[dizastor]

Wow. I'm really impressed with apple's leap into the supercomputer arena.

This has to be good for business.
Not that supercomputers are going to sell like ipods, but never the less.... it's still great news.

[Mudbug]

Viriginia Tech has apparently completed its transmutation to G5 Xserves from the original G5 desktop models. The system is currently undergoing stability testing, and benchmarking should begin on or around July 19th.

http://www.tcf.vt.edu/

[sinisterdesign]

A G5 super computer is getting to be more "in" than an iPod :D

next we'll have G5 Super Computer DJ Parties - bring your favorite 15 enomics or proteomics components and sequence the night away!

or maybe spontaneous Cluster Parties - bring your own G5, hook them all up into a cluster and analyze nucleic acid sequences 'til you can't analyze nucleic acid sequences no mo'!

ok, maybe they're not quite as fun as iPods...

[Lancetx]

Viriginia Tech has apparently completed its transmutation to G5 Xserves from the original G5 desktop models. The system is currently undergoing stability testing, and benchmarking should begin on or around July 19th.

http://www.tcf.vt.edu/

That's great news, thanks for the update and link. I can't wait to see the results to see where it ranks this time on the world supercomputer list.

[xtekdiver]

So who will build the Xserve super computer that will take the number 1 spot away from the Earth Simulator? I predict it is going to happen soon. Maybe when the 3GHz version ships; and for a fraction of the cost. Imagine Apple owning the top 10 fastest supercomputers in the world! :eek:

[jholzner]

So these clusters all run Mac OS X...I wonder how much they will increase in performance once they release Tiger...since it's 64 bit native! Anyone have any idea what kind of impanct 10.4 might have?

[JW Pepper]

If this keeps up Apple will never release an iMac G5!

There is no doubt that the xServe G5 is a real hit and now that Oracle 10g is at release stage demand for this product is clearly going to grow fast. I am sure that Apple makes a LOT more money from xServes than iMacs so they have got their prioritys right.

The eMac represents tremendous value right now and so I doubt that Apple will loose to many Edu sales, bu it is obvious that IBM need to scale their production to meet the demand from Apple. This could be good news for Apple stock.

[themadchemist]

This is exciting...With more and more of these supercomputers popping up, the Xserve is really becoming a mainstay of this market. Way to break out of the consumer & design stereotype, Apple! :D

[sharky2313]

i like the rapid cam updates, watching some total stranger is bringing out the mac stalker in me :p

[wPod]

so, with xserve supercomputers becoming so popular, i was wondering if anyone would be interested in investing in a supercomputer. . . you know just for the fun of not being the only kid on the block that doesnt have one! ive got $20 for starters, is anyone else in?

[Earendil]

-Virginia Tech
-US Army
-UCLA
-UMaine
-Princeton

...They've got that Appley feel and are down to earth, no BS places.

Never thought I'd hear the US Army as being a real Appley feeling down to earth no BS sorta organization ;)

May all very well be true, just never thought I'd hear it :D

Tyler
Earendil

ps
who the f rated this negative? I don't care how you look at this, it's POSOTIVE.

[nagromme]

If this keeps up Apple will never release an iMac G5!

These clusters are great to promote Apple by proving the power, quality, and cost-effectiveness of their offerings. But 256 Xserves here, 1000 there, are nothing compared to the volume of mainstream Apple products.

In other words, whether IBM can make enough G5s or not, Xserve sales aren't going to make or break other products' availability.

Also, iMacs if they have a G5 at all are likely to be 1.x GHz.

[Scubee]

So who will build the Xserve super computer that will take the number 1 spot away from the Earth Simulator? I predict it is going to happen soon. Maybe when the 3GHz version ships; and for a fraction of the cost. Imagine Apple owning the top 10 fastest supercomputers in the world! :eek:

The word is that COLSA's Mach 5 will be tripleing in size next year.

[cr2sh]

These clusters are great to promote Apple by proving the power, quality, and cost-effectiveness of their offerings. But 256 Xserves here, 1000 there, are nothing compared to the volume of mainstream Apple products.

In other words, whether IBM can make enough G5s or not, Xserve sales aren't going to make or break other products' availability.

Also, iMacs if they have a G5 at all are likely to be 1.x GHz.

Indeed, the first quarter that the g5's were available Apple shipped 221,000 of them.... an order of this magnitude accounts for less than 1% of the quarters shipments. It won't effect your shipping date... much.

[Freg3000]

Anyone else having a problem with the webcam? It just is stuck at 5 sec delay and nothing ever happens. :confused: :confused: :confused:

[The Red Wolf]

It's not a G5 PowerBook. Its 128 G5 PowerBooks. No on can carry it, but running on battery wouldn't it be cool to have a lorrie full of them and a portable powersource/battery back up? Would a military one one? Sure. Would a school want one? Why not? Everyone needs twin fiber ports on their new G5 PowerBook anyway. Possible uses for a mobile supercluster? #1 saying G5 PowerBook in a post and getting people riled up about it. #2 I'm sure everyone will think of a use... And then some company will make a 256 node portable supercluster. Or a military will buy one and make a 1024 node portable supercluster... Maybe a Cargo Jet would be a good place to set it up? Hmmm. Sorry, I know, off topic. But it could have some uses. Especially if they are dual G5 Powerbooks. (Insert smiley of your choice.)

[Scubee]

Anyone else having a problem with the webcam? It just is stuck at 5 sec delay and nothing ever happens. :confused: :confused: :confused:

Tried clicking where the photo should be? It will bring up an 640x480 diplay in a new window.

[Freg3000]

Tried clicking where the photo should be? It will bring up an 640x480 diplay in a new window.

It just does not work in Safari for me. I got it to work in Firefox though. The broken image only shows up in Firefox.

http://www.clusters.umaine.edu/cam.php

That is a direct link.

Firefox left / Safari right

[Mudbug]

It just does not work in Safari for me. I got it to work in Firefox though. The broken image only shows up in Firefox.

http://www.clusters.umaine.edu/cam.php

That is a direct link.

Firefox left / Safari right

it worked for me in Camino, but not in Safari. Dunno why...

and:
The word is that COLSA's Mach 5 will be tripleing in size next year

What word? Who's word? Where'd you hear that?

[macridah]

Just keep them coming ...

I wonder what is apple share of the server market. It's probably still small, but the percent increase for apple has to be big.

[RubberChicken]

We ordered BTO XServe with raid card late Jan, STILL NOTHING. :mad: :mad: :mad:

[iChan]

seeing as the earth simulator costed $300m+, isn't it just a matter of time before some one out there plops down $100m+ on an xserve Supercomputer? I mean, Apple supercomputers scale linearly and therefore, the extra money put into an XServe cluster will have a direct correlation with the increase in power.

let's see... going on VT's original "Big Mac" which cost $5.2M, they got 10.28TFlops.

The Earth Simulator cost $300m+ and "only" produced 35.86TFlops.

therefore, if someone out there spent $300m on an apple cluster, then they can probably get 593TFLOPS!!!!! WHOAH!!!!! that would OWN anything out there in the world!!! Probably more powerful than all the top 500 put together!! (exaggerating)


what does every think...?? should it be done? is my reasoning incorrect?

edit: miscalculation. Sorry/Thanks Smirk!

[smirk]

let's see... going on VT's original "Big Mac" which cost $5.2M, they got 10.28TFlops.

The Earth Simulator cost $300m+ and "only" produced 35.86TFlops.

therefore, if someone out there spent $300m on an apple cluster, then they can probably get 201TFLOPS!!!!!

what does every think...?? should it be done? is my reasoning incorrect?

I get 300 / 5.2 * 10.28 = 593 TFlops. Even better.

[superstring]

I just noticed MacMall has 18 of the Dual 2gig xServes in stock. So for the poster who is still waiting, might be worth checking them out. It seems like IBM/Apple have been getting better with the chip yields. still waiting for the 2.5 though...

[BlkBear]

I mean, Apple supercomputers scale linearly
...
is my reasoning incorrect?

NO cluster computers scale linearly. I don't care what it is. How these large clusters scale depends heavily on the innterconnect. The cost of the low latency/high bandwidth interconnects does not scale linearly either. MACH 5 is only using gigabit ethernet, so you'll see them quite a bit lower than their theoretical maximum performance. If they doubled the number of nodes and still used gigabit ethernet, their % efficiency (actual performance / theoretical performance) will definately go down, that is each node you add gives you less and less additional performance.

[whooleytoo]

Odd, check this (http://my-pbs.sourceforge.net/authors.php) out. One of the authors on this project has already been working on an XServe cluster in the University of Maine.. I wonder how long that page has been up..

[mcdawson]

seeing as the earth simulator costed $300m+, isn't it just a matter of time before some one out there plops down $100m+ on an xserve Supercomputer? I mean, Apple supercomputers scale linearly and therefore, the extra money put into an XServe cluster will have a direct correlation with the increase in power.

Remember that some (large?) part of the $300 million cost was for infrastructure--a building that could withstand an 8.5 earthquake, etc. The VT build included air conditioning, but not the building (that was already there). The Apple cluster is still a great cost/TFLOP machine; however, you have to subtract all the infrastructure of the earth simulator to get a more fair comparision.

As to scaling linearly, that probably depends on the software running on the clusters. I believe that one of the "breakthroughs" that VT made was some great software that mangaged a growing cluster welll. I've heard that not all clusters always scale well…

[wordmunger]

Just remember, IBM is not sitting down. They are designing their own supercomputers using the same chips that power the xServe. I suspect the computer that defeats the Earth Simulator will will be an IBM, not a Mac.

[xtekdiver]

The word is that COLSA's Mach 5 will be tripleing in size next year.

Well, that didn't take long!

[BlkBear]

Odd, check this (http://my-pbs.sourceforge.net/authors.php) out. One of the authors on this project has already been working on an XServe cluster in the University of Maine.. I wonder how long that page has been up..

I think you'll find most of those project authors listed on this pages as well: http://www.clusters.umaine.edu/about_us.php

[JW Pepper]

These clusters are great to promote Apple by proving the power, quality, and cost-effectiveness of their offerings. But 256 Xserves here, 1000 there, are nothing compared to the volume of mainstream Apple products.

In other words, whether IBM can make enough G5s or not, Xserve sales aren't going to make or break other products' availability.

Also, iMacs if they have a G5 at all are likely to be 1.x GHz.

Last year the server market was worth $46 billion, that it is a lot of servers and a lot of processors. If Apple are tapping that market , and the signs are that they are starting to make inroads then the sales might not be as small as you might think.

http://www.theregister.co.uk/2004/02/27/server_vendors_cheer_strong_q4/


Oh and another thing, I don't believe for one minute that Apple will release the new iMac at anything less than a single 2ghz processor. It would just be like shooting their own foot if they did. Apple will need a 2ghz processor to get anywhere with consumer sales and they know it.

[virividox]

go apple, but dude with all these super computers being powered by apple stuff how long will it take for normal ppl to receive their computers

[nagromme]

... I don't believe for one minute that Apple will release the new iMac at anything less than a single 2ghz processor. It would just be like shooting their own foot if they did.

Clearly the higher the GHz the more will sell--but I don't the difference will be that great. Apple will sell a 1.x G5 if they have to, and that's a marketing challenge that's only getting easier as people slowly realize MHz isn't what to look for in a computer.

And if IBM is having trouble making enough 2.x G5x, then that's a reason Apple MIGHT have to put 1.x in the first G5 Macs.

And for most users, the difference in speed is irrelevant--the other features of the system matter more. We aren't all running 500 MB Photoshop files or rendering 3D movie effects.

Also, a new G4 iMac wouldn't shock me--I expect a G5, but a G4 at a cheap price (or a choice of G4/G5) could be just fine. Again--pure marketing: position the G5 for pros and power users if you have to.

[robertgreen94]

I believe the reason they are picking 256 nodes is because the US government use to require an inordinate amount of paperwork to justify the need for more than 256 nodes. I doubt they (US GOV) have changed much since I had to look into this a couple of years ago.
Though I could be wrong, maybe they just like the number 256! :)

[Kiwi-Todd]

Just imagine the inroads Apple might make with offering these server clients a LOW cost G5 'enterprise' box.

[budgej867]

I'm going to MSYM music camp at the University of Maine in Orono on the 18th. What good timing! I may be able to go see this supercomputer under construction.

[clusterman]

I believe the reason they are picking 256 nodes is because the US government use to require an inordinate amount of paperwork to justify the need for more than 256 nodes. I doubt they (US GOV) have changed much since I had to look into this a couple of years ago.
Though I could be wrong, maybe they just like the number 256! :)

The number 256 is not entirely arbitrary. 256 is a perfect square, which makes for nice, neat problem decomposition.

Plus, the price was right.

We looked at buying more nodes of a different architecture, but the Xserves optimized our A/C and power for the room. If we had the facilities, and the budget for a MACH5, we would have bought it ourselves.

[Henriok]

Just remember, IBM is not sitting down. They are designing their own supercomputers using the same chips that power the xServe. I suspect the computer that defeats the Earth Simulator will will be an IBM, not a Mac.Indeed. IBM is hard at work on BlueGene/L that will soundly kick Earth Simulator from the number one position before it's even completed. As it is now, there are two BlueGene/L prototypes on the Top10-list, and they are only a small part of the completed machine which will consist of approximately 130 000 custom built dual core PowerPC 440 processors running at 700 MHz. They are aiming at 350 Tflops next year.

IBM will sell a lot of BlueGene type super computers due to its unprecedented computing density. And.. on top of that we'll probably see some PowerPC 970 based supercomputers based on their JS20 Blade Center (the lastest using 2.2 GHz 970FX). There's actually one JS20 based supercomputer on the Top500 list already and it's Russias Joint Supercomputer Center in Moscow, using just 168 processors (84 JS20 blade modules). Expect more of those, and perhaps some POWER5 based too.

If this is a new niche market for Apple I really hope that they will design a sexy rack cabinet. These open racks which shows endless cluster nodes, switches and cables running are geeky enough but they are not sexy. Look at super computers from Sun (http://www.sun.com/smrc/photos-prod/downloads/ppserve10k-506sm.jpg), SGI (http://harpo.wustl.edu/Images/altix128.png) or even IBM (http://domino.research.ibm.com/Comm/bios.nsf/pages/bluegene2003.html/$FILE/rendition_2.jpg).. they are really stylish, and I wish that Apple could make something similar. I almost expect them to do it.

For those of you who are savvy with 3D apps, if you need suggestions for doing a cool Apple product, make this: A cool rack cabinet, the Apple way.

[BlkBear]

I'm going to MSYM music camp at the University of Maine in Orono on the 18th. What good timing! I may be able to go see this supercomputer under construction.


When you are in orono call me at 866-6513. I can arrange a tour for you.

Glen

[rdowns]

I'm going to MSYM music camp at the University of Maine in Orono on the 18th. What good timing! I may be able to go see this supercomputer under construction.

This one time at band camp...

[whooleytoo]

Heh! It's pretty cool the guys working on the cluster are here in the forums.

So, here's some things we need to know:

1) XBench scores..
2) Halo framerates.
3) The combined wind-power of 256 XServes' fans (if you don't know this off hand, maybe you can calculate it.. maybe you have a calculator nearby? ;) )
4) Boot times.
5) The name of the guy who has to run around and hit the power button on all the machines.
6) The average air-speed velocity of an unladen sparrow - particularly when flying with (3) as a tailwind.

I'm sure others can come up with more...

[AliensAreFuzzy]

6) The average air-speed velocity of an unladen sparrow - particularly when flying with (3) as a tailwind.


I believe it would be an uladen swallow. But then of course you'd have to differentiate between African or European.

[clusterman]

Heh! It's pretty cool the guys working on the cluster are here in the forums.

So, here's some things we need to know:

1) XBench scores..
2) Halo framerates.
3) The combined wind-power of 256 XServes' fans (if you don't know this off hand, maybe you can calculate it.. maybe you have a calculator nearby? ;) )
4) Boot times.
5) The name of the guy who has to run around and hit the power button on all the machines.
6) The average air-speed velocity of an unladen sparrow - particularly when flying with (3) as a tailwind.

I'm sure others can come up with more...


1) In due time.
2) Halo a game? Sheesh, we're a university. We don't have time to play games, and can't afford it after dumping out the money for this machine! ;)
3) I'm sure we can calculate it based on the CFM of the machine. I know that when idle, these things are HOT.
4) Unknown for a couple of weeks. We'll be trying out a bunch of different things, most notably diskless nodes.
5) No one needs to press reset. We have remote power on/off equipment that can control every port in the cluster. I insisted on it this time 'round.
6) No supercomputer can determine this answer.

[army_guy]

NO cluster computers scale linearly. I don't care what it is. How these large clusters scale depends heavily on the innterconnect. The cost of the low latency/high bandwidth interconnects does not scale linearly either. MACH 5 is only using gigabit ethernet, so you'll see them quite a bit lower than their theoretical maximum performance. If they doubled the number of nodes and still used gigabit ethernet, their % efficiency (actual performance / theoretical performance) will definately go down, that is each node you add gives you less and less additional performance.

How about RED STORM 10'000 Opteron CPU's using CRAY interconects, if anything will take number 1 it wont be by an Apple machine. And dont confuse a cluster with a REAL supercomputer as they are two completly different things, a cluster may have higher CPU performance but a true supercomputer will have much higher bandwidth, I/O and lower latency.

[nubero]

The cluster nodes don't seem to have the huge air intakes look at apples website. I ownder why the otherones need them then?

If you look again, you'll see, that the air intakes are even larger than on the normal G5 servers. it's just that they dont have two big holes but many tiny ones. But the G4 had 4 HDs with tiny holes, the G5 has 3 Hds with two big ones in between and the new node has only on HD with the rest of the space as one big hole. I could imagine that you make the choice for small holes like this whe you want to hold back large particles of dust... (even though they probably stand in a controlled environement at most places anyway)

--- --- --- --- ---
Free Desktop Pictures!
http://homepage.mac.com/nuber

[~Shard~]

This doesn't surprise me at all. I think we're going to see a lot more of these types of clusters popping up due to the precedence VTech set as well as the obviously low costs involved with building a G5 supercomputer. I'll be curious to see their benchmarks.

So when do the 2.5 GHz G5 xServes come out? ;)

[whooleytoo]

1) In due time.

Sigh.. can't be all that fast then.. ;)


2) Halo a game? Sheesh, we're a university. We don't have time to play games, and can't afford it after dumping out the money for this machine! ;)


Oh, come on.. you know you want to be kicking some spotty teenager's ass online, and are just waiting for him to say "Well.. I'm playing on a top of the line Alienware rig, what are YOU playing on?..."


3) I'm sure we can calculate it based on the CFM of the machine. I know that when idle, these things are HOT.


Pretty noisy too, I'd imagine.


4) Unknown for a couple of weeks. We'll be trying out a bunch of different things, most notably diskless nodes.


Interesting.. I guess you'd probably want everything in RAM anyhow - disk latencies being what they are.


5) No one needs to press reset. We have remote power on/off equipment that can control every port in the cluster. I insisted on it this time 'round.


I kinda suspected as much - though I seriously wouldn't want to be the one who trips and hits THAT button by accident!


6) No supercomputer can determine this answer.

That statement is only true outside a Jobsian Reality Distortion Field..

[clusterman]

Hiya dedicated fans:

We've added an additional webcam for you to feast your eyes on:

http://www.clusters.umaine.edu/~wages/cam2/

This shows the second half of our Xserve build, and peeks into our old P3 cluster, Blackbear.

[Tyler_D]

Hi,
I just watched the webcam. Why the hell have you installed an iSight on the head node?

Edit:
Ah okay, i get it.

[Scubee]

What word? Who's word? Where'd you hear that?

COLSA's tripleing of their MACH5 is expected next year. From the same source, word is that GigE will not be used for the upgrade.

[Mudbug]

COLSA's tripleing of their MACH5 is expected next year. From the same source, word is that GigE will not be used for the upgrade.

that's great. You heard this where?

[Maclicious]

What do you have against Harvard?



Sorry, can't resist:

http://foi.missouri.edu/usenergypolicies/harvardharkenwsj.html

[WinX]

seeing as the earth simulator costed $300m+, isn't it just a matter of time before some one out there plops down $100m+ on an xserve Supercomputer? I mean, Apple supercomputers scale linearly and therefore, the extra money put into an XServe cluster will have a direct correlation with the increase in power.

let's see... going on VT's original "Big Mac" which cost $5.2M, they got 10.28TFlops.

The Earth Simulator cost $300m+ and "only" produced 35.86TFlops.

therefore, if someone out there spent $300m on an apple cluster, then they can probably get 593TFLOPS!!!!! WHOAH!!!!! that would OWN anything out there in the world!!!
what does every think...?? should it be done? is my reasoning incorrect?


A major difference between a real supercomputer like the Earth Simulator and a cluster built as a collection of inexpensive machines is data bandwidth.

The 1GHz FSB "elastic interface" between a Xserve G5's 2GHz CPU and its memory controller has an effective bandwidth of about 3.2GBytes/sec. That bandwidth is only sufficient to send about 400 million 64-bit floating-point numbers per second to each CPU for number crunching -- i.e. about 400Megaflops per CPU, or 0.8GFlops per dual-CPU 2GHz Xserve.

As for inter-node networking, the Virginia Tech cluster's Infiniband interfaces have a raw bandwidth of 10Gigabits/sec.
The Army's MACH-5 Xserve cluster uses a networking technology that is even slower -- Gigabit ethernet with a raw bandwidth of 1Gigabits/sec.

With the Earth Simulator's 640 eight-CPU nodes, the memory bandwidth to each CPU is 32 Gigabytes/sec -- i.e. an order of magnitude larger than an Xserve's. In fact, the E.S.'s 10 Terabytes of shared memory has a speed comparable to a 2GHz G5 CPU's on-die L1 cache ! :eek:
The memory bandwidth is sufficient to feed 4 billion 64-bit floating-point numbers per second to each one of the E.S.'s 5,120 CPUs -- i.e. to sustain about 20 Teraflops with central memory-based data. As a comparison, a 1100-node dual 2GHz Xserve cluster can sustain a memory-based throughput of about 0.88 Teraflops. The two architectures are thus not really in the same league.

Each link in the network interconnecting the E.S.'s 640 nodes has a raw bandwidth of 128 Gigabits/sec, which is thus significantly faster than the VTech cluster's Infiniband.
It's also interesting to note that each link of that custom network has a bandwidth that is several times that of an G5 Xserve's CPU-RAM bandwidth !

Furthermore, on a thousand-node switched Infiniband or Ethernet network, data packets must statistically travel through numerous Infiniband or Ethernet switches before reaching their destination. The traffic will thus experience significant latency. The E.S.'s network, on the other hand, is fully meshed, and each node can thus communicate with any other node in a single low-latency network hop !

Thus, for problems that are not embarassingly parallel, and which require a significant amount of data exchange between the CPUs or across nodes, or for problems that require crunching through data sets too large to fit in a CPU's small L1 cache, the Earth Simulator might be an order of magnitude faster than a PC/Mac-based cluster whose theoretical peak performance, based on CPU speed alone, might appear to be superficially comparable.

[army_guy]

Exactly WinX, for supercomputing applications its not the processor performance which matters, its the bandwidth regardless of what people think. This is what CRAY concentrates on, memory/IO bandwidth, interconnects etc.. Another reason why redstorm will be a hit taking the advantage of clustering, the Opteron architecture and alleviating most of dissadvantages associated with clustering.

Like I said PROCESSOR PERFORMANCE IS NOTHING WITHOUT THE BANDWIDTH.

[clusterman]

Actually, gents. It's ENTIRELY problem-dependent.

Many people were bashing the design of MACH5 as shortsighted, being built without any form of "traditional" top-end interconnect such as Myrinet, Infiniband, Quadrix, etc.), but from our research, we found that the particular code that MACH5 was built for only communicates on the order of 0.01%-0.1% of the time. That's pretty darned low. And the bandwidth required, for a given problem, also varies, but cluster-wide we've seen bandwidth in 50MB/s (where each machine has a 100Mb/s network connection). When the MACH5 machine is built, each node will obviously have 10x the bandwidth available, with roughly the same latency as Fast Ethernet. This will be interesting to see, as this could quite possibly be the largest machine for quite some time, that utilizes traditional COTS-based networking.

Our first-gen cluster was built with Myrinet, and we determined that it was a complete and total waste of money for our problem. With costs ranging upwards of near 50-60% of total machine cost for these top-end interconnects, you want to be damned well sure that you need it.

Now, the next-gen COLSA machine is rumored will be multi-purpose. AFAIK, MACH5 is single-purpose.

[WinX]

Actually, gents. It's ENTIRELY problem-dependent.
[..]
Now, the next-gen COLSA machine is rumored will be multi-purpose. AFAIK, MACH5 is single-purpose.

As far as specialized, single-purpose machines are concerned, one interesting architecture is the GRAPE GRAvity piPE, optimized for Coulomb force simulations, as occur e.g. in N-body problems with gravity, such as galaxy evolution.

Note that "[..] MD-GRAPE2 can also be used for other problems, such as incompressible fluid simulations with the particle-vortex method, compressible fluid simulations with smoothed particle hydrodynamics, plasma physics with magnetic and electric forces, magnetics, and astrophysics with gravity. Algorithms that have been used include hierarchical tree codes, multipole methods, the Ewald method, and Particle-Particle/Particle-Mesh methods."

http://www.research.ibm.com/grape/grape_mdgrape2.htm

With a peak speed of 48 Teraflops, the GRAPE-6 computer at the University of Tokyo is probably the fastest supercomputing cluster in the world today.

http://www.ids.ias.edu/~piet/act/comp/hardware/

[clusterman]

Hiya folks,

Not sure if any of you are still following our build, but we've gotten another 30 nodes today, another 30 tomorrow, and another 30 on Monday for sure.

We've got updated pics, camera angles on our site so you can follow the build.

[5300cs]

This is very cool for Apple.

I don't know why, but as soon as I read that the army was funding/supporting/whatever, this thing, I immediately though of Ed Rotberg quitting Atari when he was forced to make a military version of Battle Zone.

[macsrus]

Actually, gents. It's ENTIRELY problem-dependent.
I Agree

Computer Systems are tools....

Super Computers and Clusters are tools....

Scientists and Computer Engineers who use and design these tools.... do so to solve problems....

It doesnt matter whose system is the fastest.... that is pure bravado...

What matters is ....what problem do you need to solve....
Once you know the characteristics of the the problems you are trying to solve....
Then design a system that best solves that/those problems.... In the most efficent and cost effective manner....

This is exactly why clusters have become so popular lately....

For many problems..... CFD.. Genome Research... Protien folding.. etc.
Clusters simply provide the most performance for the dollar spent...

While Large shared memory systems like the Earth Simulator do have a place... And do solve some problems that clusters are less suited for...
Suprisingly they also are poor performers at the tasks clusters excel at.

We specifically tested on IBM Regattas...and a SGI Altix.... which are both large shared memory machines with very fast crossbar interconnects.... heck the SGI Altix's Numa memory architecture is amazingly fast....

And for our code and problems...They were both out performed by a Opteron Cluster....
Not to mention The price difference.

So people dont get hung up on machine specs.... Remember computers are tools...
Use what best suits your needs/pocketbook

[army_guy]

Opterons are cool ;) only a few weeks left for this.

http://www.appro.com/image/product/4145h_front.jpg

Engineered for EDA Applications 32/64-bit Windows and LINUX (REDHAT AS 3)
QUAD OPTERON CPU's
64GB DDR 400 ECC REG
0.6TB SCSI 320
NVIDIA Quadro 4000 AGP

This is a workstation....

[clusterman]

We looked at opterons VERY closely. Turns out, we could have bought a TON of 'em (more than we could have cooled). But the mac was faster on our given problem so there you have it.

Plus, the mac is much, much cooler than any opterons we've seen.

[~Shard~]

We looked at opterons VERY closely. Turns out, we could have bought a TON of 'em (more than we could have cooled). But the mac was faster on our given problem so there you have it.

Plus, the mac is much, much cooler than any opterons we've seen.

There you have it, straight from the horse's mouth! Macs are cooler than Opterons, in more ways than one... ;)

[clusterman]

There you have it, straight from the horse's mouth! Macs are cooler than Opterons, in more ways than one... ;)

Cooler as both temperature, and style!