Virginia Tech Upgrades to 2.3GHz Xserves [Archive]

MacRumors

Sep 28, 2004, 04:48 PM

According to AppleInsider (http://www.appleinsider.com/article.php?id=664) and a column (http://www.usatoday.com/tech/columnist/andrewkantor/2004-09-03-kantor_x.htm) at USA Today, the System X (http://www.tcf.vt.edu/systemX.html) supercomputer at the Virginia Tech Terascale Computing Facility has been upgraded to Apple Xserve Dual G5s running at 2.3GHz, up from 2.0GHz.

The cluster consists of 1100 Apple Xserve G5s, each with 4 GB RAM, and was previously upgraded (http://www.macrumors.com/pages/2004/01/20040127014910.shtml) in January from Power Mac G5s to Xserves.

Apple still describes the Xserve G5 (http://www.apple.com/xserve/) as running only up to 2.0GHz, although specs in a leaked graphic (http://www.macrumors.com/pages/2004/01/20040107153003.shtml) in January showed 2.3GHz. The 2.3GHz version was reportedly made available exclusively to Virginia Tech.

pgwalsh

Sep 28, 2004, 04:51 PM

Maybe that's why the PowerMac went from 2.0 to 2.5 because the 2.3 part was spoken for.

Steamboatwillie

Sep 28, 2004, 04:54 PM

I guess they get the special treatment because they are so publicly visible?

mgescuro

Sep 28, 2004, 04:57 PM

Private Public

Sep 28, 2004, 05:04 PM

Glad to see Apple making some good headway in the supercomputing and sciences arena.

I second that. Apple is headed in the right direction with the xserve

AmigoMac

Sep 28, 2004, 05:06 PM

I'm happy, although that one makes me feel my iBook damn slow... ;)

Hey, I use a Mac! Well done apple!

shamino

Sep 28, 2004, 05:10 PM

i really hope they are getting new processors and not entirely new machines...
It takes more than a processor upgrade to turn a PowerMac into an Xserve. I'm pretty sure they got more than new CPU chips :)
VT tuition must be going up ;)
(^^it's a joke people...)
VT sold their G5 PowerMac cluster nodes. They were available through (I think) MacMall for a few months. So they didn't lose their investment. IIRC, they actually sold for close to Apple's original price - people were willing to pay that much because the computers were virtually new and included some certificate of authenticity stating that the computer was a VT cluster node. (Yes, VT didn't get the full purchase price, since the store took a cut, but VT also didn't pay full retail price - they paid Apple's educational institution price.)

nagromme

Sep 28, 2004, 05:24 PM

Cool! (This was reported some time ago, although I don't remember where. Maybe the 2.3 part was just rumor, but it was mentioned--and the Xserve upgrade was done a while ago. Those Xserves have been up and running at VT for some time--pictures an all have been available for several weeks at least.)

And to clarify, this isn't ANOTHER Xserve upgrade. There was no upgrade to Xserves in Jan., just an announcement. This is that announcement now come to pass. That's what I've understood from eveything else I've read--that Big Mac was down for several months waiting, and there was no immediate upgrade with the towers were sold. Anyone know for sure?

Also, if IBM had enough 2.3s to offer on a mass scale, Apple could have done so--and that may happen in future. If not, then at least Apple could offer them to one high-profile Apple customer.

Mantat

Sep 28, 2004, 05:28 PM

wrldwzrd89

Sep 28, 2004, 05:31 PM

Not too shabby, I guess. 2,200 processors running at 2.3GHz.
The Mach5 has 3,132 processors running at 2.0Ghz.

Even better... Mach5 is getting a LOT of attention from the government and NASA.

Glad to see Apple making some good headway in the supercomputing and sciences arena.
Just imagine how many teraflops Mach5 will be capable of if/when it gets upgraded to faster XServes (not necessarily the 2.3 GHz ones - maybe the Mach5 maintainers will wait for the upgrade after 2.3 GHz to make the jump). We'll soon have our first 30 Teraflop Mac cluster.

LimeLite

Sep 28, 2004, 05:32 PM

I really doupt this. If its true, this is the most ridiculous thing I have seen this week. Just imagine the time wasted to switch the computers for a 15% CPU increase. Very hard to believe. And if true, the guy leading this switch must have an angry bunch of scientist at his back. Dont forget that these clusters arent there for braging rights, they are supposed to do some science stuff...
Except for that the Xserves take up way less space than the towers, and all in all produce far less heat, which both saves room in the facility, and creates less cost for cooling the whole thing. In addition, having more space makes it easier for them to expand and add more processors if they want to. This is not just speculation, this is the reported reason.

AoWolf

Sep 28, 2004, 05:35 PM

Hmm so they have 2.3 Ghz x serves the thing I find odd is that apple canada had them on there site(as an error) when the top pro model was still duel 2.0 Ghz. Just think of the stuff the lucky bastards in apple R@D get to test...

Chaszmyr

Sep 28, 2004, 05:37 PM

Old news. I read an article about the virginia tech cluster which said it now used 2.3ghz XServes about a month ago.

munkle

Sep 28, 2004, 05:37 PM

Manzana

Sep 28, 2004, 05:45 PM

Any posted benchmarks? Or what's the difference in teraflops in towers vs xserves.

I mean have they run xbench on it yet :p
And if they could fit all those g5's in that space, that must mean the g5 PB is just around the corner (sorry couldn't help myself)

MacSlut

Sep 28, 2004, 05:55 PM

Mord

Sep 28, 2004, 05:57 PM

Makes you wonder what computer Steve is using! :p

quad core 8GHz g6 cube i bet, with four 80" flat panels connected to some mad ati prototype x900 xt platinum ****.

Macmaniac

Sep 28, 2004, 06:09 PM

I think Apple should release the 2.3ghz XServes to the general public, Apple should keep their servers updated on a frequent basis, that way they will be seen as a bigger IT competitor.

bcsmith

Sep 28, 2004, 06:16 PM

Makes you wonder what computer Steve is using! :p

Probably his beloved cube. Or a Lisa.

wrldwzrd89

Sep 28, 2004, 06:18 PM

I think Apple should release the 2.3ghz XServes to the general public, Apple should keep their servers updated on a frequent basis, that way they will be seen as a bigger IT competitor.
I think Apple's waiting to build up some supply of the 2.3 GHz processors before making them widely available. Apple doesn't want to announce it now and force customers to wait until November to get one. That would be a PR disaster.

broken_keyboard

Sep 28, 2004, 06:26 PM

So the 2.3 G5 does not need liquid cooling, but the 2.5 does?

joesporleder

Sep 28, 2004, 06:42 PM

You've got to be careful with those duel G5 PowerMacs, especially when they start dueling with each other, fighting for bandwidth and other system resources. It'd seem hard for them to be productive at all with little if any system unity! :D

Chaszmyr

Sep 28, 2004, 06:59 PM

So the 2.3 G5 does not need liquid cooling, but the 2.5 does?

I don't think its a matter of needing liquid cooling, I think the liquid cooling is just quieter than the necessary fans.

azdude

Sep 28, 2004, 07:04 PM

Ahh... so the true reason for the previously odd-sounding upgrade emerges. Nice.

applekid

Sep 28, 2004, 07:36 PM

Remember, the Dual 2.5 GHz reach about 105 degrees Celcius at tops on die. Liquid cooling helps keep the fans running quieter, even though I hear it can get pretty loud. The PowerMac G5 could use a redesign for the next line of chips. Gonna need to keep that liquid cooling around for a while.

Mechcozmo

Sep 28, 2004, 07:52 PM

Correction: the Dual 2.5Ghz G5s require the cooling. I read something abou this...they got too hot with the 90nm stuff, and it wasn't practical to add a bijillion more fans. So, they went liquid.

Longey Nowze

Sep 28, 2004, 08:05 PM

So the 2.3 G5 does not need liquid cooling, but the 2.5 does?

the liquid cooling is to reduce the noise, you don't need as much noise reduction in a server environment compared to a PC environment, meaning they can use louder fans in the Xserves

AidenShaw

Sep 28, 2004, 08:55 PM

Just imagine the time wasted to switch the computers for a 15% CPU increase. Very hard to believe. And if true, the guy leading this switch must have an angry bunch of scientist at his back. Dont forget that these clusters arent there for braging rights, they are supposed to do some science stuff...

It wasn't for the speed increase - it was because the lack of ECC memory made the PowerMac G5 cluster so unstable that it was unusable for doing any real work.

The "sugar-coated" version from the VAtech website is

Well with the concept proven we now had to make sure we had a system capable of conducting scientific computation.

We needed to upgrade the system to something with error correcting code (ECC) RAM.

The Power Macs did not support it and the XServes were coming.

And, I bet that you're right that there are a bunch of angry scientists. Angry that the university spent millions on a PR "proof of concept" that was unusable from the get-go. Angry that a year after the system was installed and announced there was still nothing usable by the users. Angry that all the press releases and advertisements didn't translate into computing resources usable by the people from the university.

Maybe a little less angry because they didn't ever have System X taken away from them - since it would never run long enough to have been given to them in the first place.

Do you think that it's a coincidence that the company that Dr. V. is with is pushing Xeons and Itania, including the world's fastest cluster (Itanium, not PPC970)? (http://www.californiadigital.com/)

Did you also see that Dell is selling pre-packaged 64-bit Xeon clusters based on the same Infiniband networking that VAtech is using? (http://news.com.com/Dell%2C+Topspin+tout+InfiniBand+clusters/2110-1010_3-5387085.html) The "cheap supercomputer" just got a lot cheaper!

Fukui

Sep 28, 2004, 09:00 PM

Probably his beloved cube. Or a Lisa.
...or a Toshiba Tecra...

swissmann

Sep 28, 2004, 09:11 PM

Land a few more deals like this Apple. You have the hardware to do it.

EminenceGrise

Sep 28, 2004, 09:32 PM

It wasn't for the speed increase - it was because the lack of ECC memory made the PowerMac G5 cluster so unstable that it was unusable for doing any real work.

That's patent nonsense. The original cluster was not "so unstable that it was unusable for doing any real work" by any stretch. The lack of ECC RAM merely meant that they had to run simulations more than once to verify the run and ensure that bit errors in memory had not skewed the results. With ECC, any simulations can be run once, without any such double checking for bit errors. Hence, you get results much faster (even without the clock speed increase), because you don't have to run simulations more than once. That doesn't make the original cluster worthless, just less efficient.

It's also rather widely accepted that they were willing to live with certain trade-offs in the original cluster so they could 1) prove the concept and 2) make the all important supercomputer "Top 100" for last year's deadline. Near as I can tell, the original cluster was a stop along the way to get the "real" cluster going - time to shake down all of the software and all that - and those actually building the thing knew that, if not from the very beginning, certainly by the time they were putting together 'BigMac I'. I don't see how they could have realistically just put together a perfect working cluster right off the bat - with any computer. They were also able to sell off many of the original machines to pay for the new XServes, so it's not like all of that money just disappeared into a pit. This was a research project in and of itself - its sole purpose was not simply to provide a big cluster for people to use, it was also to see if it could be done in the first place. Maybe they could have spent the money on something else, but for every funded science project, you can find someone who didn't get their project funded and thought theirs was 'better'. This isn't the first time, and it won't be the last.

ClimbingTheLog

Sep 28, 2004, 09:34 PM

It wasn't for the speed increase - it was because the lack of ECC memory made the PowerMac G5 cluster so unstable that it was unusable for doing any real work.

Do you know something we don't? I thought Varadarajan wrote error detection code that was integral to Big Mac? An un-clever algorithm would reduce the speed by half, hardly unusable.

jackieonasses

Sep 28, 2004, 10:14 PM

I was thinking of something today after i read this. What if the dual 2.5's are actually 2.3's but liquid cooled cause the are OC'D? I know I know... if i read what i am saying now i wouldn't believe it either. I just wanted to say something someone else would eventually (and mean it)

MegaSignal

Sep 28, 2004, 10:24 PM

the liquid cooling is to reduce the noise, you don't need as much noise reduction in a server environment compared to a PC environment, meaning they can use louder fans in the Xserves

Site your source, please.

While Apple discusses at length both liquid cooling as well as a quiet operation of their new G5 towers on their web site, I do not recall a link made between these two items; I believe Apple was forced into liquid cooling due to what is commonly known as "heat density". Please refer to the following, copied from someone very knowledgable in this area:

The 970fx 2.0 chips run a LOT cooler than the 970 chips.

Speed CPU Process Typical power Die size Typ. power/area
-----------------------------------------------------------------
1.8 GHz 970 130 nm 51 Watts 118 mm2 0.43 W/mm2
2.0 GHz 970 130 nm 66 Watts 118 mm2 0.56 W/mm2
2.0 GHz 970FX 90 nm 24.5 Watts 66 mm2 0.37 W/mm2
2.5 GHz 970FX 90 nm 50 Watts 66 mm2 0.76 W/mm2

What's important here is the value of W/mm2.

Now, then, the question is what type of chip the 2.3GHz is...

sushi

Sep 28, 2004, 11:20 PM

VT sold their G5 PowerMac cluster nodes. They were available through (I think) MacMall for a few months. So they didn't lose their investment. IIRC, they actually sold for close to Apple's original price - people were willing to pay that much because the computers were virtually new and included some certificate of authenticity stating that the computer was a VT cluster node. (Yes, VT didn't get the full purchase price, since the store took a cut, but VT also didn't pay full retail price - they paid Apple's educational institution price.)
As I stated before, this creates even a more cost effective solution.

Anyone using these clusters can stay current in a more effective manner as compared to prior methodologies. In this case, just sell the old systems and buy new. No need to upgrade the infrastructure each time which saves resources (time and money).

Plus, the initial cost is much less as well.

Go Apple! :D

Sushi

sushi

Sep 28, 2004, 11:33 PM

BeoVir

Sep 28, 2004, 11:34 PM

So I understand that ECC memory somehow portects all those little 'bits' in memory but I am wondering how often they are lost in non-ecc ram? I mean wouldn't our computers always crash if this was something that actually happened alot?

Sir_Giggles

Sep 28, 2004, 11:57 PM

This is a good day for Mac supercomputing. Maybe Virginia Tech's cluster will hit 12 teraflops with this upgrade. Over time, I can see this growing to 15, 20, 30, 40 teraflops as Apple releases new XServes. By the time it hits 40 teraflops - bye bye #1 spot for the Earth Simulator!

Not anymore. IBM's Blue Gene succeeds the Earth Simulator as the fastest supercomputer in the world.

I read it here. http://www.nytimes.com/2004/09/29/technology/29computer.html

virividox

Sep 29, 2004, 12:20 AM

daym i wonder what the heck vtech needs all thsi computing power for

MrCommunistGen

Sep 29, 2004, 12:46 AM

Now, then, the question is what type of chip the 2.3GHz is...

Pretty sure its the fx. Even the 2.0Ghz XServes are FX's so it would be kinda dumb if Apple's new and improved G5 was running on a 130nm process. :D

-mcg

MacinDoc

Sep 29, 2004, 01:04 AM

Do you think that it's a coincidence that the company that Dr. V. is with is pushing Xeons and Itania, including the world's fastest cluster (Itanium, not PPC970)? (http://www.californiadigital.com/)

The company is pushing BOTH platforms. And the fact that the Itanium cluster is the world's fastest cluster might have something to do with the fact that it includes nearly 4100 Itaniums.

MacinDoc

Sep 29, 2004, 01:22 AM

Did you also see that Dell is selling pre-packaged 64-bit Xeon clusters based on the same Infiniband networking that VAtech is using? (http://news.com.com/Dell%2C+Topspin+tout+InfiniBand+clusters/2110-1010_3-5387085.html) The "cheap supercomputer" just got a lot cheaper!
Those clusters will start at $55000 for eight-server clusters. Are you sure that's a lot cheaper? One DP XServe and seven DP cluster nodes would be $25000, less than half the cost!

(Sorry for the double post - didn't know how to add a second quote with your link when I tried to edit my last post.)

nagromme

Sep 29, 2004, 02:00 AM

...I bet that you're right that there are a bunch of angry scientists. Angry that the university spent millions on a PR "proof of concept" that was unusable from the get-go...

The deadline last year wasn't just for the Top500 and it wasn't just for PR (although the need for PR is a fact of life too). It was a funding deadline for the National Science Foundation (their Cyber Infrastructure program).

The company VT FIRST wanted to use--Dell--was unable to offer sufficient power for the available budget, even though Dell was offering a special pricing deal just for VT. Only Apple was able to offer sufficient power for a low enough cost and on time to get the NSF funding--even though Apple charged VT full education price.

(But Apple WAS willing to cut deals like Dell the second time, clearly. The question I've never seen answered is whether an Xserve deal was worked out with Apple from the very start or not.)

And yes, silly though it may seem, benchmarks and meeting a certain date is what the funding hinged upon. VT wasn't in control of that system. I'm sure they'd have rather waited for XServe G5s, but they simply didn't exist yet.

Here's an article explaining some details, and touching on VT's "Deja Vu" fault-tolerance software.

http://www.unirel.vt.edu/vtmag/winter04/feature1.html

"We believed that we could build a very high performance machine for a fifth to a tenth of what supercomputers now cost, and we did."

Doctor Q

Sep 29, 2004, 02:28 AM

(Sorry for the double post - didn't know how to add a second quote with your link when I tried to edit my last post.)For future reference, here are two techniques I've used:

1. To avoid having to retype the text you want to quote, control-click on the Quote button for the first post, and pick Open Link in New Window. Select the content and copy it to the Clipboard. Do not click the Submit Reply button! Close that window to return to the thread. Click Quote for the second post, and paste from the Clipboard at the top, above the second quote. Then add your comments.

2. More simply, you can type the "QUOTE=" syntax yourself and cut and paste the content from the thread. The syntax is as follows:

[QUOTE=membername]blah blah blah[/QUOTE]

Just be sure to quote correctly and attribute it to the right member.

Zaty

Sep 29, 2004, 02:49 AM

The 2.3 GHz Xserves make sense to me. As we all know, there's a 500 MHz gap between top-end and mid-range PMs. So, I've been wondering for quite while what happens to all those 970FX rated between 2.1 and 2.4 GHz. This seems to be the answer.

cc bcc

Sep 29, 2004, 03:52 AM

So, I've been wondering for quite while what happens to all those 970FX rated between 2.1 and 2.4 GHz. This seems to be the answer.

VT uses just 2200 cpu's, so this can't be the answer.

ssamani

Sep 29, 2004, 04:07 AM

Category: Apple Hardware
Link: Virginia Tech G5 supercomputer upgraded to 2.3GHz Xserves (http://www.macbytes.com/link.php?sid=20040928154926)
Posted on MacBytes.com (http://www.macbytes.com)

Approved by Mudbug

When do these boys ever do any work on their machine? They always seem to be upgrading it; from PowerMacs to XServes to faster XServes. I suppose with the physical space saved from the first upgrade they had enough room to install the 2.3GHz XServes with the old one still in place. Where do they get the money and time?

Sanj

nagromme

Sep 29, 2004, 04:32 AM

But did they REALLY ever have 2.0 Xserves? I thought the racks were sitting empty for quite some time after the PowerMacs were gone. And I never saw any news of "Xserve update complete" until several weeks ago--with 2.3s.

Someone at AI said that they called VT and was told a complete Xserve upgrade was done in January. But most things I read still suggest that Big Mac has only been upgraded once, and that January was just when they announced their plan.

Zaty

Sep 29, 2004, 04:41 AM

VT uses just 2200 cpu's, so this can't be the answer.

Okay, then it's part of the answer. However, we know that supplies of 2.5 GHz is still pretty limited, so we don't know how many 2.0 + GHz CPUs IBM can ship to Apple. Perhaps, VA Tech is not the only customer getting 2.3 GHz Xserves.

thatwendigo

Sep 29, 2004, 05:47 AM

This makes me want to see what they can do when System X is upgraded in capacity, and not only in current hardware. Virginia Tech's TerraScale Computing Facility is already working out plans to boost the overall size of the cluster from current levels. The current system is known as System X, but it's derivatives will be called System C and System L, in tune with roman numeric systems.

Go on, do the math...

They're talking about a five and tenfold increase in the servers, to 5,500 boxes and 11,000 boxes. If COLSA is any indication, that could lead to an enormous boost in overall power, since the Mach5 cluster is a bit less than twice as fast with an addition of a mere 400-500 machines.

Rod Rod

Sep 29, 2004, 06:01 AM

I agree with thatwendingo that it'll be interesting to see how fast Mac clusters can become with 5x and 10x the nodes as the current System X.

However, the forthcoming IBM Blue Gene/L system will be just sick.

A large-capacity version of the Blue Gene/L system is scheduled to be installed early next year at the Lawrence Livermore National Laboratory in Livermore, Calif. That machine will have about 130,000 processors, compared with the 16,000-processor prototype that set the speed record.

In other words, IBM just surpassed NEC's Earth Simulator with 16,000 processors at 36 teraflops. IBM's forthcoming 130,000 processor machine will be amazing. 8.125x the power makes for 292+ teraflops if it scales linearly, which of course it likely won't. Even at 200+ teraflops, wow.

Put another way, IBM's "pro" line of supercomputers is Blue Gene/L, and the "consumer" line is PPC970fx / Apple Xserve G5. Blue Gene/L is haute couture and Xserve is off the rack (no pun intended).

HiRez

Sep 29, 2004, 07:19 AM

Put another way, IBM's "pro" line of supercomputers is Blue Gene/L, and the "consumer" line is PPC970fx / Apple Xserve G5. Blue Gene/L is haute couture and Xserve is off the rack (no pun intended).
Sure, but how much does it cost (both to build and to operate). Let's see lists of the top purchase and operational price per performance ratios. I bet Apple would dominate there, even based on the current, relatively-primitive G5 core (compared to a Power-5-derived core which we'll probably see late next year).

thatwendigo

Sep 29, 2004, 07:52 AM

In other words, IBM just surpassed NEC's Earth Simulator with 16,000 processors at 36 teraflops. IBM's forthcoming 130,000 processor machine will be amazing. 8.125x the power makes for 292+ teraflops if it scales linearly, which of course it likely won't. Even at 200+ teraflops, wow.

The COLSA Mach5 cluster is hitting about 25 teraflops with 1,566 nodes, at a cost of roughly $7 million for the entire installation process. The Earth Simulator hits 35.86 teraflops with 5,120 processors and a cost of over $350 million. BlueGene L doesn't have a stated cost yet, but assuming a 50% scalability from its 11.68 terraflop, 8162 node performance legel (http://top500.org/list/2004/06/), you'd get 93.44 terraflops at 130,592 nodes. If it were to scale at 75%, you'd see 140.16 teraflops.

By contrast, System X got 10.28 teraflops at a cost of $5 million and using the previous generation of xServes. If you expand that by a factor of five, you'd see a performance boost to 25.7 with a 50% scalability, or 38.55 teraflop with a 75% scaling. At that step, the cluster would cost something like $25 million if one assumes that all expenses had to be encountered again (which likely isn't the case). This also hammers the hell out of the LLNL Tiger4 Itanium cluster, which costs $20 million and scores less than 20 teraflops. Take it another iteration to 11,000 nodes and you encounter a theoretical 51.4 teraflops at 50% scaling, or 77.1 teraflops for 75% scaling, at a cost of $50 million (with the same assumption about expanding costs, which is still unlikely to be true).

Mathematically:
System X (1,100) - 10.28/5 = 2.056 teraflop per million spent
COLSA Mach5 (1,566) - 25/7 = 3.57 teraflop per million spent
NEC EarthServer (5,120) - 35.86/350+ = 0.102 teraflop per million spent
BlueGeneL Prototype (8.162) - 11.68/5 = 2.336 teraflop per million spent (assuming the generously low $5 million pricetag)
Lawrence Livermore Tiger4 Itanium2 (4,096) - 19.94/20 = 0.997 teraflop per million spent

Are we noticing a trend here on bang for buck?

bunkre

Sep 29, 2004, 08:58 AM

It takes more than a processor upgrade to turn a PowerMac into an Xserve. I'm pretty sure they got more than new CPU chips :)
VT sold their G5 PowerMac cluster nodes. They were available through (I think) MacMall for a few months. So they didn't lose their investment. IIRC, they actually sold for close to Apple's original price - people were willing to pay that much because the computers were virtually new and included some certificate of authenticity stating that the computer was a VT cluster node. (Yes, VT didn't get the full purchase price, since the store took a cut, but VT also didn't pay full retail price - they paid Apple's educational institution price.)

yeah, my boss bought one for the office as a refurb (macwarehouse maybe?) he kept all the packaging, but intentionally threw out the certificate. i have no idea why...

anyway, it has that boisterous fan issue and it's sitting on an adjacent desk about 3 feet away. LOUD.

AidenShaw

Sep 29, 2004, 09:33 AM

That's patent nonsense. The original cluster was not "so unstable that it was unusable for doing any real work" by any stretch.

Sorry, but I've talked to some of the engineers from Mellanox, and they've described the problems with stability. It's no secret inside the IB community that the first cluster was never put into production use.

Also, your descriptions of "voting" schemes for error detection are really simplistic. You seem to assume that any memory error will show up as an error in a calculation.

That's not at all the case. Some will show up as math errors, other will manifest themselves as kernel panics or program crashes. Some will corrupt files or send bad data over the network, and the really nasty ones will cause wrong branches to be taken on code paths - so that almost anything could happen as the program (or OS) goes screaming down the wrong code paths.

All the descriptions of Déjà vu (like http://www.apple.com/education/science/profiles/vatech/scheduling.html and http://www.vtnews.vt.edu/Archives/2004/June/04263.htm) describe it as "transparent parallel checkpointing and recovery" - in other words it deals with system failures and restarts a job. Nothing is ever mentioned about voting schemes and "software ECC" - nothing.

Remember that the VAtech cluster was dismantled and sold almost as soon as the XServe G5 was announced.

It would have been quite easy to run the cluster, and rack-by-rack replace the Power Macs with the XServes - little or no downtime would be needed.

Instead, the cluster was completely dismantled and has been offline for the last nine months.

Can you explain why they would have taken a "usable" cluster offline for nine months? I can't explain that - and therefore find it much easier to believe the stories from the Mellanox guys that it was unstable to the point of unusable.

They would not have pulled the plug on a working multi-million dollar machine.

They would not have suffered the embarrassment of falling completely off the Top500 list unless there were real problems with usability.

AidenShaw

Sep 29, 2004, 09:57 AM

The COLSA Mach5 cluster is hitting about 25 teraflops with 1,566 nodes, at a cost of roughly $7 million for the entire installation process. The Earth Simulator hits 35.86 teraflops with 5,120 processors and a cost of over $350 million.

By contrast, System X got 10.28 teraflops at a cost of $5 million and using the previous generation of xServes.

The $/FLOP comparisons are mostly nonsense, because different methods of calculating the cost are used for each system.

The Earth Simulator price covers the entire building, including special seismic bracing. It's a "bare lot to supercomputer" pricetag.

The Virginia cluster $5.2M figure doesn't include things like the air conditioning and power - it leaves out much of the expenses that are included with other systems.

So, yes it was a bargain - but calculating $/FLOP isn't that meaningful when different methods of arriving at the cost are used.

Mathematically:
486DX2 66MHz ($1 on eBay) (1) - 0.00000240/0.000001 = 2.4 teraflop per million spent ;)

(And, BTW, it used Power Mac G5 systems, not an older XServe.)

thatwendigo

Sep 29, 2004, 10:01 AM

The Virginia cluster $5.2M figure doesn't include things like the air conditioning and power - it leaves out much of the expenses that are included with other systems.

Source, please. As I remember reading it, the price tag includes the HVAC and other considerations for the cluster.

(And, BTW, it used Power Mac G5 systems, not an older XServe.)

Oh, yeah. That's what I get for posting early. :o

AidenShaw

Sep 29, 2004, 10:19 AM

Source, please. As I remember reading it, the price tag includes the HVAC and other considerations for the cluster.

http://www.computerworld.com/softwaretopics/os/macos/story/0,10801,86704p3,00.html

The system costs $5.2 million for the G5s, racks, cables, and Infiniband cards.

Virginia Tech spent an additional $2 million on facilities, $1 million for the air conditioning system and $1 million for the UPS and generator.

So, $9.2M (or maybe $7.2M, depending if they meant "facilities=air+power" instead of the "facilities+air+power" that they wrote) and we haven't factored in building and real estate costs, the "slave labor" and other things unique to the VAtech cluster.

Another http://www.nwfusion.com/news/2004/0531environmental.html : "Shinpaugh says the school spent about $2 million for the cooling devices and adding power"

So, it was a bargain, but please don't create tables showing $/FLOP :mad: .

Editor's note: An obvious typo was corrected in the first quote.

Rod Rod

Sep 29, 2004, 11:43 AM

The COLSA Mach5 cluster is hitting about 25 teraflops with 1,566 nodes, at a cost of roughly $7 million for the entire installation process. The Earth Simulator hits 35.86 teraflops with 5,120 processors and a cost of over $350 million. BlueGene L doesn't have a stated cost yet, but assuming a 50% scalability from its 11.68 terraflop, 8162 node performance legel (http://top500.org/list/2004/06/), you'd get 93.44 terraflops at 130,592 nodes. If it were to scale at 75%, you'd see 140.16 teraflops.

I don't know why you're extrapolating performance from the 8,192 processor prototype. Really, please read the New York Times article. You don't even have to register at nytimes.com because c|net has the article too (http://news.com.com/IBM+supercomputer+sets+world+speed+record/2100-1006_3-5388015.html?tag=nefd.top).

It makes much more sense to begin your extrapolation from the 16,000 processor system rather than the 8,192 processor one.

From the article it's not clear what type of processor is in IBM's 16,000-processor, 36-Tflops system, but look: It's roughly double the processors and TRIPLE the Tflops compared to the prototype.

The new system is notable because it packs its computing power much more densely than other large-scale computing systems. Blue Gene/L is one-hundredth the physical size of the Earth Simulator and consumes one twenty-eighth the power per computation, the company said.

HiRez, my bets are on the Blue Gene/L for lower costs. 28x the power efficiency and 1/100th the size should make for some savings, sort of like comparing vacuum tubes with transistors.

AidenShaw

Sep 29, 2004, 12:48 PM

HiRez, my bets are on the Blue Gene/L for lower costs. 28x the power efficiency and 1/100th the size should make for some savings, sort of like comparing vacuum tubes with transistors.

Amdahl's Law would favor the Earth Simulator - since its processors are 30 times faster than Blue Gene/L processors.

The LINPACKD benchmark is very good for parallelization - it can be carved into as many pieces to run in parallel as you want.

For applications that aren't as easily parallelized - say those that can't be cut into more than a few hundred jobs - adding *more* small processors won't help, you want *faster* processors.

Just be careful extrapolating too much from one particular (and some say "peculiar") benchmark like LINPACKD.

HiRez

Sep 29, 2004, 01:20 PM

\HiRez, my bets are on the Blue Gene/L for lower costs. 28x the power efficiency and 1/100th the size should make for some savings, sort of like comparing vacuum tubes with transistors.
You could be right but even though IBM could be considered a competitor to Apple in this area, I'm rooting for IBM for any Power-based success because that success will eventually trickle down to the Mac. And not just Xserve-based superclusters, it should eventually benefit my lowly desktop PowerMac too :)

tortoise

Sep 29, 2004, 11:55 PM

Sure, but how much does it cost (both to build and to operate). Let's see lists of the top purchase and operational price per performance ratios. I bet Apple would dominate there, even based on the current, relatively-primitive G5 core (compared to a Power-5-derived core which we'll probably see late next year).

BlueGene is a totally superior architecture from top to bottom, and scalable in far more ways than a cluster. A cluster is NOT a supercomputer, and there are a lot of applications that will drag ass on a cluster (worse performance than a medium SSI box even) that will scream on a genuine supercomputer. Apples and oranges. Clusters are very limited in their application, true supercomputers generally are not.

At the end of the day, for most apps the specific core matters less than other architectural details such as memory and I/O architecture. The POWER series processors aren't better than the G5 because they have a radically superior core, but because they have superior memory and I/O architectures. What will be interesting to see is if IBM merges their POWER series into their PPC series eventually. The primary reason that the Opterons are a little faster than G5s (assuming Pathscale and IBM high performance compilers respectively) all things generally being equal is that the Opteron has some of the features that IBM reserves for their POWER series processors. As it stands the biggest limitation of the PPC processors is their markedly inferior memory subsystem compared to AMD and arguably even Intel. IBM knows how to make good memory subsystems, they just didn't put them on the PPC.

tortoise

Sep 30, 2004, 12:08 AM

This also hammers the hell out of the LLNL Tiger4 Itanium cluster, which costs $20 million and scores less than 20 teraflops.

LINPACK teraflops aren't real teraflops (more like DSP TFLOPS really), and so for many codes $20M worth of Itanium will soundly beat $20M worth of G5 -- Itanium is the king of general purpose floating point, and the memory subsystem is decent. (Actually, with the best compilers available for both architectures, Opterons have slightly better general purpose floating point performance than G5s.) LINPACK measures DSP-like floating point loads, a narrow application space that PPC does extremely well at. Apparently LLNL has one of the many, many supercomputing applications for which LINPACK is a ridiculously uncharacteristic benchmark. Many people do.

My supercomputing applications are similarly uncharacteristic, but in a different fashion, being memory intensive mixed loads. The Opterons thrash the G5s for these codes, and so we use big Opterons for supercomputing purposes. I still use Apple systems for most everything else though.

LINPACK TFLOPS are pretty marginal as a benchmark, and you'll find that the reason so many clusters are still built on other architectures is that other architectures often give superior bang-for-the-buck for the types of applications those clusters are being built for.

wrldwzrd89

Sep 30, 2004, 05:21 AM

LINPACK teraflops aren't real teraflops (more like DSP TFLOPS really), and so for many codes $20M worth of Itanium will soundly beat $20M worth of G5 -- Itanium is the king of general purpose floating point, and the memory subsystem is decent. (Actually, with the best compilers available for both architectures, Opterons have slightly better general purpose floating point performance than G5s.) LINPACK measures DSP-like floating point loads, a narrow application space that PPC does extremely well at. Apparently LLNL has one of the many, many supercomputing applications for which LINPACK is a ridiculously uncharacteristic benchmark. Many people do.

My supercomputing applications are similarly uncharacteristic, but in a different fashion, being memory intensive mixed loads. The Opterons thrash the G5s for these codes, and so we use big Opterons for supercomputing purposes. I still use Apple systems for most everything else though.

LINPACK TFLOPS are pretty marginal as a benchmark, and you'll find that the reason so many clusters are still built on other architectures is that other architectures often give superior bang-for-the-buck for the types of applications those clusters are being built for.
That explains why VTech went with PPC G5s (and others went with something else) - the chosen architecture performs better with the specific task(s) the creators of the cluster had in mind. I'm a little surprised to learn that Linpack FLOPs are not applicable to the real world in most cases.

AidenShaw

Sep 30, 2004, 07:43 AM

LINPACK TFLOPS are pretty marginal as a benchmark, and you'll find that the reason so many clusters are still built on other architectures is that other architectures often give superior bang-for-the-buck for the types of applications those clusters are being built for.

Another factor that makes the LINPACK results for PPC (and POWER) better than real-life performance is that the PPC ISA has a MADD instruction - a floating point MULTIPLY and ADD in one instruction.

To quote IBM:

The total cluster will have a theoretical peak capacity of over 40 trillion floating-point operations per second. The following table explains how theoretical peak capacity is calculated.

http://www-306.ibm.com/chips/products/powerpc/newsletter/jun2004/images/theo-peak_table_1b.gif

In practice, only a small portion of peak capacity is achieved because a processor is rarely scheduled to do simultaneous “multiply and adds” in double precision.

However, the LINPACK benchmark, which is often used to rank supercomputers (the Top500 Supercomuter list), makes extensive use of simultaneous multiply and add. Massively parallel systems regularly achieve between 50 and 90 percent of peak performance.

http://www-306.ibm.com/chips/products/powerpc/newsletter/jun2004/ppc_process_at_work.html

There's an article discussing other LINPACK issues and a new supercomputer ranking system at Supercomputer ranking method faces revision (http://news.com.com/Supercomputer+ranking+method+faces+revision/2100-7337_3-5238405.html).

AidenShaw

Sep 30, 2004, 07:52 AM

BlueGene is a totally superior architecture from top to bottom, and scalable in far more ways than a cluster.

This quote doesn't make it sound that "totally superior" at all:

Don Dossa, a computational physicist at the Livermore lab, said in an earlier interview that Blue Gene/L is well adapted for some calculations but not for others.

For example, it's appropriate for simulating materials in a relatively unchanging environment, such as a solid that's cracking under pressure. But when it comes to the nuclear weapons simulations at the heart of the lab's mission, the 512MB of memory in each Blue Gene/L computing node isn't enough to deal with the broad range of radiation, temperature and pressure conditions, Dossa said.

For those nuclear weapons simulations, IBM is building a system called ASCI Purple that uses a smaller number of more powerful computers that rely on IBM's Power5 processor.

http://news.com.com/IBM+claims+fastest+supercomputer+title--for+now/2100-7337_3-5388797.html

So, IBM is building a cluster with 12,544 POWER5 CPUs to do jobs that Blue Gene/L isn't suitable for....

tortoise

Sep 30, 2004, 02:44 PM

This quote doesn't make it sound that "totally superior" at all

Actually, I agree, and was mixing points. BlueGene is not really a general purpose supercomputer, though it does share some of the features of one. It is more of a hybrid architecture that splits the difference but is still optimized for a particular subset of applications.

One could argue that the determinant of what is and isn't a general-purpose supercomputer has less to do with how many processors it has and a lot more to do with memory architecture. Not having a single massive address space limits the applications you can use a supercomputer on effectively. Something like an SGI Altix system is more of along the lines of what I was thinking of. I think there is a trend toward more specialized architectures and applications in supercomputing, which is giving us things like BlueGene, Octiga Bay (now Cray XD1), with application optimized interconnects, FPGAs, etc.

JOD8FY

Sep 30, 2004, 08:33 PM

stephenli

Oct 2, 2004, 01:04 AM

Probably his beloved cube. Or a Lisa.

He is using a 17" Powebook i guess.....
he said that after the surgery.... :cool:

wrldwzrd89

Oct 2, 2004, 05:16 AM

VA Tech's system gets more updates than Mac OS X! :rolleyes:

I doubt that Apple would update XServes to 2.3Ghz from 2.0Ghz. My guess is they'll go to 2.5Ghz. Apple was probably planning on releasing 2.3Ghz XServes earlier this year but couldn't get enough supply of the chips from IBM, so they gave the ones they did have to VA Tech.

JOD8FY
There's a reason why Apple would go to 2.3 GHz rather than 2.5 GHz: cooling. Remember that the 2.5 GHz chip requires liquid cooling in the PowerMac G5. There isn't room for such a system in the XServe. That doesn't change the supply issues Apple/IBM are having with the 2.3/2.5 GHz chips right now, though.

AidenShaw

Oct 2, 2004, 09:07 AM

Remember that the 2.5 GHz chip requires liquid cooling in the PowerMac G5. There isn't room for such a system in the XServe.

Two points....

1. As has been said repeatedly, the liquid cooling in the PM 2.5 is to reduce the noise, not because there's some magical boundary somewhere between 2.3 and 2.5 that *requires* liquid cooling. The XServe makes a lot of noise, it isn't a big deal if it makes a little more because the fans need to spin faster for a 2.5.

2. While the big brick of a cpu/cooling assembly from the PM won't fit in a 1U rack, a redesigned unit could easily fit in the 1.5" space available in a rack system. Again, the lack of concern about the noise level means that a smaller radiator with higher speed fans, and a more powerful pump, could be used.

Note that you can buy 1U systems with dual 100+ watt 3.6 GHz 90nm Nocona CPUs (or a 1U with quad 90watt Opterons) - surely Apple's engineers could be as innovative as those on the x86 side....

wrldwzrd89

Oct 2, 2004, 10:23 AM

Two points....

1. As has been said repeatedly, the liquid cooling in the PM 2.5 is to reduce the noise, not because there's some magical boundary somewhere between 2.3 and 2.5 that *requires* liquid cooling. The XServe makes a lot of noise, it isn't a big deal if it makes a little more because the fans need to spin faster for a 2.5.

2. While the big brick of a cpu/cooling assembly from the PM won't fit in a 1U rack, a redesigned unit could easily fit in the 1.5" space available in a rack system. Again, the lack of concern about the noise level means that a smaller radiator with higher speed fans, and a more powerful pump, could be used.

Note that you can buy 1U systems with dual 100+ watt 3.6 GHz 90nm Nocona CPUs (or a 1U with quad 90watt Opterons) - surely Apple's engineers could be as innovative as those on the x86 side....
That disproves the cooling hypothesis....so there's GOT to be a reason why Apple's going with 2.3 GHz PPC G5s for the XServes...I just can't determine what that reason would be.

AidenShaw

Oct 2, 2004, 11:41 AM

That disproves the cooling hypothesis....so there's GOT to be a reason why Apple's going with 2.3 GHz PPC G5s for the XServes...I just can't determine what that reason would be.

A couple of possibilities:

1. The *existing* XServe heat sinks and cooling system can handle the chip at 2.3 GHz, but not the extra heat from running at 2.5 GHz. The chips at 2.3 could be stuck in with little effort, but 2.5 would have required re-engineering the cooling system.

2. Apple didn't have 2200 chips that would run at 2.5 GHz, and didn't want to shut down sales of the PM 2.5 in order to supply VAtech. They did, however, have a bunch of chips that wouldn't run at 2.5 GHz but which were OK at 2.3 - viola!

Note that there isn't a "2.5 GHz" manufacturing line and a "2.0 GHz" manufacturing line. There's one line, and all the different speed chips start out identically.

After the chips are manufactured, they are tested. Due to differences at the atomic level between chips from the same wafer, different chips from the same wafer will have different maximum speeds. You could easily have a 1.6 GHz chip come from right beside a 2.5 GHz chip on the same wafer.

Typically, you'll find that chips that will run at super-fast speeds are rare compared to slower chips.

Since Apple is currently has no product between 2.0 and 2.5 GHz, one can assume that there are a bunch of chips around that will run faster than 2.0 GHz, but won't quite make it to 2.5 GHz.

(IBM is only selling 1.6 GHz systems, although the Spanish cluster was described as using 2.2 GHz CPUs.)