Virginia Tech Supercomputer Cluster Photos

[MacRumors]

Virginia Tech posted an extensive collection (http://computing.vt.edu/research_computing/terascale/images/) of photos documenting the assembly and deployment of their Supercomputer Cluster which consists of 1100 PowerMac G5s.

Virginia Tech's Supercomputer cluster draw significant interest after it was revealed that it was to be composed of over 1100 PowerMac Dual 2.0GHz G5s (http://www.macrumors.com/pages/2003/09/20030902190056.shtml) at the end of August. Additional information is available at the Terascale site (http://www.computing.vt.edu/research_computing/terascale/) at Virginia Tech.<P>

The initial target date for start of operations is October 1, 2003.

[Chaszmyr]

That's the most beautiful thing I've ever seen


(Yet not beautiful enough to make me happy that my Dual G5 ordered on 6/23 still hasnt shipped)

[Doctor Q]

I want one! (The T-shirt, that is.)

[Sol]

Nine G5 PowerMacs per rack. Not bad. Hopefully this project will be an incentive for Apple to release G5 Xserves sooner rather than later.

[charlesc]

Originally posted by Chaszmyr
That's the most beautiful thing I've ever seen

ditto that.

Gotta love that shirt the guy is wearing!!!

[Powerbook G5]

Wow, that is so cool! It looks like something out of a sci-fi movie with all those G5s stacked next to each other wired together like that. I cannot imagine what kind of performance that setup will have or how loud/hot it would get. And to think, I am all excited over just a *single* 1.25 GHz G4!

[Waluigi]

I'm in more awe of this then when I went to Versallies.....

--Waluigi

[Powerbook G5]

Originally posted by Doctor Q
I want one! (The T-shirt, that is.)

I want one, too! (The 1100 node G5 cluster, that is.)

[Java]

If this isn't heaven, I don't know what is.

Style + power, one sweet combo!

[cb911]

cool!! i'm going to have to have a look at those photos when i get some time...

[Doctor Q]

Odd that they organized the photos alphabetically by category instead of chronologically.

[reyesmac]

When I see all those G5's, what I think about the most is how outdated they will seem in about 3 years. If IBM is going to be delivering G5's that are a full gigahertz faster every twelve months, then this supercomputer wont seem so super. Just imaging how fast 1100 Dual 4ghz G5's would be in just two years from now.

[Ramsos]

Man I would love to just have one of those racks:D

[SteveC]

One word: WOW! :cool:

As a long-time Mac faithful, this is glorifying to witness. :)

While I'm thinking of it... you know, one thing VT *could* also definitely use.... is a web server upgrade! ;) lol... I have a super fast broadband connection here... but the photos on the VT web site are taking *forever* to load. Wonder what the deal is.

Ahh well... that doesn't spoil my celebrating one bit. :cool:

[Ramsos]

I agree viewing that site took me back to my AOHell (DSL now) dial up days. Sloooow.

[Sol]

Originally posted by SteveC
While I'm thinking of it... you know, one thing VT *could* also definitely use.... is a web server upgrade! ;) lol... I have a super fast broadband connection here... but the photos on the VT web site are taking *forever* to load. Wonder what the deal is.

Well, the G5 super-computer is not used to serve their web-site, we know that much.

[Ramsos]

True true.

[skharris]

Its really a shame that people that ordered Dual 2Ghz G5's were bumped just for a project like this. - It basically sucks given the fact that people like myself are still waiting for our orders that where placed back in June! Either way, best of luck to the lab!

[plinkoman]

Originally posted by Sol
Well, the G5 super-computer is not used to serve their web-site, we know that much.

at the rate these pictures are loading, maybe they should...

[e2chris]

Lets hope the Web Server is Intel... my god is it slow... Broadband here as well.

[e2chris]

Looking at these pics I see why there was a delay in shipping them to people! VT didnt have to wait obviously. I am not complaining I just find it funny.

[e2chris]

Good thing I am not on the Project cause there would 1099 G5's not 1100. =P

[Powerbook G5]

I'm sure no one would notice if just one were missing... ;)

[DrGruv1]

I really didn't think of the support needed or the amopunt of racks, cables, space and other that were needed - impressive very cool. A rack of nine would do nicely for me :)

[arn]

Small Mirror:

http://www.chaosmint.com/mac/techclusterphotos/

[A@ron]

They are working on lots of equipment yet I cannot tell if those are grounding strips or wrist watches. God I hope they are grounding bands!

A@ron

[mvc]

Reminds me of the rack scene from the matrix - We need G5s - lots of G5s!

[pgwalsh]

Originally posted by Sol
Nine G5 PowerMacs per rack. Not bad. Hopefully this project will be an incentive for Apple to release G5 Xserves sooner rather than later. Exactly... They could fit a lot more into a much smaller area if they were all dual Xserver G5's. Any guestimations how many xservers would fit in one of those racks?

[me_17213]

I can't seem to get into the pictures part of the site.

It says I don't have permission.
Anybody else getting this message?

[chmorley]

Originally posted by me_17213
I can't seem to get into the pictures part of the site.

It says I don't have permission.
Anybody else getting this message? Yep. Got the same message.

Go back one page--arn posted the picture.

Chris

[TrenchMouth]

anyone know what sort of performance this thing with throw out? i was trying to find some info on the site. i know that its not an exact science but how much power are they looking for? i assume because of the name we are looking at a multi teraflop machine, but any numbers?

[PowerBook User]

That's nice they're putting pictures up. What was the largest number of Macs used in a cluster before this? I seem to remember a project called Appleseed where they would hook up numerous Power Macs. Also, I think Apple should sell that shirt on it's web site. They would sell a ton!

[tribalogical]

assuming the DP G5 Xserve would be a 2U box, maybe 15~20? Not sure how many U those racks are. (I'm visually estimating 5 or 6 per "block", but remove the shelves, and you could squeeze a few more in).

Are those standard rack towers? If they're 48U racks, then you could fit 24 @ 2U each....

It would still take 45.8 of those racks to assemble 1,100 machines...!

as it is, it takes *122* of those rack units to house the beast @ 9-per rack.... amazing quantities of hardware...

Hey, I'll have one of those, thank you! Let the market wait.... :) :)

Unfortunately, the pictures over there seem to have been removed or blocked....

Can anyone post more here?

tribalogical

[charlesc]

Originally posted by TrenchMouth
anyone know what sort of performance this thing with throw out? i was trying to find some info on the site. i know that its not an exact science but how much power are they looking for? i assume because of the name we are looking at a multi teraflop machine, but any numbers?

This will be a very interesting figure when they get it optimized. Makes me wonder how many man-hours in took from start to finish, and how many total people have worked on it thus far. Even with a crew of people it still took some serious time to assemble all those racks, modify all those boxes, not to mention the planning phase, etc.......

[neutrino23]

Wasn't it posted somewhere that they had ordered these before they actually came out so that in fact no one was bumped?

Also, it makes me think that if they ordered these computers the G5 Xserve is not coming out before January. Perhaps the Xserve will wait for the 90nm 970s?

[charlesc]

Can anyone post more here?


this is the only one I got. before they shut down access. Damn pics averaged 300k each. That is a lot of data being transferred. Just run a fiber line into that cluster..

[janey]

Originally posted by chmorley
Yep. Got the same message.

Go back one page--arn posted the picture.

Chris

Just got that message. Only got through the G5 Racks and half of the assembly thing...here's another pic-i'm guessing these are the custom built racks for the G5s :)

eh nevermind my iDisk is going all weird...

[arn]

Small mirror of 3 pics

http://www.chaosmint.com/mac/techclusterphotos/

[Sol]

Originally posted by neutrino23
Also, it makes me think that if they ordered these computers the G5 Xserve is not coming out before January. Perhaps the Xserve will wait for the 90nm 970s?

90nm 970s might be necessary for G5 PowerBooks but Xserves have a bit more space to utilise. Apple could be waiting for PowerMac sales to settle down and/or OS X Server 10.3 to become ready. There must also be enough CPUs from IBM for Apple to announce a hot (as in, everyone-wants-one hot) product like this. If this cluster is a success there will be a lot of universities eager to replicate it.

[charlesc]

The 'slideshow' has some great information on the project. Safari doesn't like it though, looking at it in ie.... That's just wrong.
http://computing.vt.edu/research_computing/terascale/tcfslides_files/frame.htm

[Doctor Q]

Originally posted by arn
Small mirror of 3 pics

http://www.chaosmint.com/mac/techclusterphotos/ Your site is pleasantly speedy, arn, in contrast to VT.

I suspect that the Cluster Photo category pages with the thumbnails were pre-loading the full-size photos, which would explain why the category pages were so slow to load while the full-size photos came up relatively quickly.

[T.Rex]

I wonder if all those computers have optical drives, or if Apple built them special without them? Seems like an extreme waste if all those computers have drives that will never ever be used - and for that matter, video cards.

...At the very least I hope they are only combo drives and not superdrives! The horror!

[Tastannin]

Don't those racks actually hold 12 G5's rather than 9? Look at the 3rd pic of Arn's mirror.

Just a minor quibble. :)

Other than that, it's awesome, I agree.

[Sol]

Originally posted by T.Rex
I wonder if all those computers have optical drives, or if Apple built them special without them? Seems like an extreme waste if all those computers have drives that will never ever be used - and for that matter, video cards.

One of the benefits of a cluster-based supercomputer would be that once it is time to upgrade its components, the old computers that were part of it could be sold. So if in two years the students of this university are offered the G5 PowerMacs from these pictures they would rather have SuperDrives and video cards included.

[machinehien]

wow, I feel so deprived at this moment

[macphoria]

That is so awesome! I want that setup in my room!

[Doctor Q]

Originally posted by Sol
One of the benefits of a cluster-based supercomputer would be that once it is time to upgrade its components, the old computers that were part of it could be sold.Are they going to have a problem if the next generation Macs come in a different form factor? One of the Xserve benefits, of course, is the standardized size (1U high, width for 19" rack) and V.T. doesn't have that benefit with these Power Macs.

[iSmell]

Originally posted by T.Rex
I wonder if all those computers have optical drives, or if Apple built them special without them? Seems like an extreme waste if all those computers have drives that will never ever be used - and for that matter, video cards.

...At the very least I hope they are only combo drives and not superdrives! The horror!

It looks like they do have video cards from the picture. Say they're worth $100 each, that's $110,000 down the drain. I bet they have super drives, too. Bastards.




My eMac feels so much more than one year old.

[macphoria]

It looks like they do have video cards from the picture. Say they're worth $100 each, that's $110,000 down the drain. I bet they have super drives, too. Bastards.
I would love to see Unreal Tournament frame rate using this set up. Few hundred fps at 1024x768?

[arn]

Originally posted by iSmell
It looks like they do have video cards from the picture. Say they're worth $100 each, that's $110,000 down the drain. I bet they have super drives, too. Bastards.


you can BTO downgrade to a combo drive.

arn

[3.1416]

Originally posted by Sol
Nine G5 PowerMacs per rack. Not bad.

I count 12 per rack (4 rows, 3 each). It's really impressive how they're doing this with off the shelf machines intended as desktops/workstations. If it works out Apple will get some fantastic PR.

[3.1416]

Originally posted by TrenchMouth
i assume because of the name we are looking at a multi teraflop machine, but any numbers?

Probably not until they actually have it up and running. But we can estimate; they have 2200 G5s at 2 GHz each. Even if overhead is such that each processor only averages 1 operation per second of "real" work, that's 4.4 teraflops. If they can take advantage of Altivec, that number will be much higher.

[nagromme]

It's good PR for Apple that a supercomputer can be built from G5s and used for such high-end research applications.

It's even BETTER PR that doing so offered more power for less money than all other hardware and OS options, including Dell and including Linux PPC--and as VA Tech announced, that's exactly the case, even though Dell offered special pricing.

But it's better PR STILL that VA Tech had such a tight deadline, and managed to build a homemade 1100-node supercomputer in a few short weeks!

(I wonder what vendor is providing the secondary storage? XServe RAIDs or something else?)

[cbfro]

VT's internet connection may be suffering from the hurricane, cable internet has been off all day in Blacksburg (town VT is located) and not to mention all of us mac lovers that have bombarded that site trying to see pictures.

Like I have posted before, those machines do have video cards in them bc they are most likely going to be resold to students. All of the boxes are being stored to repackage the units in when they are replaced. Unfortunately I never got to assist in the assembly because unboxing and installing the infiniband cards went extremely fast (296 units processed in 2 1/2 hours).


But hey, I still get one of those sweet tshirts for volunteering.

[MacBandit]

I sure hope they're putting all the heat produced by those space heaters to good use.;)

[mgescuro]

If you look closely at the photo... it looks like 12 G5's per rack...


Originally posted by charlesc
this is the only one I got. before they shut down access. Damn pics averaged 300k each. That is a lot of data being transferred. Just run a fiber line into that cluster..

[nagromme]

They ought to lack optical drives and have the most bare-bones graphics card if any (one you can't BTO officially). They ought to have shipped at a discount without keyboards and mice, too.

But my guess is, VT wanted these NOW--they had an Oct. 1 deadline, and Apple was still the best power for the money. So they accepted standard configs rather than wait. Those extras are a waste in a sense, but I bet it was a deadline issue.

[gerardrj]

Originally posted by neutrino23
Also, it makes me think that if they ordered these computers the G5 Xserve is not coming out before January. Perhaps the Xserve will wait for the 90nm 970s?
Not really. Knowing the dealine that this system needs to be operational (not necessarily optimized) by Oct 1 for it to be included in the next super computer listing.
What that means is that the G5 X-Serves just wouldn't be shipping in time to be installed and working as a cluster by then. Apple could announce G5 1U servers next week and that would have been too late for the goals of this project.

[madrobby]

Originally posted by charlesc
The 'slideshow' has some great information on the project. Safari doesn't like it though, looking at it in ie.... That's just wrong.
http://computing.vt.edu/research_computing/terascale/tcfslides_files/frame.htm

That's because it's made with Powerpoint and uses mostly Microsoft-specific html-tags. Next time, they should try Keynote and save a nice PDF or Quicktime.

[Belly-laughs]

Originally posted by reyesmac
When I see all those G5's, what I think about the most is how outdated they will seem in about 3 years. If IBM is going to be delivering G5's that are a full gigahertz faster every twelve months, then this supercomputer wont seem so super. Just imaging how fast 1100 Dual 4ghz G5's would be in just two years from now.

Yes, quite right. But what´s now interesting is to see when a single processor mac will be as powerful as this cluster. In 10-15 years perhaps?

A modern chip will always be compared with the supercomputers of the past:

... a single G4 could be used to run the entire Apollo programme...

Now we´ve got our own modern supercomputer to compare with!

And, yes the G5´s look really good side-by-side. Funny Apple didn´t design a nice rack for them.

In a couple of years when they ditch the old to get up-to-date nodes, imagine the second hand ad...

"Previous Supercomputer seeks new homes!"...

[tazznb]

I bet their just going to use this setup just for... for... for photoshop or something
(Just joking) :p

[Kai]

Trenchmouth:

We don't have to guesstimate Rpeak since it's just simple multiplying of known numbers! ;-)
The G5 can do 4 64bit (double Precision) FPops per Cycle due to 2 FPUs and the PPCs Multiply-Add Command, which counts as 2 FPops.

4 FPops/Hertz * 2 GigaHertz * 2 CPUs/Node * 1100 Nodes = 17.6 TFLOPS Rpeak

Now, Rpeak is just -as always- the THEORETICAL Maximum, keep that in mind! The current #2 in the Top500 "only" has 20.4 TFLOPS, too! It's just a question how close they can get to Rpeak with Rmax...

Linpack uses 64bit-Floats, but if they find Projects that can do with Single Precision (32bit) Floats, they can employ Altivec, too!

And Altivec can do 4 Multiply-Adds per Cycle! That means just using Altivec alone (think NASA, think Jet3D!) this thing could reach 35.2 TFLOPS!

Now, if they manaage to mix scalar and vector code nicecly scheduled and employ BOTH the FPUs and Altivec, we could even assume a total of 52.8 TFLOPS Rpeak (The current #1 in the Top500 has 40.9 TFLOPs Rpeak!). but i consider it unlikely they will get very close to THIS Rpeak because it's pretty tricky to keep ALL Units fed constantly without any Pipeline stalls or delays! ;-)

Now, as for the "superfluous" GFXcards: There are quite a few projects underway where Universities work with the new programmability of current GPUs (Shaders) for crunching research data. Just imagine what would happen if VUtech managed to also use THIS! ;-) These Things have AGPx8, which is 2GByte/s, a pretty handy bandwidth (though i don't know how fat the "backchannel" to the CPU/RAM/IOsubsystem is, which has been a major obstacle in the past for using GPUs for other purposes as GFXcards are traditionally designed to be "one-way"!). A GeForce FX has 51 GFLOPS (one G5 2GHz "only" has 24 GFLOPS, 8 for the FPUs and 16 for Altivec!), and the 9600 is probably just as good!

[bobindashadows]

Abruptly, a week after the cluster is assembled and operational, VT requests a return:

"It has come to our attention that on a mac, you in fact cannot play counterstrike, thereby making all these things useless. Please accept our request for a full refund...."

[madrobby]

There are quite a few projects underway where Universities work with the new programmability of current GPUs (Shaders) for crunching research data.

I hope the card manufactures don't play tricks anymore - benchmark tweaked, inaccurate drivers may be cool for showing of Quake, but when doing cancer research... :rolleyes:

I'd rather stick to "Cure for Cancer" in Civ 3... :D

[Macmaniac]

Maybe they should hook up 1100 moniters and have a huge warcraft 3 tourney:)
Its just amazing I hope they get it up and running in time to be on the October list.

[droid]

Minor point to note, but they can fit 12 per rack.

You can see it best in this picture:
http://www.chaosmint.com/mac/techclusterphotos/rack3.jpg

[whfsdude]

geek porn :D

[MacEyeDoc]

I think I see the one I ordered on the second row from the top, 1st pic. Maybe he's happier with the group he's in than at my house. You can always come home if you want too!

[missing2ghz]

I placed my order for a 2ghz G5 within 30 minutes of the announcement and I still don't have it!

[Flynnstone]

VT is keeping all the boxes and package :)
The PowerMacs are standard configuration :)
In a year or two Apple will come out with a Power Mac with (hopefully) an IBM 980 dual core hot rod of some sort, that takes another jump in performance :)

Perhaps the VT students will be able to purchase these units as VT revamps (& reinvests) in their cluster :)

Or we could see them all on eBay :p

[MacKenzie999]

There really is nothing like a nice rack...

[ITMediaCo]

now who is going to push all 1100 of those power buttons?

In the photo that shows the back of the machines it appears that the dual ADC/DVI video card is installed. I wonder if Apple shipped these Power Macs with iMovie...

[saabmp3]

I don't know why people are saying that use Mac users could kill their entire internet connection. That place (judging by my school) must have ATLEAST 2 0C-3's and multiple T1's as backups. I bet it has something to do with the hurricane cause we sure arn't doing it.

BEN

[FlamDrag]

It's also interesting to note that the new G5 form factor allows for a more densely populated rack. That is to say, since the G5's are not rounded like the G4 towers.

[bredlo]

Has anyone heard if these machines are stripped down on the inside?

If not, that's a HUGE, GIGANTIC, MASSIVE, ENORMOUS, TITANICALLY BIG waste of all those firewire and USB interfaces, high end video cards, even ethernet plugs! I would think Apple could've saved themselves (and Virginia Tech) a ton of cash over the span of all that wasted technology just sitting in there by leaving those elements out during assembly.

However, that would make the machines practically useless if and when the school ever decided to disassemble the cluster in the future and move the machines into other uses, such as in computer labs.

Just wondering aloud.

[Rocketman]

Originally posted by macphoria
That is so awesome! I want that setup in my room!

You're going to need a bigger room. About 3000 square feet ought to do.

Rocketman

[madrobby]

Originally posted by bredlo
Has anyone heard if these machines are stripped down on the inside?


In the presentation they state the configuration as 4GB RAM ea., 160GB S-ATA Storage ea. -> that amounts to 176TB total storage with 4,4TB of RAM. They state there are 4 head nodes, so I assume the rest of the nodes are headless?

Every node uses the built-in gigabit ethernet as "secondary communications channel", while the primary interconnect is with specially added Infiniband adapters (more than twice the Speed of Gigabit Ethernet).

I also assume it would cost more to remove stuff like firewire connectors than to just leave it in.

Sidenode: There planning on a second cluster to go online 2006. So guys, there will be a nice wait when you're going to order that new G6... ;)

[Rocketman]

Originally posted by bobindashadows
Abruptly, a week after the cluster is assembled and operational, VT requests a return:

"It has come to our attention that on a mac, you in fact cannot play counterstrike, thereby making all these things useless. Please accept our request for a full refund...."

Sudden rush PO for 1100 copies of VPC at edu pricing :)

[dho]

Just imagine all the possible internet bandwidth if all the g5's in VT dialed into the internet each with just a 56k modem.
~5kBps x 1100 = 5500 kBps

That is how the should fix the VT bandwidth issue;)

Headline: "Today, aol has tripled their dialup users":D

this is just how i think

[benski]

Virginia Tech, as I remember, has 3 oc11 lines. They aren't hurting for bandwidth. I used to cap out my 10mbit connection frequently when connecting to some friends at Georgia Tech (also on Internet2) =)

However, their backbone goes up along I-81 and meets up with a major router in Washington, DC. I'm sure Hurricane Isabel has knocked a few of these connections out and their are probably running on backup bandwidth.

[e2chris]

Well the way that www was working last night(while it was up) Maybe they are using dial-up. LOL

[Photorun]

Man that's cool! Er, it's definitely NOT cool i.e. heat, odd how they mount those dual fan things in the ceiling above the racks, I wonder if those are venting air out or in?

One thing I'm confused about is all the blue wiring just looks like your basic 10baseT Ethernet cable you'd buy at Staples to connect them all. I'd think you'd need or want optical cabling to pipe everything to supercomputer status?

[Appleseed]

Originally posted by skharris
Its really a shame that people that ordered Dual 2Ghz G5's were bumped just for a project like this. - It basically sucks given the fact that people like myself are still waiting for our orders that where placed back in June! Either way, best of luck to the lab!


I don't think anyone got bumped because of this. I'm sure this was in the works far before Apple even officialy announcd the G5's. So, it's not like they started shipping and along comes this university that order 1100 units that causes other peoples orders to be sent to the school instead.

[AidenShaw]

Originally posted by Photorun
One thing I'm confused about is all the blue wiring just looks like your basic 10baseT Ethernet cable you'd buy at Staples to connect them all. I'd think you'd need or want optical cabling to pipe everything to supercomputer status?


Cat6 cable looks just like Cat3....

The cluster uses GigE for normal TCP/IP traffic, that's probably the blue cable. GigE over fibre is a lot more expense (a *lot* more) for exactly the same speed.

Even Fibre Channel is often copper-based - you don't use optical unless you need to!

The InfiniBand fabric is also copper-based, but it doesn't look like much is installed yet... The GigE cables are dangling, the power cords are dangling....

[tortus]

Look at all those empty PCI slots and wasted aluminum. Time for some XServes with g5s.

[AidenShaw]

Interesting that the slideshow talks about 64-bit systems (the Dell bid was Itanium II) at length....

And then it says that they are using a 32-bit operating system! Why insist on a 64-bit chip, but settle for a 32-bit OS???

Something doesn't connect here....

[tduality]

Originally posted by MacEyeDoc
I think I see the one I ordered on the second row from the top, 1st pic. Maybe he's happier with the group he's in than at my house. You can always come home if you want too!

You know, one day you have to let them go, to let them live their one life. Sigh.

;)

[tokyo2002]

WOW, I need 1100 G5's!!!

[niter]

Originally posted by Flynnstone
VT is keeping all the boxes and package :)
The PowerMacs are standard configuration :)
In a year or two Apple will come out with a Power Mac with (hopefully) an IBM 980 dual core hot rod of some sort, that takes another jump in performance :)

Perhaps the VT students will be able to purchase these units as VT revamps (& reinvests) in their cluster :)

Or we could see them all on eBay :p

This is my first post via my "new" PowerMac G4. I am officially an apple owner today thanks to Tech. I picked up my computer at Tech's surplus auction. You can count on it that I will forever watch the list for when those G5 start trickling into surplus.

[big]

Man that's cool! Er, it's definitely NOT cool i.e. heat, odd how they mount those dual fan things in the ceiling above the racks, I wonder if those are venting air out or in?


remember, you will need both Air In & Air Out... any mechanical enigineers here? architect's are not allowed to "practice" any engineering.... can someone count the ceiling tiles or give us a sq. footage of the room?

secondly , how long do you think these suckers will be kept around, before selling the whole lot on ebay?

[rickvanr]

i still dont think theyd miss just one... haha

[MrMacMan]

Originally posted by Doctor Q
I want one! (The T-shirt, that is.)
Yeah thats a hot shirt.


Originally posted by Tastannin
Don't those racks actually hold 12 G5's rather than 9? Look at the 3rd pic of Arn's mirror.

Just a minor quibble. :)

Other than that, it's awesome, I agree.

Yeah, they do hold 12.


BTW, you think they could add more fans?

Jebus, PowerMac's 9 + like 2 More fans on top?!?

:eek:

[dombi]

I do not want to ruin their fun, but they will be so pissed when they find out that they cannot run OS 8 on them.

Sriously, what good is a computer (OK, let's even call it cluster) for, if you cannot troubleshoot the Extensions folder.

Hah.

:P

[Doctor Q]

Originally posted by dombi
I do not want to ruin their fun, but they will be so pissed when they find out that they cannot run OS 8 on them.

Sriously, what good is a computer (OK, let's even call it cluster) for, if you cannot troubleshoot the Extensions folder.

Hah.

:P I hope they remember to use the MODE32 extension and they check that their applications are 32-bit clean rather than settle for the 24-bit ones! ;)

[Sublime]

Originally posted by whfsdude
geek porn :D


NICE RACK!



Doh! MacKenzie999 beat me to it...

[Rocketman]

Originally posted by AidenShaw
Interesting that the slideshow talks about 64-bit systems (the Dell bid was Itanium II) at length....

And then it says that they are using a 32-bit operating system! Why insist on a 64-bit chip, but settle for a 32-bit OS???

Something doesn't connect here....

64 bit Panther is "coming soon". Based on the slide show I would have to estimate 1-1-04.

Rocketman

[AidenShaw]

Originally posted by Rocketman
64 bit Panther is "coming soon". Based on the slide show I would have to estimate 1-1-04.

Rocketman

I find that highly unlikely. I'd wouldn't expect to see true 64-bit until OS X 10.4 "Wildcat" in about a year. Or even OS XI - 64-bits would be worth a major version number increment.

As to the slideshow - no way that they'd do an OS change as they go into production. They'll have spent months getting whatever OS they are using (10.2.7, right) working smoothly.

You don't do a major OS upgrade on a $5M computer, especially to a new 64-bit OS, at the last minute.

[TVGenius]

Originally posted by skharris
Its really a shame that people that ordered Dual 2Ghz G5's were bumped just for a project like this. - It basically sucks given the fact that people like myself are still waiting for our orders that where placed back in June!

Yep. Apple told us our order (edu inst.) would ship by the end of August... still hasn't shipped, Apple still can't answer why not. If we don't have them by October 1, we lose the $27,000 in grant money that bought them, and our order. So if we still want to order them, we have to dig up another $27,000... and get back at the end of the line. And my cinema display on my desk is getting dusty. But what's the point of cleaning it? Doesn't look like I'll be using it anytime soon. At least the Z-680s can plug into my trusty W2k PC. I pity the business professor in the next office. :p

[arn]

Someone posted a full mirror:

http://tartarus.uwa.edu.au/~trs80/Terascale/

of all the images.

arn

[MrMacMan]

Originally posted by AidenShaw
Interesting that the slideshow talks about 64-bit systems (the Dell bid was Itanium II) at length....

And then it says that they are using a 32-bit operating system! Why insist on a 64-bit chip, but settle for a 32-bit OS???

Something doesn't connect here....

Oh... my... god.

Why do you not understand that the reason Panther is *Not* a 64 Bit OS is because then 32 bit users can't *RUN IT*.


This does not mean however it is not G5 compatible, it is and they optomized Panther so it has 64 bit code in in!

Originally posted by arn
Someone posted a full mirror:

http://tartarus.uwa.edu.au/~trs80/Terascale/

of all the images.

arn

Very Very nice.

http://tartarus.uwa.edu.au/~trs80/Terascale/G5%20Modification/IMG_2112.jpg


You don't think they will notice if one is missing eh?

:p

http://tartarus.uwa.edu.au/~trs80/Terascale/G5s%20in%20Racks/IMG_2355.jpg

A sea of Powerful computers... ahh

[Doctor Q]

Wouldn't it be fun if they ran SETI or Folding on the cluster to see how many units per hour they could complete? Anyone care to guesstimate?

[Flynnstone]

Originally posted by AidenShaw
Interesting that the slideshow talks about 64-bit systems (the Dell bid was Itanium II) at length....

And then it says that they are using a 32-bit operating system! Why insist on a 64-bit chip, but settle for a 32-bit OS???

Something doesn't connect here....

Perhaps they don't need a 64 bit OS.
Perhaps they need 64 native math, this is most likely for scientific apps. So for 64 bit math, the Itanic and 970 are both good choices.
Or perhaps they are basing of Spec marks.

[macphoria]

Wouldn't it be fun if they ran SETI or Folding on the cluster to see how many units per hour they could complete? Anyone care to guesstimate?
What about FPS of Unreal Tournament running at 1024x768?

[AidenShaw]

Originally posted by Flynnstone
Perhaps they don't need a 64 bit OS.

Obviously. And to the post about "bit of 64-bit in Panther", if the O/S doesn't let the application use 64-bit virtual memory, "a few bits" don't amount to much. Can you tell me what the "few bits" are?

Originally posted by Flynnstone
Perhaps they need 64 native math, this is most likely for scientific apps.

The G4, Pentium, and even G3 do native 64-bit floating point, which is more important for most science codes. (The "LinpackD" benchmark for the Top500 list is mostly 64-bit float.)

The PPC970 has 64-bit native integers, which might be a help for some codes.


Originally posted by Flynnstone
Or perhaps they are basing of Spec marks.

But Apple's SPEC marks are way below the official listings for Pentium/Xeon - and the Itanium absolutely creams the PPC970.

[Matty P]

Go see http://www.unit69.com/terascale

The very best of all the pics

[MacRAND]

Each rack holds 3 G5s abreast, 4 high = 12 (with cooler fans above)
That's 92 racks for just the G5s, more to hold RAIDs and Backups, etc.

[kibit]

could this be the reason why the rest of us are waiting to receive our G5s??? hehehhehe :D

[backdraft]

Originally posted by PowerBook User
That's nice they're putting pictures up. What was the largest number of Macs used in a cluster before this? I seem to remember a project called Appleseed where they would hook up numerous Power Macs. Also, I think Apple should sell that shirt on it's web site. They would sell a ton!

I'm surprised Apple doesn't sell Apple shirts, hat's, coffee mugs, keychains, etc... in the Apple Store. It would be another source of revenue and free advertising for Apple.

–backdraft

P.S. Petition anyone? ;)

[MacBandit]

Originally posted by backdraft
I'm surprised Apple doesn't sell Apple shirts, hat's, coffee mugs, keychains, etc... in the Apple Store. It would be another source of revenue and free advertising for Apple.

–backdraft

P.S. Petition anyone? ;)

They do sell them in a Apple store. The company store in Cupertino. I go there every year and pick up a few trinkets and clothes.

[giba]

Sweet! I could steal a G5 from there an nobody would even notice.

[Independence]

sure looks nice. wish i could have one. :)

[MacRAND]

Hey guys, design your own and print it on a T-Shirt transfer printed on any Ink Jet. It works great. Copy what you see and like, then add color and personalize it. Forget Cupertino, why do you think you own a Mac? Photoshop, Illustrator, MS WORD, whatever. Go wild. Let's see who can design the coolest T-shirt, print it off, then IRON ON a shirt. No guts, no glory. I've done it and it's fun. So, just do it.
:D

My Other Mac is a Supercomputer

or

Cluster Geek
Supercomputer Freak
Mac G5 is not Meek

Come on, I'm not creative, just lame.:rolleyes:

[Steven1621]

wouldn't it make more sense for them to wait for the new xserves to come out?

[giba]

Originally posted by Steven1621
wouldn't it make more sense for them to wait for the new xserves to come out?

Maybe I'm wrong (and I think I am), but I think a supercomputer cluster is different from a server farm.

[MacRAND]

Yes, waiting for G5 Xserve with dual 2.0 chips would be more practical and would take up significantly less space, but Virginia Tech could NOT WAIT to build their Cluster because they had to have it completed, and up and running by an October 1, 2003 deadline to be listed in the annual edition of a SuperComputer List of the top 500. Over $5-million dollars is at stake, and their venture seeks government and commercial grants in the $10s of millions of dollars as a critically important source of income for their academic program. They charge high dollar to do research projects, which pays for their program and then some.

Fiber Optic connectivity of G5 chips is the main component, so I think a "server farm" or G5 cluster would essentially be the same depending upon the software used and the hardware configuration.

Apple still has to design a COOL Xserve box to handle the heat like the current G5 boxes. There is no published time line for that to happen with the Xserve yet. The G5 is an IBM server chip and I don't think anyone disagrees that it is destined for the Mac server program. Plus, Virginai Tech is NOT using OS X software, but is developing their own UNIX based application.

When the 1,100 unit cluster was first announced, I asked around if they should wait for an Xserve model, or maybe phase in faster and faster chips, and the answer was NO. They need them all the same and they needed them NOW. And, that's why so many people who had put an early order in on the dual are still waiting, Virginia got the first 1,100 and other students thereafter.

[appleguy]

G5s in XServes I hope so...
I work for a reseller (dual platform one) I have a customer that needs a few rack mount servers... (like 30 odd) I want macs but they need to be 64-bit... and AMD have the only decent one out there... BRING ON THE XSERVE

[kiwi_the_iwik]

It brings new meaning to the term:

"Nice rack..."

;)

(Oops - somebody already beat me to that comment!)

Still - at least you won't get slapped for saying it...


Originally posted by Chaszmyr
That's the most beautiful thing I've ever seen

Dude - you've really got to get out more!!!!

:D

[lduncan]

Originally posted by appleguy
G5s in XServes I hope so...
I work for a reseller (dual platform one) I have a customer that needs a few rack mount servers... (like 30 odd) I want macs but they need to be 64-bit... and AMD have the only decent one out there... BRING ON THE XSERVE

Who's needs that sort of power in NZ? What are they to be used for?

Layton

[appleguy]

Originally posted by lduncan
Who's needs that sort of power in NZ? What are they to be used for?

Layton

Just wish I could say... But it will be used for research.

[lduncan]

Thats enough info for me. Hope you go with macs. There is always the added bonus when buying desktops for this sort of project, you can sell them off easier when you need to upgrade. I think it would be easier to sell 30 G5 desktops than 30 Xserves.

[Genie]

I've always wondered why they don't just combine the xserve and the tower into one unit, maybe make it 2 rack spaces.
http://www.geniesongs.com/genie/endorsements/apple/Genie-G5Computer50pix.jpg (http://geniesongs.com/personal.html)

[peterj1967]

Since they have to have the machine benchmarked by today, I am amazed there is so little news about it. Does anybody have any info on how it is going? At least their web site with the images works again.

http://computing.vt.edu/research_computing/terascale/

[MacRAND]

In terms of the number of processors, Big Mac's closest analog is a cluster of 2,300 2.4 GHz Xeon processors at Lawrence Livermore National Laboratory. Clocked at 7.6 teraflops, the cluster is currently ranked third. "It will be interesting to see where the G5 comes in comparison to this machine," he said. So far, Big Mac appears to be 2nd behind a $350-million Japanese supercomputer.

Today, Joyce Randall at Virginia Tech responded to my request for performance data on their "Big Mac" Supercomputer, and about the T-Shirt "My other Mac is..."

Good morning Rand,
Thank you for your interest in and support of the Terascale Supercomputer Cluster.

We have received a number of inquiries regarding the Cluster (quite a few for the T-shirts specifically) and right now we are working on a website where all questions can be submitted, answered and then posted for everyone interested to query. Our main focus right now is to get everything up and running and as you can imagine that has been and continues to be a major effort. Things are looking great!
I am working with the University on the possibility of making the T-shirts available for purchase. As soon as that is accomplished, we will post the information to our current website:

http://computing.vt.edu/research_computing/terascale/

In the meanwhile, here are a couple other websites I think you might find interesting:

<http://www.bbcworld.com/content/template_clickonline.asp?pageid=666&co_pageid=3>

<http://www.wired.com/news/technology/0,1282,60821,00.html>

Please feel free to send questions and/or comments to my attention and I will certainly do my best to get them answered.

Sincerely,

Joyce Randall
Assistant to the Associate Vice President
High Performance Computing
1700 Pratt Drive, Suite 209-C
Virginia Tech

[MacRAND]

2nd most powerful computer on Earth is a Mac

The G5 cluster of the University of Virginia, in the USA, is up and running, under Mac OS X, a version of Panther. Its performances were measured for the first time under the Linpack Benchmark, and scored an amazing 17.6 TFlops !

This result makes this supercomputer the second most powerful one, behind the famed Japanese Earth Simulator, and it is the most powerful of all in the United States. Not bad, when you know it cost a mere 5.2 million dollars, half the estimated price from the competitors during the bid, and most of all, far less than the $100 million that the other current supercomputers did cost. ;)

"Big Mac" (as nicknamed by Terascale staff) will officially join the Top5 Supercomputers during the SC2003 conference in Phoenix, AZ, USA, from the 15th till the 21st November.

P.S. : To help the comparison,
Earth Simulator cost $350 million,
Virginia Tech Mac Cluster Supercomputer $ 5.2 million (1,100 G5 dual 2GHz cluster)
ASCI Q, the (2nd best ...until now), cost $200 million, and
ASCI White (4th position) "only" $100 million.

So, pray tell, are Macs still that expensive?


Mac Supercomputer costs $5.2 million
October 9th 2003

Supercomputers - everyone would love one but with a price tag of around $100 million each they're not easy to come by. But in the United States staff and students at Virginia Tech have built one of the world's most powerful Supercomputers for a cool 5 million dollars. It's all to do with plugging together smaller Mac G5 machines.

Ian Hardy went to Virginia to see the supermachine for himself.

This is the project that has caused heads to turn in the world of Supercomputing. It is, in fact, 1,100 brand new Apple G5 towers placed side by side making it the world's most powerful homebuilt system, capable of 17.6 trillion floating point operations per second, with a combined storage capacity of 176 terabytes.

Pat Arvin, Project Coordinator, Virginia Tech: "Each individual G5 is a dual processor, 2GHZ machine with 4GB of memory. So it's extremely fast." This network is linked using 2900 cables and runs at about 100 times faster than an average corporate network that uses today's top of the line Ethernet connections. But the hard part of making a Supercomputer is stability.

The project's chief architect, Srinidhi Varadarajan, had to write a special program called Déjà Vu to ensure that if any individual computer crashes in the middle of a calculation lasting weeks, if not months, another computer will take over seamlessly. Varadarajan not only helped build the supercomputer, he developed the software package employed in the cluster, “Déjà vu”, helping bring stability to the computers. Varadarajan said most quality computers fail about once per year and clustering 1100 computers together creates a multiplied failure rate that would be unacceptable. The software transparently transfers a failed application to a new place without the knowledge of the computer, preventing the application from being altered in any way.

With the addition of the supercomputer, the Tech community is open to receiving funds from a proposed National Science Foundation program called The Cyberinfrastructure Funds. The program is scheduled to have a $1 billion annual budget and will offer $100 million to centers like Tech, which house large supercomputer centers supported by the National Science Foundation.

Srinidhi Varadarajan: "This is pretty much like open heart surgery because you're working on a computer and moving an application while it still continues to run. You cannot stop the program, actually, and that's the speciality of this system."
The Supercomputer, unofficially nicknamed Big Mac, was built in just three months. Right from the start there were major hurdles that could only be overcome with significant construction in and around the building. Running 1100 computers in a 3000 square foot area sends the air temperature well over 100 degrees Fahrenheit. In fact the heat is so intense that ordinary air conditioning units would have resulted in 60 mph winds. So specialized heat exchange cooling units were built that pipe chilled water into the facility.

Kevin Shinpaugh, Director of Research/Cluster Computing: "There are two chillers for this project. They're rated 125 tonnes each in cooling capacity, and they pump 750 gallons per minute each. The water is at about 45 degrees Fahrenheit."
The power supply was another huge challenge. The Supercomputer uses the same amount of electricity as 3000 average sized homes. Meanwhile, in early September, the G5s started arriving by the lorry load.

Jason Lockhart, Director of High Performance computing: "The machines were taken off the trucks on their pallets. The pallets were broken down and we processed machines. We did 238 machines in little under two hours, so we were humming along as an assembly line."
A speedy installation was essential because Virginia Tech had to comply with an October deadline set by the National Science Foundation. Missing that deadline would have meant automatic disqualification from the NSF's global Supercomputer rankings, thereby denying the college any chance of competing for top scientific research projects worth hundreds of millions of dollars per year.

Srinidhi Varadaraja: "A system of this size generally sees its most application in what is known as big science research; massive simulations, models, computational engineering systems. Examples of these include things like nanoscale electronics, if you're trying to invent computer chips 30 years from now you're looking at atomic levels with a single atom acting as a switch."
Arguably Virginia Tech has revolutionized the world of Supercomputing with a simplistic setup that can be duplicated around the globe by other institutions.

Hassan Aref, Dean of Engineering: "We're going to document how we did this from start to finish and if you want to build one we'll send them a kit and tell them how to do it."
Supercomputers, like PCs, become old in technology terms within about 3 to 5 years. But whereas you or I would go to a shop and buy one, Virginia tech has to start again from scratch. That's why they are building a new multi million dollar facility opening in 2006 for the next generation of Supercomputer.

1st most powerful computer on Earth
Costs $350-million, achieves 40 Tflops

Distributed Memory Parallel Computing System which 640 processor nodes
interconnected by Single-Stage Crossbar Network

Processor Node: 8 vector processors with shared memory
Peak Performance: 40 Tflops
Total Main Memory: 10 TB
Inter-node data transfer rate: 12.3GB/s x2
Target for Sustained Performance: 5 Tflops using AGCM.
*The goal above was its thousand times' performance of the AGCM in those times (1997).