Dual GTX 980 internally powered in a Nehalem Mac Pro - a theoretical design

[Machines]

This is a new thread that was started , because I realized continuing discussing it in the dual GTX 970 thread would have made it perhaps OT . Anyways , here goes:

I propose that it is possible to install two GTX 980 video cards for Compute purposes inside a Nehalem Mac Pro 5,1 or an upgraded Mac Pro 4,1 > 5,1 . And , furthermore, to power the cards entirely internally and have enough power remaining to operate a high performance PCIe secondary storage device . We should also use just a single processor six core machine with a reasonably useful amount of memory .

All these components are necessary to make a GPGPU array useful in rendering operations .

The Nehalem Mac Pro has a large PSU rated for 980 W . It is a very efficient power supply and provides system builders with a lot of opportunities to install high performance after-market components .

We just have to be careful where we acquire the power sources for all these components since Apple made the Nehalem Mac Pro beautiful inside . Components are not directly powered by the PSU , but receive power indirectly through motherboard traces . It is vital that the power drawn through these motherboard traces do not exceed their rating , or the motherboard will become damaged and there is the possibility of a case catching on fire .

And let's be conservative with overall system power draw , so no beefy 12 Cores and 128GB memory configurations . A nice humble Hexacore will do . Remember, rendering is not CPU centric anymore , but having lots of energy sipping threads available probably won't hurt .

Here's the proposed system configuration :

Mac Pro 5,1 (or 4,1 > 5,1)
One x L5639 Xeon . Six Cores @ 2.13 GHz . 12 threads . 60W TDP .
32 GB 1333 MHz DDR3 ECC 1.35v low voltage memory modules (4 X 8GB) .
2 X nVidia GeForce GTX 980 Reference video cards (PCIe slot 1 and 2) .
1 TB PCIe Sonnet Tempo Pro Plus card with 2 X 512 GB Samsung 850 Pro SSDs in RAID 0 (PCIe slot 4) with 6G eSATA .
No optical drives .
Blue tooth , factory .
Wifi , factory .

I will dispense with instructions on how to install the processor , memory and Sonnet Card . They are available elsewhere and these particular upgrades are not difficult to install . We just want to discuss a possible method of installing two highest end Maxwell GPUs available .

Nvidia GeForce GTX 980 reference cards consume 165 W of power, officially .

But there are some concerns they actually consume 183 W at load in our Macs . So, let's over-engineer our system so there are no nasty surprises .

Here's how we can install two GTX 980s right now .

Additional components needed for power delivery are two ordinary Mac PCIe 6 pin power booster cables, two powered 2.5” to 3.5” SATA HDD form factor adapters and two SATA power to 6 pin PCIe booster adapter cables . Not too many parts , but they are uncommon and need to be special ordered .

Anyways, here's our installation so far :


GTX 980 card number 1 powered from PCIe slot 1 (75 W) + PCIe booster Aux A (75 W) + SATA Backplane #3 (33 W) . Total 183 W consumed . PCIe slot 1 .

GTX 980 card number 2 powered from PCIe slot 2 (75 W) + PCIe booster Aux B (75 W) + SATA Backplane #4 (33 W) . Total 183 W consumed . PCIe slot 2 .


That was quick ! And it will work , too . But dream time is over, folks , because ...

We have our array up and running internally powered , but we also just consumed all available PCIe power resources (300 W) .

There's no juice left for the two remaining PCIe card slots . Hmmmm . And we really need one of them .

How are we going to fire up that gorgeous Sonnet PCIe high performance drive card we will definitely need to write all the data our GPGPU array is so busy processing ?

We'll need to reroute just one of those 6 pin power booster cables and stick it somewhere else …

But where internally ? We're not gonna cheat and route from the exterior . Clients hate external aux power supplies . Every time I mention it , I get cold stares . So, time to think like a Borg again ! Enter the ODD power point on the mobo .

That point fires up two energy hogging optical drives through a harness in the Optical Bay .

We reroute the optical harness from the Bay and into the middle of the enclosure , where the video cards are . There are two SATA power connectors on that harness . We get a PCIe 6 pin booster connector to dual SATA power connector splitter cable and connect it to one of GTX 980 card number 2's booster connectors .

This will require some minor cutting and super-gluing . But , at least, we have all our devices connected and powered .

Concern is , how much juice does Apple allow to the two Optical drives ? A DVD-RW drive likely needs 27 W max . So, the two drives will be provided with 54 W bare minimum through that harness . And there is going to be some extra provided as a reserve . What this amount is , is anyone's guess .


So, we now have the following GPU installation :

GTX 980 card number 1 powered from PCIe slot 1 (75 W) + PCIe booster Aux A (75 W) + SATA Backplane #3 (33 W) . Total 183 W available .

GTX 980 card number 2 powered from PCIe slot 2 (75 W) + ODD SATA (54 W) + SATA Backplane #4 (33 W) . Total 162 W available .


We now have more than enough juice (75 W freed up) to power the PCIe Sonnet high speed drive .

But, will a GTX 980 work properly with just 162 W in a Mac ?

And , we also need to be concerned about melting mobo traces with all the draw the components will be demanding . We're grabbing resources from all over the place . So, maybe we should take baby steps, one step at a time, with this build, folks .

And, if it does work as described as above, there may be some automatic down throttling involved , due to insufficient power provided. Who knows until someone tries the design , but ...

If the above set up does not work, then we will need to take a page from Prince134's beautiful dual 7970 Mac Pro experiment and down voltage the cards to reduce the power draw .

I bet this system build would work , especially with down volt-aging the cards as a last resort . The 980s perform no higher than 20 percent compared to the 970s in the passmark direct compute score . Compute is what this is all about .

And I think my Dual 970s draw 153 W peak each in one of my other Macs right now . So, add 20 percent to the 153 W and we get 183 W peak . The 970s at load probably averages around 135 W each .

This is really a close call...

And all of this is still theoretical , as no one has done this yet (install two GTX 980s in a Mac Pro entirely internally powered .)

Given enough time , I'll do this build and write up precise instructions . But someone else will probably beat me and these notes are intended to help and to open a general discussion .

Overall , the system should not consume a large amount of power relative to the high performance the build should deliver , especially once drivers are optimized for the Maxwells .

I estimate total system power consumption at around 500 - 600 W at load .

And just as a disclaimer , by the way, try all this at your own risk . Sparks may fly . You're on cutting edge of internal Mac Compute, after all . And the cutting edge is rarely safe territory .

 

[wonderspark]

My input is to consider pulling power directly from the power supply before it goes into the motherboard. Doesn't the power supply have a fat wad of cable that runs from a harness to the motherboard? Use some of that directly.

It's not necessary for all power, just tap what you need and run them hidden here, tucked there and finally all neatly wrapped in cable wrap to the final destination.

 

[lowendlinux]

For this I say $45 dollar power supply and done why all the drama?

 

[Machines]

My input is to consider pulling power directly from the power supply before it goes into the motherboard. Doesn't the power supply have a fat wad of cable that runs from a harness to the motherboard? Use some of that directly.

It's not necessary for all power, just tap what you need and run them hidden here, tucked there and finally all neatly wrapped in cable wrap to the final destination.

Excellent ! It would be great if you could do the pin out of the PSU connector please and share with the community .

 

[h9826790]

May I know where you get the reference about that 300W limit is including the boosters?

My understanding is that the 300W limit is only for that 4 PCIe slot, and boosters can provide extra power when required.

 

[Machines]

For this I say $45 dollar power supply and done why all the drama?

I think there would be a lot of drama if that 2 out of 5 star Newegg reviewed aux PSU failed while a pair of 980s were rendering , or even at start up . To stick a cheapie $45 PSU in an expensive Mac is not the wisest action , I think . The goal is to build a system that can run at load , safely and reliably , for the long term . Mac Pro power supplies are very high quality and a responsible builder uses as few points of failure as possible in a build .

 

[wonderspark]

May I know where you get the reference about that 300W limit is including the boosters?

My understanding is that the 300W limit is only for that 4 PCIe slot, and boosters can provide extra power when required.

This is also my understanding - that 300W is delivered to the four PCIe slots - but not necessarily in 4x75W form. So a GPU in a slot may draw 75W from the slot itself, plus 75W from each additional boost cable. That's how they get to the so-called 225W maximum on a single card using the two additional 6-pin cables.

I'd pull my power supply out and measure all the pins, but I'm only curiously interested in doing so, rather than putting my business on the line. If I'm building machines for others as a business, I won't be trusting someone else's pinout, you know? A respectable business needs to verify this first hand.

Having said that, I did a little Google search, and found a guy posting his pinout findings here:
Beware - it may be a 980W unit from a 2006, 2007 or 2008 Mac Pro, as opposed to a 2009-2012. I know there are various part numbers, and I'm unaware of what exact differences there are.

 

[Machines]

May I know where you get the reference about that 300W limit is including the boosters?

My understanding is that the 300W limit is only for that 4 PCIe slot, and boosters can provide extra power when required.

I have not found any official reference .

But the one important reference , Lucas Godfrey , the fellow that just beat me to building the first Dual GTX 970 in a Mac Pro , says there is that 300 W limit . In his words ...

It's not just about the 6pins, it's that a Mac pro has 300w limit for PCI, that can be spread either through 75w 4x pci slot power, or 2xslot + 2 booster cables. The 980s draw 180 watts or so (165 maybe?) so it exceeds the max available ouput for PCI on the Mac pro.

The 970s work because they need less then 150w each so they slot in under the output threshold.

His statements caused me to write up the design theory .

 

[wonderspark]

Okay, logically, how does it make sense to say it's either 4x75W from only the four slots, OR 2x75W from two of four slots + 2x75W from the two 6-pin cables on the other end of the motherboard?

If there is a total of 300W being delivered to the four slots AND the two 6-pin connectors, as a combination of all six, I'd be surprised. The only real solution for those held responsible for selling such modified systems is to do the measurements and the math, and figure out exactly where power is distributed.

 

[Machines]

This is also my understanding - that 300W is delivered to the four PCIe slots - but not necessarily in 4x75W form. So a GPU in a slot may draw 75W from the slot itself, plus 75W from each additional boost cable. That's how they get to the so-called 225W maximum on a single card using the two additional 6-pin cables.

I'd pull my power supply out and measure all the pins, but I'm only curiously interested in doing so, rather than putting my business on the line. If I'm building machines for others as a business, I won't be trusting someone else's pinout, you know? A respectable business needs to verify this first hand.

Having said that, I did a little Google search, and found a guy posting his pinout findings here:
Beware - it may be a 980W unit from a 2006, 2007 or 2008 Mac Pro, as opposed to a 2009-2012. I know there are various part numbers, and I'm unaware of what exact differences there are.

Image

Those look like 2006 - 2008 Mac Pro PSU pin outs . They are found on four independent or semi independent cable leads from the PSU . We are in need of the Nehalem (2009-2012) Mac Pro PSU pin out .

I thank you for your efforts so far !!!

 

[wonderspark]

My theory is that you have 450W of power total to draw from. 300W from the PCIe slots, and 150W from the two 6-pin cables.

That leaves 530W for the rest of the box... CPU(s), drives, fans, lights, etc.

If I ever get bored enough to completely tear down my 4,1 Mac Pro, I'll let you know what and where everything goes.

 

[Machines]

Okay, logically, how does it make sense to say it's either 4x75W from only the four slots, OR 2x75W from two of four slots + 2x75W from the two 6-pin cables on the other end of the motherboard?

If there is a total of 300W being delivered to the four slots AND the two 6-pin connectors, as a combination of all six, I'd be surprised. The only real solution for those held responsible for selling such modified systems is to do the measurements and the math, and figure out exactly where power is distributed.

I've sold three or four of the Dual AMD 7950 GPGPU systems and they're still in the wild , rendering without issue for the last 3 or so months .

I tested the power draw on all the points accessed by all the components during the build process . Load wattage for the 7950s was around 130 W each . Very safe . I test with Hardware Monitor and a Kill-A-Watt device and I have a multimeter to verify things if the need arises . Do you know , I sleep in the room next to each and every Mac I build when they go through their final burn ins (8-12 hours ) ? Each build takes about three days . I literally stake my life on these builds and I have a great local reputation . They're functionally perfect when the client takes possession .

----------

My theory is that you have 450W of power total to draw from. 300W from the PCIe slots, and 150W from the two 6-pin cables.

That leaves 530W for the rest of the box... CPU(s), drives, fans, lights, etc.

If I ever get bored enough to completely tear down my 4,1 Mac Pro, I'll let you know what and where everything goes.

Do you need high res photos of the board ? That, I can do . There are parts galore around here

----------

My theory is that you have 450W of power total to draw from. 300W from the PCIe slots, and 150W from the two 6-pin cables.

That leaves 530W for the rest of the box... CPU(s), drives, fans, lights, etc.

If I ever get bored enough to completely tear down my 4,1 Mac Pro, I'll let you know what and where everything goes.

I used to think that, too . But I cannot prove or disprove it . I will tell you there is an immense reserve amount of available power . I've built some very high performance rigs and I've never seen a cMP consume more than 650 W , system wide . Even taking into account some waste energy , there must be around 300 W of additional power that can be accessed away from the motherboard even in high end builds .

 

[wonderspark]

Sure, I'd love to look at a high-res pic of the back of the backplane board. Might as well shoot the front of the backplane board, while you're doing it.

 

[Machines]

Sure, I'd love to look at a high-res pic of the back of the backplane board. Might as well shoot the front of the backplane board, while you're doing it.

I'll try to do it over the weekend .

Tomorrow , I have a bunch of studio guys from Chicago visiting . They want the Dual 970 system , but I don't think WB is paying them enough to afford it .

I spent 1400 bucks on the cards and the RAID system alone . She's awesome in Blender .

 

[Anonymous Freak]

Obviously haven't done the teardown, but from experience with other systems, I agree with the "300W is for slots only, each aux connector is separate" sentiment. It would be exceedingly odd to design those aux power connectors to draw from the same feed that feeds the motherboard/PCIe slots. Every other implementation I know of has the PCIe aux power cables on completely different rails than the motherboard PCIe feed.

Of course, you could also go for an optical-bay auxiliary power supply: Adding extra Internal PSU in optical bay for graphics cards [Build log] (If you can find one of these longer than 2 feet, you can even have it so that you don't have to use up two wall outlets.)

 

[wonderspark]

I used to think that, too . But I cannot prove or disprove it . I will tell you there is an immense reserve amount of available power . I've built some very high performance rigs and I've never seen a cMP consume more than 650 W , system wide . Even taking into account some waste energy , there must be around 300 W of additional power that can be accessed away from the motherboard even in high end builds .

Is that 650W max you've seen on a single CPU or a dual CPU? I'm sure most of the power is going to the processor board, since they come both single and dual CPU, and therefore a single CPU model is going to have a lot of unused power routed to that board. Since you're proposing an ideal build of a single CPU model, it would make sense to pull some of the power you're looking for from the power supply that is normally sent to the Backplane Interconnect, as it won't be used otherwise.

 

[wonderspark]

Looking at the block diagrams and such from the technician guide, we can see the power supply is putting out 12v @ 80A, and 5V @ 5A. That makes 960W from the 12V and 25W from the 5V, totalling 985W. Given the label on the PS showing five watts less at 980W, it's likely the block diagram's 80A and 5A designations are not precise, but hey, close enough.

The technician guide also mentions specifically that 300W is the max to draw from the four PCIe slots, but you can see they are not pointing out the additional two 75W 6-pin connections. Not proof of anything, but worth noting.

It should be pretty simple to determine where power is going by following traces and measuring the power supply.

 

[Machines]

Is that 650W max you've seen on a single CPU or a dual CPU? I'm sure most of the power is going to the processor board, since they come both single and dual CPU, and therefore a single CPU model is going to have a lot of unused power routed to that board. Since you're proposing an ideal build of a single CPU model, it would make sense to pull some of the power you're looking for from the power supply that is normally sent to the Backplane Interconnect, as it won't be used otherwise.

Even Dual Hexacore 3.46 GHz systems with one high end 7950 compute and one 5770 UI card and 32GB of memory don't exceed 650 W at load . Or I as sometimes call them DuoDecaHexaCores

Sometimes I wonder if there's a Zero Point Energy source in these rigs ... if only I could access it all .

 

[Machines]

Is that 650W max you've seen on a single CPU or a dual CPU? I'm sure most of the power is going to the processor board, since they come both single and dual CPU, and therefore a single CPU model is going to have a lot of unused power routed to that board. Since you're proposing an ideal build of a single CPU model, it would make sense to pull some of the power you're looking for from the power supply that is normally sent to the Backplane Interconnect, as it won't be used otherwise.

There's a lot of juice that goes there , for sure . Look at the 8 thick bus bars on the CPU Tray's Interconnect . But I'm not certain we can grab the freed up juice and re-purpose it .

 

[Machines]

Of course, you could also go for an optical-bay auxiliary power supply: Adding extra Internal PSU in optical bay for graphics cards [Build log] (If you can find one of these longer than 2 feet, you can even have it so that you don't have to use up two wall outlets.)

Not again The AUX PSU thing was already discussed . I hate them . They will never be more reliable than our MP PSUs . Do you have any idea how much power is underutilized in the factory PSU ? We just have to find a way to grab it ! We need a build that is not likely to fail . If that AUX PSU were to fail , say bye bye to the GPGPU array when it's in high gear .

 

[Machines]

It would be exceedingly odd to design those aux power connectors to draw from the same feed that feeds the motherboard/PCIe slots. Every other implementation I know of has the PCIe aux power cables on completely different rails than the motherboard PCIe feed.

This is Apple and we have to be prepared for their special way of thinking differently, usually undocumented . Inquiring minds are already looking into the rail issue .

 

[h9826790]

This is Apple and we have to be prepared for their special way of thinking differently, usually undocumented . Inquiring minds are already looking into the rail issue .

I am actually thinking the other way around. This is Apple, they don't want to take any legal responsibility because of poor description of an system limitation, and someone misunderstand the limit, end up make the machine catch fire...

They usually very conservative when writing about system limitation, especially something that is safety related.

 

[A Hobo]

Just to chime in, it appears I've become a point of reference; so i thought id just clear up some things

I think i was mistaken regarding total power available to the slots/ boosters. it appears 450w is available (4xslot power + 2x booster All @ 75w)

I had looked in the tech guide and had misread the section regarding total power draw. i now see that they say "slot" power cannot exceed 300w. that seems to imply the boosters are not included.

to test this, i loaded up my machine which should be theoretically at the limit i thought (295 watts or so Guesstimate.) and then i added a sonnet card I had laying around. Not sure if it consumes anywhere near 75 watts, so this isn't scientific by any means. but lo and behold it booted and was detected..

Ill have the weekend to carry out some more tests so ill have an update then.

 

[Machines]

Just to chime in, it appears I've become a point of reference; so i thought id just clear up some things

I think i was mistaken regarding total power available to the slots/ boosters. it appears 450w is available (4xslot power + 2x booster All @ 75w)

That was my thinking for years , but I find I cannot prove it now that its important . Apple's service materials are delightfully incomplete :\ . Wonderspark is going to look at the traces on the back of the backplane board to discover where the two rail bundles from the PSU wind up . That will explain a lot . Here's a picture of a 2009 Mac Pro PSU label to confirm there are only two rails we need to be concerned about .

 

[Machines]

to test this, i loaded up my machine which should be theoretically at the limit i thought (295 watts or so Guesstimate.) and then i added a sonnet card I had laying around. Not sure if it consumes anywhere near 75 watts, so this isn't scientific by any means. but lo and behold it booted and was detected..

I went to Sonnet's site to discover the power requirements of the card and came up empty . They don't list those specs . I might have the opportunity today of comparing the system power consumption at idle with and without the card installed . We do know SSDs consume about 2 watts each under heavy usage .