The Quad Chronicles - Restoring a HOT G5 Quad

[mac57mac57]

Hello All, I recently acquired a Power Mac G5 Quad, the last and fastest of the PPC Macs. It was missing its hard drives and was sported a failing video card. I replaced the video card with an nVidia Quadro FX 4500 and added two hard drives that I had in stock. I turned the Quad on, only to discover that the fans quickly ramped up to 3300 RPM in order to hold CPU temperatures in check (...which they did! CPU temps were in the mid 40s C, which is perfect). However, the noise from the fans running at such high RPMs made the machine effectively unusable. I burned Apple Service Diagnostics 2.6.3 to a DVD and ran Thermal Calibration on the Quad. Both CPUs failed. Clearly it is time to service the Liquid Cooling System (LCS), the most feared and complex maintenance task that I am aware of in all of the Mac world.

That is where this thread starts. This thread will chronicle the journey from a hopelessly noisy, barely cooled G5 Quad to a hopefully quiet and peaceful G5 Quad that can be productively used. This is a long, complex and challenging maintenance task - I expect that it will takes weeks, if not months, to complete.

I invite you to come along with me for the ride. Please feel free to contribute your thoughts, possible steps and any other wisdom you may wish to impart, as you see fit. As I am going along, I am going to document all of my work, and when done, produce a concise guide to servicing Quad LCS units which I will post here and at my retro-computing blog, www.retro-computing.com.

So, where do we start? Well, the machine went "on the bench" this afternoon, as you will see from the attached photo. My first step was the Thermal Calibration. Now that this has definitively failed, the next steps will be to do a thorough interior cleaning and then extract the CPUs. I will repaste them, put them back and see if that helps. If it does not, and "not" is the expected outcome, I will proceed to the much more difficult job of draining and recharging the LCS, a task about which volumes have been written by other authors in various places.

I will post more in this thread as the work progresses. Stay tuned!

BTW, Thermal Calibration did in fact fail, but it was not without value and impact. Thermal Calibration rewrites the data in the calibration ROMs to match current circumstances, and this had the effect of lowering the idling fan revs on my new Quad from 3300 RPM to about 2200 RPM. This is still rather more noisy than I want to deal with, and so the LCS work is definitely "on", but it is an improvement. If you have a Quad whose fans are running just a small amount too fast, Thermal Calibration may just bring fan revs back to a more peaceful level. Remember however, as the Apple diagnostics guide says, you cannot calibrate your way out of a failing LCS; you may be able to buy some time, but eventually, the unit has to be serviced.

 

[mac57mac57]

First update:

Yesterday I opened the case, removed the only PCI card (the nVidia Quadro FX 4500 video card), cleaned out all the obvious dust (just the tip of the iceberg, really), and removed and cleaned all the fans except the nearly "hidden" one in the center that blows air down the back of the motherboard and critically, across the Northbridge chip.

Then I was able to pry off the freakishly strong plastic rivet that holds the "G5" labelled CPU cover/shield, remove that cover, then remove the bezel "framing" the grid/radiator at the front of the CPU assembly and then was able to thoroughly clean all the dust lurking there.

There is a thermal insulation sheet that lies directly below the CPU assembly, more or less in-between that assembly and the power supply, and it is in bad shape... pieces of it have broken off, it is flaking over its entire surface and it is generally probably not all that effective anymore. I have order some new ultra-thin insulating sheets, and will install one of those in place of the existing, deteriorating, sheet.

I am now ready to being the process of extracting the CPU assembly. I will then isolate and repaste the CPUs, put the whole thing back together again and test. If that solves the heat problem, I am done. If it does not, and honestly I don't expect it to, then it is on to the more complex job of draining and recharging the LCS.

Below is a photo of the machine at the end of the first session:

 

[mac57mac57]

Good progress today. Today's update:

I realized that I could/should remove the metal plate that separates the upper peripherals area of the Quad from the lower CPU area. This was an easy job: remove three screws mounted to the motherboard and lift the plate straight up and out. No muss, no fuss.

Next I methodically removed or loosened (as directed by Cameron Kaiser's excellent guide to Quad CPU swapping) the 18 screws holding the CPU assembly in place and then lifted the entire assembly out of the machine. The below photo shows the Quad with the CPU assembly lifted out and sitting beside it.

The machine is very dirty around the CPU area, mostly because of the deteriorating thermal sheet between that area and the power supply. This is plainly visible in the photo below:

I will do the best I can with a vacuum and a feather duster (yes, a feather duster - they are the best darn tool for cleaning in the close quarters of a computer case that I have found!) to clean this up. I *may* simply remove the sheet entirely if parts of it disintegrate when the vacuum head is near.

As I undid each screw, I taped it to an 8.5x11 piece of paper and labelled it with the screw number, ensuring that there will be no confusion as to which screw goes where when I reassemble. Here is a photo of that sheet at the end of the session:

... and that is where I had to stop for this session - other commitments require presence elsewhere. More updates to come!

 

[mac57mac57]

More progress yesterday:

I thoroughly, but gently, vacuumed out the entire interior of the Quad and ultimately, discarded the thermal sheet between the CPU area and power supply area entirely; there was nothing left of it from a structural perspective; it would flake away if you even looked at it! ... well perhaps that is a BIT of an exaggeration, but not much!

Then, flipping the LCS assembly on its side, one-by-one I unscrewed all the screws holding each CPU card in place and removed the card from the LCS assembly - and there they were, the vaunted PowerPC 970MP chips, grey from old thermal paste.

I got out some Isopropyl alcohol ("rubbing alcohol") and gently cleaned the old thermal paste off both the CPUs and the brass plate they couple to on the LCS underside.

After the CPU chips were cleaned up, I was disappointed to see that there are no visible markings on them at all – nothing at all that hints at the power lying within. Instead, the surface of the CPU chip is so perfectly polished, and so perfectly reflective that it is like a tiny mirror. This probably aids thermal transfer to the LCS, but I was hoping for something a bit more ostentatious to mark the presence of such a powerhouse chip.

Looking at this photo, you can clearly see that there has been no coolant leakage around the CPUs – they are clean and clear, except for the thermal paste, now perhaps nearly 20 years old and in definite in need of replacement.

You might also note the pleasing visual symmetry of most of the components on the CPU cards. Everything is laid out in clean lines and nicely organized. The old Steve Jobs’ obsession with machines looking as good on the inside as they do on the outside clearly lived on in the Power Mac G5 Quad (and, I am going to guess, the entire G5 product line).

So, at the end of yesterday's session, the CPUs were removed, cleaned and ready for repasting and reassembly. The picture below shows the scene at the end of the day:

Next up, repasting and reassembly, and then, fingers crossed, Thermal Calibration. Lets hope that it passes this time!

 

[mac57mac57]

Quick update from today's efforts: some limited success: read on...

I repasted the CPUs, reassembled the LCS/CPU unit, and then the entire machine.

Then I hooked it back up to the monitor, keyboard, mouse, etc. and ran Apple Service Diagnostics 2.6.3, and in particular, Thermal Calibration. Regrettably, both CPUs still fail thermal calibration. Sigh... I knew it would be too easy to just be able to stop there!

HOWEVER, Thermal Calibration does write new thermal data into the calibration ROMs, so I rebooted into the Sorbet that is already loaded on the machine. It started out about the same as it used to be, with the CPU exhaust fans running around 2277 RPM - still quite noisy.

Then I noticed that the RPMs started to drop. Over ten minutes or so they dropped down to 1772, with the CPU showing 45/46 C. At 1772 RPMs, this is almost a tolerable noise level. Unfortunately, as I type this, the RPMs are slowly rising again, now in the mid 1800s.

SO... general cleaning and repasting of the CPUs knocked about 500 RPM off the fan speed, which is a nice result... it is not as nice as I would have liked, but it is nice nonetheless. Honestly, I am really pleased that the machine started up and ran at all, after lying around the bench in parts for several days!

I have to wrap this up for today, but more to follow. I guess I have no choice - I now have to repeat the whole teardown process, but this time, instead of repasting the CPUs, I have to flush and recharge the LCS, not a trivial undertaking. I had hoped to avoid this, but oh well.... here we go!

 

[mac57mac57]

A quick update today. I repasted, reassembled and retested the G5 Quad. Unfortunately, repasting was not enough. It did knock about 500 RPM off of the idle fan RPMs, but even so the machine was still running in the 1700-1800 RPM area at best, ramping up sharply under any significant load. So, still too noisy, and at any rate, after repasting, both CPUs still failed thermal calibration.

So, I am in the process of draining, cleaning and refilling the LCS itself. At this point, the machine is apart, the LCS has been drained, it has had one round of cleaning with a 50/50 mix of vinegar and distilled water and is now ready for one more, followed by several "rinses" with just distilled water, and then refill with the coolant I have ordered.

That coolant does not arrive until tomorrow, so today will just be more cleaning and rinsing of the LCS.

I will comment that working on the LCS is "tough stuff" for LCS newbies. I have been in the computer business for a long time, but this is the first time I have ever had to deal with liquids as part of the process. It requires a whole new set of tools and procedures! With the help of online guides and the excellent and very helpful people here at MacRumors, I am "making it up as I go"! So far, so good.

More updates once the new coolant is installed. Hopefully, both CPUs will then pass thermal calibration and peace and quiet will be re-established (from the fan perspective).

 

[mac57mac57]

Interesting development: I have two G5 Quads, with the original one's cooling slowly failing until the fans were running full blast all the time... absolutely unusable due to noise (and thermal stress on the CPUs).

Today, I booted that machine and ran Thermal Calibration, really just hoping that it might drop the fan revs enough to make the machine tolerable again. It did much more than that! As expected, the first CPU failed thermal calibration, but most unexpectedly, the second one passed calibration! After calibration, the machine now idles at about 1900 RPM, and doesn't ramp up a whole lot under load.

From this result, can I conclude that the LCS cooling loop is OK and doing its job (it successfully kept one CPU cool), leaving me to suspect that the first CPU just needs to be repasted and all will be well?

Thoughts, in favor and counter?

 

[mac57mac57]

Today's update: "short day" today working on the Quad due to other commitments. However, the LCS is now refilled and awaiting reassembly into the Quad. I got about 140 ml of coolant into it, which is about right - I keep reading that the coolant capacity is anywhere from 120 ml to 180 ml depending on the unit in question and the hosing arrangement.

Like so much of what I am doing, my solution for inserting the coolant was MacGyver'd - this is my first time working on an LCS. In this case, I inserted the coolant using a small plastic squeeze bottle that held 60 ml at a time. I was working on my third bottle when it I could not get any more into the machine. The squeeze bottle allowed me to put the coolant in under some amount of pressure, which I suspect allowed me to get more of it in than I would have been able to without the squeeze bottle. Where did I get the squeeze bottle? PetSmart! Its intended function is a kitten feeder! ...hence the "MacGyver" comment.

There MAY be some air in the hoses from this insertion method, so I will have to try to bleed that (I say "try" because it is possible that there is no appreciable amount in there) before I reassemble.

Thats it for today!

 

[mac57mac57]

Today's update: failure. After the drain, flush and recharge of the LCS, followed by reassembly into the unit, the machine will not boot. It runs upon power up, gives a nice startup chime and then the diagnostic LEDs start to come on, settling in this configuration:

1 - OFF
2 - ON/RED
3 - ON/RED
4 - ON/YELLOW
5 - ON
6 - OFF
7 - ON

If I am reading this correctly the combination of LEDs 2 and 7 tell the story of a CPU that starts up, overheats and halts. In short, the newly filled LCS is not doing a sufficient job of keeping the CPUs cool. I tried multiple reboots, resetting the PRAM and resetting the SMU, all to no effect. On one of the reboots, I actually got the OF System Picker screen to show, but the machine froze shortly thereafter. So, it is back to the drawing board I guess... sigh...

It is hard to tell what the actual root issue is at this point - clearly the general problem area is the LCS, but what is the specific problem? As I think about this just now, I think I will start by cracking open a hose at the very top of the unit and trying to add more coolant, just in case it is under filled. If that doesn't work, then I will need to take the step I was hoping to avoid: removing all the hoses, flushing the rad under pressure and then reinstalling them. Some new, transparent hosing may be in the cards, so I can see what is going on to some extent.

If anyone reading this has any wisdom to impart, I would be happy to hear it!

 

[mac57mac57]

...purchased a siphon pump today. I guess if I have to be a plumber to fix this LCS, I better have the right tools!

 

[MisterKeeks]

I rebuilt the single pump LCS this past summer. Some brief comments:
- I would advise getting new clear hoses. Otherwise you will not know what is going on WRT air pockets. I suspect there is a lot of air in the system.
- I recommend buying an electric fountain pump. Home improvement stores have them. Useful for two reasons:
1. This will let you flush large amounts of fluid through the radiator. My workflow was to place the radiator in an ultrasonic cleaner with a solution of water and a mild radiator cleaner from an auto parts store for about 30 minutes. Then I immediately removed the radiator and pumped water through it. You don’t want to leave radiator cleaner or vinegar in there. If you have two five gallon buckets you can have the fountain pump pump through the radiator from one bucket into another.
2. Having the pump makes refilling much easier. My methodology was to prepare much more coolant solution than necessary, place the solution into a 5 gallon bucket, then have the pump flush the solution into the almost sealed system. Almost all hoses were in place, I just disconnected one loop so that water could flow in from the pump and out from the other side of the loop. Once enough air had been forced out I then closed system by reconnecting the open loop.
- I tried pulling a vacuum on the system and refilling through the service port. I wouldn’t recommend this, it did not work at all and I was concerned it would damage the radiator.
- I believe I used Peak Gold antifreeze/coolant. IIRC what I used was universal so odds of a reaction with any existing coolant were minimized. I was satisfied that this would have sufficient corrosion inhibition, which was really the main concern. Coolants sold for PCs strike me as overpriced.
- I replaced the O-rings at the CPU thermal plates and the O-ring in the pump. I used Viton O-rings for both.

 

[mac57mac57]

Thanks @MisterKeeks, all very good advice, and I certainly am in need of some! Servicing a Quad LCS is turning out to be absolutely as difficult and frustrating as I thought it would, which is why I have avoided doing this for so long.

As I mentioned in another thread, I have two G5 Quads now, both with severe cooling issues. Both are unusably loud due to the fans running nearly full tilt to compensate for whatever coolant issues there are. My original G5 Quad actually passed thermal calibration on one of its two CPUs, so I extracted the entire LCS/CPU complex from it, repasted both CPUs and installed this LCS/CPU complex into my second Quad, the one that is the topic of this thread.

No joy. Things have just gone from bad to worse. The machine won't even boot now, nearly immediately turning on the "CPU Overtemp" diagnostic LED and freezing/halting. It won't even run thermal calibration, freezing in the middle of the first CPU's calibration procedure.

So, after a week and a half of work, I am now worse off than I was when I started. BOTH machines now no longer boot, whereas before I started, both booted and ran, just at very high fan revs and thus with intolerable noise. Now neither machine works at all. I am absolutely back where I started, and perhaps some distance behind that now.

On the plus side, I can now disassemble a Quad in a very short period of time. I know all the screws by sight and where each one goes, and I can reassemble the whole thing in a very short timeframe as well. That at least is a (dubious) benefit.

 

[DouglasCarroll]

Maybe a better title for this thread would have been…

“The Quad Chronicles - Destroying a Hot G5 Quad”!

😉

 

[DouglasCarroll]

Maybe a better title for this thread would have been…

“The Quad Chronicles - DESTROYING a Hot G5 Quad”!

😉

 

[mac57mac57]

@DouglasCarroll, 🤣

Sadly some elements of truth to that title, but neither machine is "destroyed". This is a setback, not the end of the road. I am nothing if not persistent and determined.

 

[mac57mac57]

Its amazing what a few more hours of work, more research and a little bit of inspiration can accomplish. FINALLY, one of the two Quads is now fully passing thermal calibration:

In this case, it is my original Quad. One CPU had passed thermal calibration a few days ago, so I took it apart, repasted both CPUs and reassembled the LCS.

Then, trying to "leap to solution", as reported yesterday I assembled that LCS into the new Quad I am restoring/destroying (!) and tried it out - as reported previously, it failed miserably.

After rereading the Apple Service Manual for the Quad, and realizing that it all but insists that the various LCS/CPU screws and attachment mechanisms must be connected in a specific order (which I had not followed during the above "leap" attempt) I decided to re-install the LCS back into my original Quad and carefully follow the Apple-stated order of screw and attachment installation.

Now realistically, since I had done nothing new to the LCS itself except uninstall/reinstall, I really doubted it would make any difference to the outcome, but that is where the "inspiration" part kicked in - I just had a feeling, so I went ahead and did it.

From previous attempts, the Apple Service Diagnostics 2.6.3 DVD was already in the DVD tray so when I got it all put back together, and with fingers and toes crossed, I hit the power button while holding down the Option key. Up came the System Picker screen and I selected the OF Diagnostics, which include Thermal Calibration. With no real expectation of success, I selected and started Thermal Calibration.

When Thermal Calibration does not fail, it takes a very long time - it says up to 30 minutes per CPU. So, I left it running, went downstairs and cooked up a special Valentines Day dinner for my wife and I. HOURS later I went back to check on it and could not believe my eyes - both CPUs had passed!

SO... I may have one fully operational Quad again, which will make working on the second one a lot more hopeful.

 

[Certificate of Excellence]

I rebuilt the single pump LCS this past summer. Some brief comments:
- I would advise getting new clear hoses. Otherwise you will not know what is going on WRT air pockets. I suspect there is a lot of air in the system.
- I recommend buying an electric fountain pump. Home improvement stores have them. Useful for two reasons:
1. This will let you flush large amounts of fluid through the radiator. My workflow was to place the radiator in an ultrasonic cleaner with a solution of water and a mild radiator cleaner from an auto parts store for about 30 minutes. Then I immediately removed the radiator and pumped water through it. You don’t want to leave radiator cleaner or vinegar in there. If you have two five gallon buckets you can have the fountain pump pump through the radiator from one bucket into another.
2. Having the pump makes refilling much easier. My methodology was to prepare much more coolant solution than necessary, place the solution into a 5 gallon bucket, then have the pump flush the solution into the almost sealed system. Almost all hoses were in place, I just disconnected one loop so that water could flow in from the pump and out from the other side of the loop. Once enough air had been forced out I then closed system by reconnecting the open loop.
- I tried pulling a vacuum on the system and refilling through the service port. I wouldn’t recommend this, it did not work at all and I was concerned it would damage the radiator.
- I believe I used Peak Gold antifreeze/coolant. IIRC what I used was universal so odds of a reaction with any existing coolant were minimized. I was satisfied that this would have sufficient corrosion inhibition, which was really the main concern. Coolants sold for PCs strike me as overpriced.
- I replaced the O-rings at the CPU thermal plates and the O-ring in the pump. I used Viton O-rings for both.

These are great recommendations. The clear tubing for example makes so much sense to me for actually seeing if/where air is trapped and using a pump to help push out remaining trapped air before closing the system - again so much sense here.

@mac57mac57 , glad you got one going!

 

[mac57mac57]

Hmmm... maybe not! Snatching defeat from the jaws of victory? The machine passes thermal calibration, but when I try to boot it, very quickly thereafter the "CPU Overtemp" diagnostic LED comes on and the machine halts. It never successfully completes a boot before the CPU freezes/halts.

How is this possible? If the CPUs pass thermal calibration, how can they overheat during booting? I understand that booting is a complex process that likely consumes a lot of the CPUs, but since those CPUs have passed calibration, I am at a loss to understand what is going on, or where to look next!

I am running the calibration again, just to see if yesterday's successful result was a fluke.

Does anyone have any other thoughts on this oddball scenario (CPUs pass thermal calibration but "CPU Overtemp" Diagnostic LED lights up when machine is booted).

 

[MisterKeeks]

I wouldn’t pay much attention to thermal calibration until you are confident the LCS is in good shape.

As to why it’d pass calibration but overtemp on boot, I don’t know. Possibly
- air in the system moved, impacting performance
- if this is the system you opened and replaced coolant in, there could have been a reaction with the previous coolant. I think it’s unlikely but I’ll put it out there.

 

[Certificate of Excellence]

- air in the system moved, impacting performance

This was my first thought as well - that there is a pocket of air that is moving through the system causing inconsistent results. I wonder if the pump is failing as in its not moving enough coolant?

 

[mac57mac57]

SO many possible causes!

Yesterday, I finally knuckled under and did the unthinkable; I cut the hose from the pump output and used it to drain the system. This is the system I have already drained and refilled, and I got all 130-140 ml of coolant back. Then I refilled using the other side of cut pipe as the fill line and this time got nearly 200 ml into the system. That 60-70 ml delta may explain the poor cooling performance last time ... not enough coolant.

Unfortunately, after I sealed it back up again, it leaked, making this a failed attempt.

I am off to the hardware store today to buy more tubing, clamps and hopefully a T-junction or two with integrated valves.

 

[mac57mac57]

Bowing to the inevitable, it seems certain that I will have to rebuild the whole cooling loop from scratch - starting with just the radiator and the pump and building back up from there.

Can anyone recommend a good set of clear hosing, of the right diameter and pressure rating, preferably with a reservoir and a T-junction? Bonus points for an integrated flow meter and coolant temperature sensor.

Thanks!

 

[MisterKeeks]

I might avoid integrated valves due to galvanic corrosion.

 

[MisterKeeks]

Bowing to the inevitable, it seems certain that I will have to rebuild the whole cooling loop from scratch - starting with just the radiator and the pump and building back up from there.

Can anyone recommend a good set of clear hosing, of the right diameter and pressure rating, preferably with a reservoir and a T-junction? Bonus points for an integrated flow meter and coolant temperature sensor.

Thanks!

I just used vinyl tubing bought by the foot from Menards.

 

[iModFrenzy]

Might be easier to air cool it at this point

Looks like a fun project but liquid cooled G5s are probably the last thing I’ll ever touch. Definitely sounded like you had air in the system though, hopefully a rebuild goes smoothly