G5 iMac + 10.4.11 = LITERAL Meltdown. Coincidence... Right?

[Makosuke]

Ok, I just had something really scary happen at work--MASSIVE iMac overheat.

I was doing updates to 10.4.11 on various Macs, and walked away from a G5 iMac (1.6GHz w/ambient light sensor, I think) for a few minutes after the update. Came back a little bit later, and tried to wake it from sleep... screen didn't come on when the light went off. Huh. I tapped the spacebar a couple times then went to work on something else to see if it was just taking longer than usual to wake.

When I came back a little later, I could smell something seriously wrong. I immediately unplugged it, but it appeared to have already shut itself off.

What appears to have happened is that it froze in a state where the fans didn't crank up to full bore like they're supposed to, and the processor massively overheated until the high-temperature cutoff killed it entirely.

A postmortem showed that:

a) The red "overtemp" light (#4) on the motherboard DID go on.
b) It appeared to only be the processor that had overheated--everything else felt (and smelled) normal.
c) The smell was, I hope, just the plastic airflow cover outgassing--nothing appeared damaged, and it was more of a "nasty plastic" smell than the odor of burning electronics I'm used to.
d) There was no apparent damage--when I powered it on, the fans ramped up until it was cool enough, and the hardware test plus TTD both said all was well.

So I'm thinking (hoping) that it killed itself safely before damage was done. Time will tell (and wouldn't you know it, AppleCare ran out last month).

But here's my question:

Anybody else EVER seen this? Is 10.4.11 working fine for everybody else? I'm DESPERATELY hoping that this was just a freak coincidence that it happened about 15 minutes after updating (combo) after 3 years of stable operation, but I'm kinda curious before I go messing with the other one.

MAN that was scary...

 

[dukebound85]

dang that sucks. hope it isnt all lost

 

[flyinmac]

I've been through this.

I had already read about this when it hit my system (it was only a couple of months old at the time). I immediately unplugged the computer, removed the cover, and placed a meat thermometer on the metal casing of the power supply. The case temperature of the power supply enclosure immediately ran the reading beyond the range of the thermometer.

I called Apple promptly, and got on the huge wait list for a replacement power supply.

This issue affected the early iMac G5 systems. Apple initiated a widespread repair procedure to address this.

Almost all of the original iMac G5 systems were affected. It turned-out to be a rather large batch of defective power supplies.

Due to a defect in the power supplies, they would overheat and die producing a burning smell. Often, the damage would continue to other components due to over-voltage produced by the power supply (and of course the extreme temperature).

Frequently, the replaced power supply would be followed with a replaced logic / main board. Often they would just replace them both at the same time because they knew it would be necessary anyway.

In my system, they replaced the power supply, then the logic board, then the LCD screen, and so on and so on.

I was still within warranty as it was only a couple months old.

For those who's warranty has expired, Apple introduced a warranty extension program. Hopefully yours will be covered.

http://docs.info.apple.com/article.html?artnum=302181-en

When I went through this, they had just discovered the issue, and they were taking between 3 weeks and 2 or 3 months to get through the backlog of repairs. The problem was that it was a huge batch of bad power supplies from a particular producer. So, they had to switch to another vendor, get enough inventory, and get through all the machines needing replacement power supplies.

For some history on the subject, start with the older posts on Apple's discussions here:

http://discussions.apple.com/search...eRange=all&searchID=1516445&rankBy=9&start=45

If that link is broken, here's a TinyURL version:
http://tinyurl.com/2e76n5

Work your way to the more recent posts on page 1. The older posts will be from when all this was actively going on. So, you'll get a better picture there.

Don't use the machine until you have had the power supply replaced. That is the major fire hazard. Run now, and unplug it. Leave it unplugged until you get it fixed with a new power supply. You may need other components after that, but the power supply is the major fire hazard.

For additional reading, check out the following Google search:

http://www.google.com/search?q='ima...s=org.mozilla:en-US:official&client=firefox-a

or TinyURL:
http://tinyurl.com/2y9xhs

Hopefully I've covered the important details on this in the post here. I've retyped this 4 times now due to Safari crashing right as I've finished each time. Hopefully, I didn't repeat anything more than once in my retyping (I was getting frustrated). I hope you'll forgive any repetitiveness.

So, hopefully I've remembered the stuff I've typed over and over again. I hate repeating myself.

Looks like 10.5.1 has a new crash bug in Safari.

Either way, I felt the warning and risk was important enough to keep trying to get a post through for you.

Unplug now. Get it fixed. Hopefully they'll cover it for free. If not, pay to get it fixed or discard the computer. It's a fire hazard at this point.

Edit:

I see now you say AppleCare expired last month. Even so, try calling them, be sure to mention the burning smell, and see if they'll replace it. If not, try it under the extension program I mentioned above. Push if you have to. This was something of a rather major incident with them. They'll know what it is, and they should repair it. Failure to do so, could lead to a liability suit for fire damage to your residence (or perhaps personal harm - which would really hurt them).


I hope I've been of help.

 

[Makosuke]

Thank you very much for the huge rundown, flyinmac. Very helpful, and I'll definitely see what Apple has to say about it even though it's out of warranty.

I'm not, however, entirely convinced that this is the same issue. I say this because the power supply was very definitely NOT hot to the touch, and it was very definitely NOT what was smelling funky--it was the first thing I checked, and I worked my way around the inside of the case checking each component and smelling it. The outlet of the processor heatsink was the only thing that was noticeably hot and definitely the only thing smelling funky.

Then again, it definitely could be just a funky edge case that appears different but isn't. So again, thanks hugely for the detailed review of the situation (not having seen the issue personally I'd forgotten about the batch of bad power supplies issue), and we'll see if that's what it was.

(Incidentally, bummer on the multi-post-loss--I hate that. A tip I've learned when typing hefty messages: Periodically hit command-A, command-C, down arrow. That'll copy the whole post and leave you back at the bottom, in the event the window dies. Or just use TextEdit or something, but I usually don't bother.)

 

[flyinmac]

You're welcome.

A point to consider, is that the cooling, circulation, and exhaust fans are generally routing air through the CPU tunnel and such. So, it could be that the smell was carried through by the fans.

Since the smell was widely reported as the first sign of trouble when all this was going on, it could be that the smell usually was carried by the fans and exhausted from the computer.

If it was isolated to the power supply, you would probably have to be at a closer proximity to smell it.

I seem to recall that on mine, that the burning smell was more coming from the computer in general when I popped the cover. I don't recall it being isolated to one specific area. But, it's been a long time, so I could be wrong.

I don't want you to burn anything down in the mean time. So, I think it would be best to consider the power supply bad until absolutely proven otherwise.

I cannot recall anyone reporting the burning smell, and having it turn out to be a different problem. All my recollection is the power supply leading to the burning smell.

It may not be that the power supply smelled. But, it was the primary cause of the burning smell.

Also, maybe you said, and I'm not remembering, but how long after the incident did you open the case?

I've done some power supply work over the years (in other electronic fields), and I remember accidentally hitting the wrong side of a live power supply a few times and causing quite a bit of damage. But, the smell from the PC-Type power supply dissipated quite quickly.

It may have more to do with certain compounds retaining scents longer than others. Just speculation.

I don't know why the smell would seem to be more from the CPU's. But, any guess is valid.

It could even be that the issue with the power supply causes an over-temp situation with the CPU's. Could be that the burning smell is in the CPU tunnel, but caused by the power supply issue.

It's all just speculation.

It's just that every case of burning smell I remember was related to the power supply issue in some way.

Keep us posted, and let us know what you learn.

In the mean time, be careful. Don't burn anything down.

 

[Makosuke]

Again, the concern and info is hugely appreciated, flyinmac.

You could definitely be right--it could be the PS and I was just smelling it in the area of the processor air outlet. But I'm inclined to believe it was overheated ducting plastic rather than burnt PCB or blown caps or something--I work at an energy research lab (where the computer is located), so I've smelled my share of blown electronics, and this wasn't in any of the categories of failure-related malodor I've smelled before.

I don't know exactly how long it had been off before I noticed, but it wasn't more than a few minutes. Hadn't been too long, since the red overheat light still came on at that point. I should have gone over the guts with a non-contact thermometer (didn't think of it at the time), but the heat was definitely isolated to directly over the processor--the entire PS case was only warm.

Regardless, the computer is never left to sleep overnight and the person who uses it is rarely away from his desk, so it's probably not a fire hazard. And since it has now been working fine (apparently) for the entire day without any odor, I'm willing to give it a cautious go before totally giving up on it or buying a replacement power supply.

I will be taking it back apart and re-checking tonight, though, as well as running some more thorough TTP diagnostics on it and doing a full backup mirror just in case.

Interestingly, the user said that the fans had been more quiet than usual, recently. He'd mentioned that the fans in that computer had a tendency to spin up pretty high when doing some things, so now I'm not sure what to think; I did resent the PMU (and NVRAM and such), so it is POSSIBLE that after a past firmware update something was left in a funky state that was causing unstable fans and just freaked out.

The other possibilities, of course, are that it fried the temp sensor so isn't thinking it's as hot as it is (though it's running fine and there's no odor so it's not AS bad, plus the thermocouples I'm used to pin high on failure, which would have the opposite effect). Or, it might just not be working right, although they ARE working (just quietly), and they definitely CAN spin up high, because they did right after I first turned it back on, and of course they pinned high when running the Apple hardware test.

Anyway, once again, thank you for the advice and info--it's very much appreciated.


Say, anybody know how to "hotwire" the power switch so I can turn it on with the back left open to check things?

 

[flyinmac]

I hope you're right, and that it wasn't a power related issue. I just haven't heard of that being the case before. But, there's always a first time.

Let us know what your research reveals.

As for turning it on with the cover off, you can do that.

There is a contact that the power button momentarily hits to turn on the computer. I think it is just shorting two contacts briefly if I remember correctly (which is exactly what the PC-type ATX power supplies do).

I also recall there being a secondary power button inside the computer on the logic board.

Yep, found it. You'll see it in this article:

http://docs.info.apple.com/article.html?artnum=300908

I think I always just bumped the two contacts that the button on the cover would bump when pressed.

 

[Makosuke]

Thanks for the tip--I double checked the power button itself, and it does just short those two contacts together, so tapping the pins was fine to turn it on.

After a more thorough examination, I'm 100% sure that if nothing else the smell was the processor and was NOT the power supply. Not that the PS couldn't have caused it indirectly, but the ariflow path of the processor doesn't have any contact with the PS, and the PS did not smell odd at ALL, so it wasn't what overheated.

I let it run full-bore to heat up, then went over the guts with a thermometer, and the temp sensors appear to all be reading above actual surface temps, which is what you'd expect (in fact, the surface-mounted probe on the hard drive was reading what my thermometer was to within 0.5C--impressively accurate), and they're responding properly, and all the fans are confirmed to be spinning visually, and ramp up with temp changes. So it appears that unless the processor later fails, there was no damage done.

The PS, incidentally (which has no built-in temperature probe that I saw) was reading about 42C on its hottest component, which is what I'd expect, and the case is cool enough to lay your hand on.

Guess it was just a freak firmware failure, unless it later turns out that something is failing intermittently.

Incidentally, I finally figured out what it smells like--rather similar to body odor. I can't think of anything electronic I'm aware of that smells similar, though a Google search did come up with someone describing a similar smell from overheated plastics in the speakers in their car.

Weird. I'll post if anything else interesting happens, and hopefully this does NOT have anything to do with 10.4.11...

 

[flyinmac]

If it wasn't the burning smell that smells like burned wire coating or an electrical short, then perhaps you only smelled the smell of extreme heat???

The G5's do run extremely hot. And, if the system crashes and the power management / cooling system fails to ramp up the fans properly, then it could over heat.

And, then the extreme heat would kick in the safe guards that cause the system to turn off.

So, that is a possibility.

It sounds like you have checked it out pretty well.

I would still monitor it closely, and if it does anything weird, then I would immediately shut it off / yank the plug, and pop it open for quick smell check and temperature reading manually.

A faulty / failing power supply could lead to over heating in the CPU area due to sending unexpected voltage to the processors.

As you probably know, the voltage fed to the processor is supposed to be very small. I don't recall the proper voltage for a G5, but I do recall that for some older AMD CPU's.

I recall them being 2.2 and 2.4 volts. And, prior to that generation, in the 3 to 4 volt range.

Running a 2.2 volt processor at 2.4 volts would cause it to run very hot (since it has extra energy to dissipate). And, voltage regulators keep the voltage approximately where they should be. But, the final voltage coming out of a voltage regulator is also affected by the voltage that enters the regulator. The regulator expects a certain feed voltage, and if that supply is not what is expected, then you could get something unexpected on the output.

So, it is possible that a failing power supply was sending unexpected voltage elsewhere to the rest of the computer, and caused some heat issues due to dissipating excess energy.

But, it is also possible that you just had a kernel panic, and the system got hot and shut down.

I'd monitor the smell closely. Tell the user to watch it very closely.

In the mean time, perhaps a reasonable safeguard would be to plug the iMac into a surge protector with a power switch. Then, ask the user to flip the surge protector's power switch off when he leaves for the night.

At least then you don't have a system with a risky power supply getting live power when no one is there.

Most surge protectors have a power switch.

Alternatively, perhaps a timer on the power line for that computer. Home depot has some that are intended to time lights and such.

For example, they have ones that replace your light switch on the wall, and turn on the lights at certain times and then turn them off at a certain time.

I'd probably trust something like that more than one of the little coffee pot timers that have the pegs that trip a switch (those scare me).

But, a $20 surge protector with a power switch would be a cheap and easy way to make sure someone is always there anytime the computer's power cord has power.

After a reasonable observation time, you could elect to plug it straight into whatever main power system you use at your company for computer power again.

 

[Makosuke]

Again, thank you very much for all the helpful suggestions, flyinmac. Very good advice all around, though you needn't worry about me burning the building down--as I mentioned before this computer is at an energy research lab, so safety with dodgy electronics is basically what we do (though usually it's dodgy stuff that we built). It's on a power strip already, and a lot of the people here flip the switches off at night anyway, to eliminate phantom loads.

Here's the final analysis after considerably more testing, for future reference:

The problem has something to do with malfunctioning sleep mechanisms. When the computer is put to sleep, one of four things will happen:

1) It will sleep and wake normally. I haven't tested thoroughly yet, but this appears to be more likely when it's warmed up.

2) The sleep light will throb weakly once, then the machine abruptly powers down completely. Will boot fine immediately afterward,though.

3) It will sleep normally, but when you try to wake it it shuts off abruptly. Again, restarts fine.

4) This is the bad one: It sleeps normally, but then when you wake it it only wakes HALFWAY up--the fans and drive kick in, and the processor turns on, but the screen stays powered off and it's dead in the water. In this state, it appears that it's running the processor at full power, but the fans do NOT respond to it heating up. Eventually (as I discovered) the machine shuts itself down due to the overtemp, but not before something gets hot enough to stink.

Although it occurred at the same time as an update to 10.4.11, that's not the cause thankfully--I tried it on a stock 10.4.0 install and the behavior was exactly the same.

There appears to have been no damage, even to the screen area above the processor, and the smell dissipated completely after a few hours, so it looks like whatever high-temp shutoff is built in did its job.

I know what just "very hot" electronics smell like (100A power supplies get toasty!), and I know what most sorts of burning stuff inside electronics smell like (melted wire insulation, blown caps, fried PCBs, blown ICs, etc), and this was none of the above, so whatever was outgassing/vaporizing isn't a common thing to cook--I'm guessing the plastic airflow guide over the G5 heatsink, though it could also have been whatever is between the processor and the back of the screen or I suppose even whatever thermal grease they use.

Whatever it was apparently wasn't necessary to the functioning of the computer, though.

I've got it set to never sleep, and it won't be used without somebody in front of it, so I think it's probably good for the time being. Since resetting any of the low-level things that can be rest didn't make any difference, I guess the only "fix" is replacing the midplane (or maybe the power supply? That'd be cheaper, though more unnerving). Dang.

Anybody else ever seen this sort of a sleep failure? Actually, I hope not...

 

[flyinmac]

Hi,

Glad to hear that you think you have it pinned-down.

The sleep issue is unusual. I cannot remember that specific behavior. I know the sleep issues with the G5 systems are not unusual. But, I don't recall that specific behavior.

As for the midplane, that would cost almost as much as a new iMac. The power supply is between $100 and $200 if I remember correctly.