16" MBP GPU Panics

[StolenPhone]

A lot of people are beginning to have GPU Panics on their machines similar to the following:

panic(cpu 0 caller 0xffffff7f8a3d1a97): GPU Panic: mux-regs 4 3 3f 1f 0 0 severity 3 WS-ready 
1 switch-state 0 IG FBs 1 EG FBs 0:0 power-state 4 3D idle HDA idle system-state 2 power-level 5:0 
power-retry 0:0 connect-change 0 : PowerPlay Failed Resume.

It seems to happen when running graphic intensive applications or when awaking from sleep with regard to the AMD processor. There is intermittent freezing of the screen as well. It starts slowly, perhaps a few times a week, then slowly progresses to multiple times per day. Folks have been advocated by Apple Engineering to try with a clean OS and nothing installed. However it appears that it still occurs. Most of those who got new machines no longer have the problem.

There are a couple of threads over in the Apple forums on the subject, with posts getting deleted by Apple for mentioning "cold solder joints". Maybe a nerve was hit. The slow creep of this getting worse and by people using the GPU heavily which creates heat would certainly be a culprit as it will worsen over time as the board heats and cools.

Has anyone else been seeing this problem? Its becoming fairly widespread.

 

[bsbeamer]

It seems to happen when running graphic intensive applications or when awaking from sleep with regard to the AMD processor.

Do you mean AMD GPU instead of AMD processor?
What exact configuration are you experiencing this issue with?

 

[StolenPhone]

AMD GPU 5500 8G. Yes the GPU. This is just the laptop alone, no external monitors. Apple Engineering had me try everything with a fresh OS install and nothing else installed. Its either a driver or hardware, but its looking like hardware as I have a friend with the exact same machine and no GPU panics. People in the Apple threads have said getting a new logic board from Apple fixed the issue. Its getting fairly widespread.

 

[bsbeamer]

I have 2.4/64GB/1TB/5500M 8GB and have NOT run into this specific panic issue.

Absolutely have issues with the AMD 5XXX drivers (especially with METAL tasks) in 10.15.2 when using 5500M. Those same issues are reproducible with RX 5700 XT in eGPU. That eGPU config also does not panic.

If this is a sleep issue, it would not be shocking. Sleep likes automatic graphics switching with the Intel GPU "better" than any other config. Hopefully one of those bugs that would be fixed in 10.15.3/10.15.4.

Have you disabled automatic graphics switching?
Have you tested with sleep disabled?

 

[StolenPhone]

Yep... all the above. BTW I have the exact same machine as you, only with a 2TB drive. I think its hardware :-(

 

[StolenPhone]

So this is becoming a fairly large issue as 300+ people are now showing this as a problem over on Apple's forums:

https://discussions.apple.com/thread/250905859

I spent 3 weeks with Apple engineering and it finally sent it in after Apple was at a loss. Its now on hold for a part to be shipped in from China. Several people have sent their MBPs in for fix and have confirmed a hardware issue and replacement. So far, Apple makes people jump through hoops to fix this when its been established that its a GPU Panic with either "PowerPlay Failed to Resume" or "PowerPlay failed to initialize". This starts out with freezing and reboots coming out of sleep maybe once every few days, but eventually degrades until it will only boot in SafeMode.

People now have their pitchforks out and are trying to get refunds on the 16" MBPs that are showing this issue. One person sent his in 3 times to get fixed, only to come back evincing the same problem. So it looks like Apple may have a serious problem on their hands with this.

This probably deserves some press on MacRumors' main site.

 

[bsbeamer]

Your exact post there appears to state:
"Yes, I have posted on MacRumors' forums, but not many people followed up. That also tells me its not as widespread"

Why does this "deserve some press" if it's not widespread by your own conclusion?
How is it a "serious problem" if it's not widespread?

And those "I have this question too" numbers are artificially inflated and do not correlate to people who also are experiencing this issue.

 

[TT_Guy]

panic(cpu 0 caller 0xffffff800c4a2d1c): Sleep transition timed out after 180 seconds while calling power state change callbacks. Suspected bundle: __kernel__. Thread 0x195d8.


Backtracing specified thread


Backtrace (CPU 0), Frame : Return Address


0xffffffa77e8bb900 : 0xffffff800be60178


0xffffff87562b3ab0 : 0xffffff800bd5d771


0xffffff87562b3b20 : 0xffffff800bd5bf6f


0xffffff87562b3b70 : 0xffffff800bd1db01


0xffffff87562b3bb0 : 0xffffff800bd4286b


0xffffff87562b3c00 : 0xffffff800c4856d7


0xffffff87562b3c60 : 0xffffff800c4852ea


0xffffff87562b3cb0 : 0xffffff800c3f7792


0xffffff87562b3dc0 : 0xffffff800c48197d


0xffffff87562b3e00 : 0xffffff800c42794b


0xffffff87562b3ea0 : 0xffffff800c427344


0xffffff87562b3ec0 : 0xffffff800bd7d7e5


0xffffff87562b3f40 : 0xffffff800bd7d311


0xffffff87562b3fa0 : 0xffffff800bce213e





BSD process name corresponding to current thread: kernel_task


Boot args: chunklist-security-epoch=0 -chunklist-no-rev2-dev





Mac OS version:


19D76





Kernel version:


Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64


Kernel UUID: A8DDE75C-CD97-3C37-B35D-1070CC50D2CE


Kernel slide: 0x000000000ba00000


Kernel text base: 0xffffff800bc00000


__HIB text base: 0xffffff800bb00000


System model name: MacBookPro16,1 (Mac-E1008331FDC96864)


System shutdown begun: NO





System uptime in nanoseconds: 10455407152528


last loaded kext at 1486356646720: >usb.IOUSBHostHIDDevice 1.2 (addr 0xffffff7f8fcb2000, size 45056)


last unloaded kext at 209083438247: >usb.!UHostPacketFilter 1.0 (addr 0xffffff7f8e5ea000, size 24576)


loaded kexts:


@kext.AMDRadeonX6000 3.0.5


@kext.AMDRadeonServiceManager 3.0.5


>!AGraphicsDevicePolicy 4.7.2


@fileutil 20.036.15


@AGDCPluginDisplayMetrics 4.7.2


>!AHV 1


|IOUserEthernet 1.0.1


|IO!BSerialManager 7.0.3f5


>pmtelemetry 1


>!APlatformEnabler 2.7.0d0


>AGPM 111.4.2


>X86PlatformShim 1.0.0


>!AUpstreamUserClient 3.6.8


@Dont_Steal_Mac_OS_X 7.0.0


>AGDCBacklightControl 4.7.2


>BridgeAudioCommunication 6.66


>!A!IKBLGraphics 14.0.4


>!AThunderboltIP 3.1.3


>!AHIDALSService 1


>!AAVEBridge 6.1


>!A!ICFLGraphicsFramebuffer 14.0.4


>!A!ISlowAdaptiveClocking 4.0.0


>!ABridgeAudio!C 6.66


>!A!IPCHPMC 2.0.1


>!AMCCSControl 1.13


>!ATopCaseHIDEventDriver 3430.1


>!AGFXHDA 100.1.424


>!AMuxControl2 4.7.2


@filesystems.autofs 3.0


>BCMWLANFirmware4355.Hashstore 1


>BCMWLANFirmware4364.Hashstore 1


>BCMWLANFirmware4377.Hashstore 1


>!ABCMWLANBusInterfacePCIe 1


@filesystems.hfs.kext 522.0.9


@BootCache 40


@!AFSCompression.!AFSCompressionTypeDataless 1.0.0d1


@!AFSCompression.!AFSCompressionTypeZlib 1.0.0


>!AVirtIO 1.0


@filesystems.apfs 1412.81.1


>!A!II210Ethernet 2.3.1


@private.KextAudit 1.0


>!ASmartBatteryManager 161.0.0


>!AACPIButtons 6.1


>!ASMBIOS 2.1


>!AACPIEC 6.1


>!AAPIC 1.7


$!AImage4 1


@nke.applicationfirewall 303


$TMSafetyNet 8


@!ASystemPolicy 2.0.0


|EndpointSecurity 1


>usb.IOUSBHostHIDDevice 1.2


@kext.AMDRadeonX6100HWLibs 1.0


@kext.AMDRadeonX6000HWServices 3.0.5


|IOAVB!F 800.17


>!ASSE 1.0


@!AGPUWrangler 4.7.2


>!ABacklightExpert 1.1.0


@kext.AMDRadeonX6000Framebuffer 3.0.5


>!AHIDKeyboard 209


>!UAudio 320.49


|IOAccelerator!F2 438.3.1


|IOSlowAdaptiveClocking!F 1.0.0


>!ASMBus!C 1.0.18d1


>X86PlatformPlugin 1.0.0


>IOPlatformPlugin!F 6.0.0d8


>!AActuatorDriver 3430.1


>!AHS!BDriver 3430.1


>IO!BHIDDriver 7.0.3f5


>!AMultitouchDriver 3430.1


>!AInputDeviceSupport 3430.1


|IONDRVSupport 569.4


|IO!BHost!CUARTTransport 7.0.3f5


|IO!BHost!CTransport 7.0.3f5


>!A!ILpssUARTv1 3.0.60


>!A!ILpssUARTCommon 3.0.60


>!AOnboardSerial 1.0


@kext.AMDSupport 3.0.5


@!AGraphicsDeviceControl 4.7.2


|IOGraphics!F 569.4


>!AGraphicsControl 4.7.2


@plugin.IOgPTPPlugin 810.1


@kext.triggers 1.0


>Core!S 551


>usb.cdc.ncm 5.0.0


>usb.!UHub 1.2


>!AThunderboltPCIUpAdapter 2.5.4


>!AThunderboltDPOutAdapter 6.2.5


>usb.cdc 5.0.0


>usb.networking 5.0.0


>usb.!UHostCompositeDevice 1.2


>!ABCMWLANCore 1.0.0


>IOImageLoader 1.0.0


|IO80211!FV2 1200.12.2b1


>corecapture 1.0.4


|IOSkywalk!F 1


|IOSurface 269.6


@filesystems.hfs.encodings.kext 1


|IOAudio!F 300.2


@vecLib.kext 1.2.0


|IOSerial!F 11


>usb.!UVHCIBCE 1.2


>usb.!UVHCI 1.2


>usb.!UVHCICommonBCE 1.0


>usb.!UVHCICommon 1.0


>!AEffaceableNOR 1.0


|IOBufferCopy!C 1.1.0


|IOBufferCopyEngine!F 1


|IONVMe!F 2.1.0


>!AThunderboltPCIDownAdapter 2.5.4


>!AThunderboltDPInAdapter 6.2.5


>!AThunderboltDPAdapter!F 6.2.5


>!AHPM 3.4.4


>!A!ILpssI2C!C 3.0.60


>!A!ILpssDmac 3.0.60


>!A!ILpssI2C 3.0.60


>!AThunderboltNHI 5.8.6


|IOThunderbolt!F 7.6.0


|IOEthernetAVB!C 1.1.0


>mDNSOffloadUserClient 1.0.1b8


|IOUSB!F 900.4.2


>usb.!UXHCIPCI 1.2


>usb.!UXHCI 1.2


>!AEFINVRAM 2.1


>!AEFIRuntime 2.1


>!ASMCRTC 1.0


|IOSMBus!F 1.1


|IOHID!F 2.0.0


$quarantine 4


$sandbox 300.0


@Kext.!AMatch 1.0.0d1


>!AKeyStore 2


>!UTDM 489.80.2


|IOSCSIBlockCommandsDevice 422.0.2


>!ACredentialManager 1.0


>KernelRelayHost 1


>!ASEPManager 1.0.1


>IOSlaveProcessor 1


>!AFDEKeyStore 28.30


>!AEffaceable!S 1.0


>!AMobileFileIntegrity 1.0.5


@kext.CoreTrust 1


|CoreAnalytics!F 1


|IOTimeSync!F 810.1


|IONetworking!F 3.4


>DiskImages 493.0.0


|IO!B!F 7.0.3f5


|IO!BPacketLogger 7.0.3f5


|IOUSBMass!SDriver 157.40.7


|IOSCSIArchitectureModel!F 422.0.2


|IO!S!F 2.1


|IOUSBHost!F 1.2


>usb.!UCommon 1.0


>!UHostMergeProperties 1.2


>!ABusPower!C 1.0


|IOReport!F 47


>!AACPIPlatform 6.1


>!ASMC 3.1.9


>watchdog 1


|IOPCI!F 2.9


|IOACPI!F 1.4


@kec.pthread 1


@kec.corecrypto 1.0


@kec.Libm 1

 

[StolenPhone]

Your exact post there appears to state:
"Yes, I have posted on MacRumors' forums, but not many people followed up. That also tells me its not as widespread"

View attachment 892888

Why does this "deserve some press" if it's not widespread by your own conclusion?
How is it a "serious problem" if it's not widespread?

And those "I have this question too" numbers are artificially inflated and do not correlate to people who also are experiencing this issue.

Widespread is relative. 300+ people is a problem and probably deserves press. Its certainly not 1000s... but its big enough to ask WTF. Its not 1 or 2 guys with the issue. It seems more with over 11 pages of discussion. Yes I think 11 pages of the same issue is probably worth some press. But that's just me.

I don't know if the problems are artificially inflated, but sure. I guess I'm one of the honest ones who clicks it once.

 

[ondrejz]

I haven't experienced the freezing yet. Once it's to that point, Apple gets to replace it.


But I've stumbled upon a "solution" that worked for the wake/powerplay kernel panic issue I have. You mention disabling sleep, try disabling standby instead. Confusing, but they are different states.

From Terminal:
pmset -a standby 0

Just change the 0 to a 1 to turn it back on if you need to later on.
To see if it's off/on and other energy saving features, use:
pmset -g

Zero issues waking for a few days now. This is with auto gfx switching turned on as well.

I'm not sure why this is working for me. Standby is set to kick in after 3 hours (standbyminimumdelay) by default. Using Chrome to watch an HD video, I could get it to consistently crash on wake waiting just a few minutes. But I can't even reproduce that anymore. Not sure what's going on low level but still looking for ideas as this isn't an acceptable fix long-term.

 

[StolenPhone]

Yes, I tried that but it didn't work. I read that standby is disbale when connected to power anyways.

At any rate, its at Apple now and it's "Waiting for a part". Its been a few days, so I'm guessing they are sending the part from China via ship... a nice looonnng wait. :-(

 

[ondrejz]

Yeah I spoke too soon... Everything turned on this time but was totally frozen then restarted. Error showed "Hibernate restore error", which I hadn't seen before. I give up on troubleshooting and believing it's Catalina. Only a few weeks old. Wonder if returning it at this point is my best option. Zero issues out of my 6 year old MBP. Super disappointed. This MB was supposed to be the one worth upgrading for...

 

[turbineseaplane]

Man - the quantity of threads about the new 16”s and issues with Catalina...

It’s all so disappointing.

It feels like Apple has lost their way somehow. Even when they finally do release something we think will be “it” —- they are still replete with endless bugs and issues.

It is maddening that a company with this amount of resources can’t (or won’t) get things right.

 

[profcutter]

I wonder if it’s a matter of needing a couple of years to right the ship. The last 4 years were a disaster on the Mac side, trash can Mac Pro, skinny throttled mbp’s, Buggy OS releases (permanently deleting emails? Really Apple?)
The optimist in me thinks that maybe they’re trying to find their way back, and the mbp 16 is a first, though unsteady step. The political economist in me says no way in hell, they’ll keep de-contenting and cheaping out, hoping they’ll stay relevant through momentum. The technologist in me says ugh... Intel.

 

[StolenPhone]

Yep.... I totally agree. They concentrate on the thinness and sexiness of the product rather than the practicality. The Air should be for people who want that thin and tight design. The Pro should be for pros. I'm a software developer and I sometimes get very dismayed at what Apple does with these machines. I write AI software and need a real graphics card in them (READ: NVidia). But due to Apple's little tiff with them, I can't do that and all AI GPU training I have to move it to a Windows or Linux machine (or Cloud for $$) in order to properly run this. Its a shame because the MacOS is the near perfect development machine. Pro users should be able to have a laptop that is easily upgradable with memory and battery. Dont get me started about the battery which if you delay fixing will check and expand, breaking your computer....

Regardless... back to the thread... this GPU is a disaster and its reminiscent of the NVidia fiasco from 2012. I'm wondering if its faulty AMD hardware and if AMD is going to pay for the fixes because this is going to cost Apple. My concerns at this stage is some people have gotten fixes back from APple with this issue only starting again for them. So this is deep and I'm not sure if Apple knows exactly what the cause is. They seem lost on this.

A quick update... I escalated this to Tara Bunch (VP of Apple Care), and the executive team took it over. They agreed this was unacceptable since I am approaching a month from when I opened the case. I spoke ot them on the phone and they did acknowledge a serious problem. So I will update when I hear back.

 

[profcutter]

Thanks for the update. I’m hoping they come up with a good working solution for you.

 

[StolenPhone]

So the executive team got back to me. They gave me more info. They are waiting on a logic board. I guess they are plumb out of them and I think they need to get it from China. They confirmed the GPU failure is caused by the logic board.

 

[simonmet]

Looks like I’ll have to find the money for AppleCare, because hardware problems are just all too common on these advanced; increasingly complicated; and increasingly fragile (It seems) devices.

For a long time “Pro” meant: stable, rugged, reliable. But Apple marketing took the word and twisted it into anything but those things. I don’t know if this endless pursuit of better performance is worth it if we have to sacrifice reliability. That in part explains why the Mac Pro isn’t the best value on a purely performance basis if you don’t factor stability and reliability. Yes pros want performance, but also those other things. That’s what they pay for; and we should be getting with the MBP as well.

 

[turbineseaplane]

Looks like I’ll have to find the money for AppleCare

Don’t kid yourself....this is all part of Tim’s plan.
Extra money from the existing user base for services, warranties and accessories.

Apple has literally turned into a very shiny, high minded design, nickel and diming operation.

 

[Jimmy Bubbles]

A lot of people are beginning to have GPU Panics on their machines similar to the following:

panic(cpu 0 caller 0xffffff7f8a3d1a97): GPU Panic: mux-regs 4 3 3f 1f 0 0 severity 3 WS-ready
1 switch-state 0 IG FBs 1 EG FBs 0:0 power-state 4 3D idle HDA idle system-state 2 power-level 5:0
power-retry 0:0 connect-change 0 : PowerPlay Failed Resume.

It seems to happen when running graphic intensive applications or when awaking from sleep with regard to the AMD processor. There is intermittent freezing of the screen as well. It starts slowly, perhaps a few times a week, then slowly progresses to multiple times per day. Folks have been advocated by Apple Engineering to try with a clean OS and nothing installed. However it appears that it still occurs. Most of those who got new machines no longer have the problem.

There are a couple of threads over in the Apple forums on the subject, with posts getting deleted by Apple for mentioning "cold solder joints". Maybe a nerve was hit. The slow creep of this getting worse and by people using the GPU heavily which creates heat would certainly be a culprit as it will worsen over time as the board heats and cools.

Has anyone else been seeing this problem? Its becoming fairly widespread.

It has been happening to me in the past couple of weeks; with a fully-loaded machine, or even a clean install after wiping and reformatting. I have a Service Non-Repair setup with my local Apple Store. If it happens again (which it has already) then I am dropping it off to the depot.

 

[simonmet]

Don’t kid yourself....this is all part of Tim’s plan.
Extra money from the existing user base for services, warranties and accessories.

Apple has literally turned into a very shiny, high minded design, nickel and diming operation.

It’s been that way for a while.

 

[Jimmy Bubbles]

Mine is at the depot as I’m typing this message. I asked to speak to a supervisor with AppleCare and he advised this was the best plan of action for this time.

 

[ondrejz]

It has been happening to me in the past couple of weeks; with a fully-loaded machine, or even a clean install after wiping and reformatting. I have a Service Non-Repair setup with my local Apple Store. If it happens again (which it has already) then I am dropping it off to the depot.

If you need to reproduce it for them, what's worked for me consistently is loading up a YouTube HD video in Chrome, full screen it, let it play for around 10 seconds, pause, close lid (keep it all open), let it sit 10-15 minutes, open laptop, crash.

 

[StolenPhone]

A quick update...

Apple shipped me back my broken machine to hobble along until they can get me a new one. I still don't have the new one as of Feb 29. 2019. They claim that the shipping labels are about to be created to ship it to me. The main delay is the supply side shipment issues from China.

More info from Apple is they are aware of a batch of MBPs with these bad logic boards and have made a "fix" to the hardware.

If you have this problem, you should open a case right now. Don't let the Apple support folks give you the run around and try everything under the sun to make this a software issue and after days find out its not. Demand to the support folks that you are clearly aware of what the problem is and it needs to be fixed. Just tell them that you tried everything. Once it hits back end engineering they will confirm the GPU panics that are caused when going to sleep/waking or when booting. My suggestion is to be sure they have logic boards in stock at their repair facility before you ship it in. You can hobble along while you wait. My laptop was in Apple's hands for 3 weeks with no movement due to inability to get a logic board and I finally had them ship it back to me to use until they could get a new MBP for me. If you have a non-customized MBP, they can usually get you a replacement in short order. The delays are generally for those with customized configurations (bigger SSD, faster processor, bigger GPU, more memory).

To hobble along, you should set your Energy settings to not go to sleep on power. Forget about using the battery. The bottom line is you do not want the AMD GPU to become active. You must have the switch GPUs automatically set, because thats the only way to use the Intel GPU. You also need to avoid any apps that require a GPU. That means no games, no Adobe products, and no Chrome. Use Firefox for web. If you can live with using apps with no need for the AMD GPU, you can keep the machine running for large swaths of time without a panic reboot.

Also, if you are unable to boot into normal mode, boot into safe mode.... check your energy settings as described above, and then you should be able to reboot into normal mode.

This is definitely a hardware issue if you are getting the "Powerplay failed to resume" or "Powerplay failed to initialize". Don't even try to hope that it is software fixable. It HAS to go back to Apple for replacement or fix. Don't delay.

 

[StolenPhone]

So hats off to visualinventor in the Apple forums. He was able to reproduce this faithfully by running Geekbench 5. He found that its actually the Intel GPU causing this by running the metal tests against the Metal api.

I decided to try this and sure enough, it GPU panics on the Intel GPU with metal right at the start with the Sobel tests. It does NOT panic with the OpenCL or either test against the AMD.

So its the Intel after all.

You can read up on what he did and his findings here as well as how to test this yourself:

https://discussions.apple.com/thread/250905859?answerId=252207020022#252207020022

The errors appear to be in the kernel with a signalHardwareError that point towards IOAccelFenceMachine::fence_timeout(IOTimerEventSource *): IntelAccelerator prodding blockFenceInterrupt.

The signalHardwareError is a strong indicator that this is indeed a hardware problem.

Also, now it makes sense why this does not occur in Safe mode because none of the metal drivers are loaded.

For those having freezing issues etc, I advise you to try this and see if you get similar results (a super low Geekbench score or a GPU panic), of which its time to get your MBP fixed at Apple.