MP 7,1 Random panic on 2019 Mac Pro

[vailance88]

Im constantly having issue with the new mac pro as it crashes and reboot randomly. below is the crash report, i found that it happened since the day I've added the extra crucial memory. I do not think is the Radeon 7 and sleep was disabled (avoid fans running max)


panic(cpu 16 caller 0xffffff7f8cb9dad5): userspace watchdog timeout: no successful checkins from com.apple.WindowServer in 120 seconds
service: com.apple.logd, total successful checkins since load (84560 seconds ago): 8457, last successful checkin: 0 seconds ago
service: com.apple.WindowServer, total successful checkins since load (84530 seconds ago): 8441, last successful checkin: 120 seconds ago

Backtrace (CPU 16), Frame : Return Address
0xffffff8ab8b53820 : 0xffffff800c13bb2b
0xffffff8ab8b53870 : 0xffffff800c2734d5
0xffffff8ab8b538b0 : 0xffffff800c264f4e
0xffffff8ab8b53900 : 0xffffff800c0e2a40
0xffffff8ab8b53920 : 0xffffff800c13b217
0xffffff8ab8b53a20 : 0xffffff800c13b5fb
0xffffff8ab8b53a70 : 0xffffff800c8d2b25
0xffffff8ab8b53ae0 : 0xffffff7f8cb9dad5
0xffffff8ab8b53af0 : 0xffffff7f8cb9d7e6
0xffffff8ab8b53b10 : 0xffffff800c86739b
0xffffff8ab8b53b60 : 0xffffff800c870443
0xffffff8ab8b53ca0 : 0xffffff800c222d12
0xffffff8ab8b53db0 : 0xffffff800c1419d8
0xffffff8ab8b53e10 : 0xffffff800c118635
0xffffff8ab8b53e70 : 0xffffff800c12f0e5
0xffffff8ab8b53f00 : 0xffffff800c24b575
0xffffff8ab8b53fa0 : 0xffffff800c0e3226
Kernel Extensions in backtrace:
com.apple.driver.watchdog(1.0)[053A5D15-51D4-3E61-978B-EB435FA4BD0A]@0xffffff7f8cb9c000->0xffffff7f8cba4fff

BSD process name corresponding to current thread: watchdogd
Boot args: chunklist-security-epoch=0 -chunklist-no-rev2-dev

Mac OS version:
19D76

Kernel version:
Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64
Kernel UUID: A8DDE75C-CD97-3C37-B35D-1070CC50D2CE
Kernel slide: 0x000000000be00000
Kernel text base: 0xffffff800c000000
__HIB text base: 0xffffff800bf00000
System model name: MacPro7,1 (Mac-27AD2F918AE68F61)
System shutdown begun: NO

System uptime in nanoseconds: 84565625799896
last loaded kext at 703353431554: @filesystems.smbfs 3.4.1 (addr 0xffffff7f944fb000, size 446464)
last unloaded kext at 84509329703252: >!A!ILpssGspi 3.0.60 (addr 0xffffff7f8cb55000, size 45056)
loaded kexts:
com.vmware.kext.vmioplug.18.7.0 18.7.0
com.vmware.kext.vmx86 1501.84.42
com.vmware.kext.vmnet 1501.84.42
com.intel.driver.EnergyDriver 3.6.1
@filesystems.smbfs 3.4.1
>!ARAID 5.1.0
>!ATopCaseHIDEventDriver 3430.1
>!AUpstreamUserClient 3.6.8
>!AMCCSControl 1.13
@kext.AMDFramebuffer 3.0.5
@kext.AMDRadeonX5000 3.0.5
@kext.AMDRadeonX4000 3.0.5
@kext.AMDRadeonServiceManager 3.0.5
>!AGraphicsDevicePolicy 4.7.2
@AGDCPluginDisplayMetrics 4.7.2
@fileutil 20.036.15
>!AHV 1
|IOUserEthernet 1.0.1
|IO!BSerialManager 7.0.3f5
>pmtelemetry 1
>AGPM 111.4.2
>!APlatformEnabler 2.7.0d0
>X86PlatformShim 1.0.0
@Dont_Steal_Mac_OS_X 7.0.0
>AudioAUUC 1.70
@kext.AMD10000!C 3.0.5
@kext.AMD9500!C 3.0.5
>BridgeAudioCommunication 6.66
>!AThunderboltIP 3.1.3
>!ABridgeAudio!C 6.66
>!AGFXHDA 100.1.424
>!AAVEBridge 6.1
>!A!IPCHPMC 2.0.1
>!A!IMCEReporter 115
>!A!ISlowAdaptiveClocking 4.0.0
@filesystems.autofs 3.0
@filesystems.exfat 1.4
>BCMWLANFirmware4355.Hashstore 1
>BCMWLANFirmware4364.Hashstore 1
>BCMWLANFirmware4377.Hashstore 1
@filesystems.hfs.kext 522.0.9
@BootCache 40
@!AFSCompression.!AFSCompressionTypeDataless 1.0.0d1
@!AFSCompression.!AFSCompressionTypeZlib 1.0.0
>!AVirtIO 1.0
>!ABCMWLANBusInterfacePCIe 1
@filesystems.apfs 1412.81.1
@private.KextAudit 1.0
>!AAHCIPort 341.0.2
>!AACPIButtons 6.1
>!ASMBIOS 2.1
>!AACPIEC 6.1
>!AAPIC 1.7
$!AImage4 1
@nke.applicationfirewall 303
$TMSafetyNet 8
@!ASystemPolicy 2.0.0
|EndpointSecurity 1
>!AHIDKeyboard 209
>!AMultitouchDriver 3430.1
>!AInputDeviceSupport 3430.1
>!AHS!BDriver 3430.1
>IO!BHIDDriver 7.0.3f5
>!ASMBus!C 1.0.18d1
@kext.AMDRadeonX5000HWLibs 1.0
@kext.AMDRadeonX4700HWLibs 1.0
|IOAccelerator!F2 438.3.1
@kext.AMDRadeonX5000HWServices 3.0.5
@kext.AMDRadeonX4000HWServices 3.0.5
>!AGraphicsControl 4.7.2
|IOAVB!F 800.17
>!ASSE 1.0
|IO!BHost!CUARTTransport 7.0.3f5
|IO!BHost!CTransport 7.0.3f5
@!AGPUWrangler 4.7.2
>usb.cdc.acm 5.0.0
>usb.serial 6.0.0
>!UAudio 320.49
|IONDRVSupport 569.4
>!A16X50Serial 3.2
@kext.AMDSupport 3.0.5
@!AGraphicsDeviceControl 4.7.2
|IOGraphics!F 569.4
>X86PlatformPlugin 1.0.0
>IOPlatformPlugin!F 6.0.0d8
|IOSlowAdaptiveClocking!F 1.0.0
@plugin.IOgPTPPlugin 810.1
|IOEthernetAVB!C 1.1.0
@kext.triggers 1.0
>usb.cdc.ncm 5.0.0
>!AThunderboltDPOutAdapter 6.2.5
>!AThunderboltPCIUpAdapter 2.5.4
>usb.!UHub 1.2
>usb.cdc 5.0.0
>usb.networking 5.0.0
>usb.!UHostCompositeDevice 1.2
|IOSurface 269.6
@filesystems.hfs.encodings.kext 1
|IOAudio!F 300.2
@vecLib.kext 1.2.0
>!AThunderboltDPInAdapter 6.2.5
>!AThunderboltDPAdapter!F 6.2.5
>!AThunderboltPCIDownAdapter 2.5.4
>!AHPM 3.4.4
>!A!ILpssI2C!C 3.0.60
>!A!ILpssDmac 3.0.60
>!AEthernetAquantiaAqtion 1.0.64
>!ABCMWLANCore 1.0.0
>mDNSOffloadUserClient 1.0.1b8
>IOImageLoader 1.0.0
|IOSerial!F 11
|IO80211!FV2 1200.12.2b1
>corecapture 1.0.4
|IOSkywalk!F 1
>!AThunderboltNHI 5.8.6
|IOThunderbolt!F 7.6.0
>usb.!UVHCIBCE 1.2
>usb.!UVHCI 1.2
>usb.!UVHCICommonBCE 1.0
>usb.!UVHCICommon 1.0
>!AEffaceableNOR 1.0
|IOUSB!F 900.4.2
|IOBufferCopy!C 1.1.0
|IOBufferCopyEngine!F 1
|IOAHCIBlock!S 316.80.1
|IONVMe!F 2.1.0
>!A!ILpssI2C 3.0.60
|IOAHCI!F 290.0.1
>usb.!UXHCIPCI 1.2
>usb.!UXHCI 1.2
>!AEFINVRAM 2.1
>!AEFIRuntime 2.1
>!ASMCRTC 1.0
|IOSMBus!F 1.1
|IOHID!F 2.0.0
$quarantine 4
$sandbox 300.0
@Kext.!AMatch 1.0.0d1
>!AKeyStore 2
>!UTDM 489.80.2
|IOSCSIBlockCommandsDevice 422.0.2
>!ACredentialManager 1.0
>KernelRelayHost 1
>!ASEPManager 1.0.1
>IOSlaveProcessor 1
>!AFDEKeyStore 28.30
>!AEffaceable!S 1.0
>!AMobileFileIntegrity 1.0.5
@kext.CoreTrust 1
|CoreAnalytics!F 1
|IOTimeSync!F 810.1
|IONetworking!F 3.4
>DiskImages 493.0.0
|IO!B!F 7.0.3f5
|IO!BPacketLogger 7.0.3f5
|IOUSBMass!SDriver 157.40.7
|IOSCSIArchitectureModel!F 422.0.2
|IO!S!F 2.1
|IOUSBHost!F 1.2
>usb.!UCommon 1.0
>!UHostMergeProperties 1.2
>!ABusPower!C 1.0
|IOReport!F 47
>!AACPIPlatform 6.1
>!ASMC 3.1.9
>watchdog 1
|IOPCI!F 2.9
|IOACPI!F 1.4
@kec.pthread 1
@kec.corecrypto 1.0
@kec.Libm 1

 

[bxs]

I've seen this a few times on my new MP7,1 as well. It typically happens when I Shutdown and or Startup at times. I've sent in a bug Report report to apple with a Sysdiagnose report. So hopefully they will figure it out.

 

[awkward_eagle]

i found that it happened since the day I've added the extra crucial memory

Does it happen if you remove the Crucial memory? Maybe the new Mac Pro doesn't liked mixed size memory. Can see the System Info window says they check out, so it might just be a bad stick. Running something like Remember ( memory test app ) might reveal something.

Haven't seen any other instances of a Radeon VII causing panics. Fingers crossed since I just got 2.

 

[Snow Tiger]

Your panic happened after your Mac was running for nearly 24 hours since last boot . Interesting .

Run memtest on your memory in its own USB drive bootable environment .

Don't run this test in a macOS session , as it will be incomplete .

This should take around 4 hours with a USB SSD .

MemTest86 - Download now!

Download your free copy of MemTest86 here for USB.

www.memtest86.com

 

[ibuick]

actually, this may not be related to any ram issues. This panic is caused by WindowServer, which a process responses for showing all the GUI objects of macOS, hangs or dead.

"watchdogd ensures that the system is healthy and able to make forward progress throughout the system lifecycle. If watchdogd or the Watchdog KEXT determine that the system is unhealthy they will attempt to take corrective action and ultimately may panic the system to get it back to a usable state."

WindowsServer and many of the system services registered to watchdogd and repeating checked in within a time interval. If one of those services failed to check in to watchdogd in the configured interval, then watchdogd think this service may be failed or hang. In order to avoid the consequences issues, watchdogd issues the panic.

So, I think the problem related to WindowsServer can be something wrong with GPU or Graphics configuration. under the /Library/Logs/DiagnosticReports/ you may find some logs ends with tailspin or something else you can grab them and then give it a review.

 

[vailance88]

actually, this may not be related to any ram issues. This panic is caused by WindowServer, which a process responses for showing all the GUI objects of macOS, hangs or dead.

"watchdogd ensures that the system is healthy and able to make forward progress throughout the system lifecycle. If watchdogd or the Watchdog KEXT determine that the system is unhealthy they will attempt to take corrective action and ultimately may panic the system to get it back to a usable state."

WindowsServer and many of the system services registered to watchdogd and repeating checked in within a time interval. If one of those services failed to check in to watchdogd in the configured interval, then watchdogd think this service may be failed or hang. In order to avoid the consequences issues, watchdogd issues the panic.

So, I think the problem related to WindowsServer can be something wrong with GPU or Graphics configuration. under the /Library/Logs/DiagnosticReports/ you may find some logs ends with tailspin or something else you can grab them and then give it a review.

u are right i noticed a few times before it crashes, my screen was frozen i cant move anything until it reset itself.

 

[goMac]

It _could_ be a bad GPU. Not necessarily all Radeon VIIs but maybe yours just nosedives every so often.

I'd try running without it and see what happens. I've seen things like this when the GPU drops out for whatever reason.

 

[awkward_eagle]

It _could_ be a bad GPU. Not necessarily all Radeon VIIs but maybe yours just nosedives every so often.

I'd try running without it and see what happens. I've seen things like this when the GPU drops out for whatever reason.

Looks like someone else with the same GPU is also getting panics.

Final Cut Pro with multi 3rd party GPU utilization

I have the same kernal panic randomly. It may be due to the GPU. I don't know.

forums.macrumors.com

Can't be a coincidence. I haven't opened mine yet so I'm returning for a pair of WX 9100s.

 

[Snow Tiger]

Looks like someone else with the same GPU is also getting panics.

Final Cut Pro with multi 3rd party GPU utilization

I have the same kernal panic randomly. It may be due to the GPU. I don't know.

forums.macrumors.com

Can't be a coincidence. I haven't opened mine yet so I'm returning for a pair of WX 9100s.

If you do rebuild a Radeon VII , please be aware it has a graphite TIM . The pieces of which are electrically conductive .

AMD WX 9100 = AMD Vega F.E. basically . The F.E.s are much , much cheaper .

 

[awkward_eagle]

I have a few Vega FEs in use at work, both in new Mac Pro and as eGPUs. They run really hot. So far they're stable. Only problem is you can't buy them outside of eBay. For production systems I always buy new from somewhere I can easily do a return if needed.

 

[Casey P]

I am the person with the same watchdog timeouts and two Radeon VII's. It could be one or both of the Radeon's. I have no idea and the panics are so infrequent that it would take forever to track down. I have also added a few other cards that could be causing the problems, including a usb audio processor that I already know causes regular panics, at least on the previous operating system.

My gut says that the Radeon VII was never really designed to be used with a Mac and probably has various issues. If there was something better and just as cheap, I would pick it up.

 

[Bradleyone]

I wouldn't be so quick to blame the Radeon VII entirely: I'm also getting occasional watchdogd panics, and this is on stock standard Apple supplied components with factory RAM and a Vega II.

It could still be graphics related, but possibly more centered around a Catalina bug occurring with non-typical graphics setups?

My use case: running a Mac Pro 7,1 overnight - headless, no display plugged in - periodically maintained with a VNC session. The virtual display is set to 1920x1080, which is a non-default resolution obtained by option-clicking the Scaled radio button in System Preferences.

The task involves two Parallels instances, each running fairly intensive Windows programs. Roughly two mornings out of seven I log in to find the Mac Pro has restarted from a kernel panic involving watchdogd, and the Windows instances are suspended (they can't be auto-resumed).

I haven't encountered a kernel panic on this Mac Pro when using it face to face since the machine arrived in December, so while I am still investigating causes, I suspect the headless operation and related graphics environment has something to do with it. It also seems to have started since Catalina 10.15.3 appeared.

 

[AlexMaximus]

I would be careful to blame the Vega VII card for panics. Those Vega VII work flawlessly for users here in the forum. So far, I have not found a single cMP 5.1 Pixlas user that reported any problems with the Vega VII. My own experience with extensive use has not produced a single instance or panic so far. Compared to many other cards that I had installed, the Vega VII has the best cooling and the best noise-to-performance ratio. I would absolutely not exchange that for the W9100 with that junk duct cooler fan.

 

[ibuick]

So, If you really concerned about what caused this problem. Just swap out all of your third party GPUs.

or, macOS kernel has some watchdog related nvram param then you can turnoff it and give it a try.

for example:

I think the. wdt_test=no_user_checkin or. wdt=-1 may be the right args to config.

 

[awkward_eagle]

I would be careful to blame the Vega VII card for panics. Those Vega VII work flawlessly for users here in the forum. So far, I have not found a single cMP 5.1 Pixlas user that reported any problems with the Vega VII. My own experience with extensive use has not produced a single instance or panic so far. Compared to many other cards that I had installed, the Vega VII has the best cooling and the best noise-to-performance ratio. I would absolutely not exchange that for the W9100 with that junk duct cooler fan.

Its comforting to know it might be general Catalina shoddiness. Think I'm still leaning toward the 9100s for a multi gpu setup.

DaVinci Resolve 15: AMD Radeon VII 16GB Performance

The Radeon VII is a very interesting card for DaVinci Resolve due to its 16GB of VRAM which is twice what you would get from a similarly priced NVIDIA GPU. But is it able to keep up in terms of raw performance?

www.pugetsystems.com

At the end :
"the cooler design is not very good for multi-GPU setups which are very common on high-end DaVinci Resolve workstations. You may be able to get two cards in a system that has enough space and airflow, but that is going to be a hard limit."

Heard enough problems about rendering with Redshift on cards with the VII blower style stacked in the same case since Redshift hits the cards harder for longer when compared to FCP or Resolve. For my workflow Im willing to sacrifice a little speed for reliability. It's why I got a Mac Pro instead of building something from scratch.

 

[Snow Tiger]

Its comforting to know it might be general Catalina shoddiness. Think I'm still leaning toward the 9100s for a multi gpu setup.

DaVinci Resolve 15: AMD Radeon VII 16GB Performance

The Radeon VII is a very interesting card for DaVinci Resolve due to its 16GB of VRAM which is twice what you would get from a similarly priced NVIDIA GPU. But is it able to keep up in terms of raw performance?

www.pugetsystems.com

At the end :
"the cooler design is not very good for multi-GPU setups which are very common on high-end DaVinci Resolve workstations. You may be able to get two cards in a system that has enough space and airflow, but that is going to be a hard limit."

Heard enough problems about rendering with Redshift on cards with the VII blower style stacked in the same case since Redshift hits the cards harder for longer when compared to FCP or Resolve. For my workflow Im willing to sacrifice a little speed for reliability. It's why I got a Mac Pro instead of building something from scratch.

It does not hurt to install blower type fan graphics cards . This is where the air gets sucked into the card and exhausted through the rear of the computer ( the card's PCIe bracket ) . This is what a workstation desires . Blower type coolers allow GPU arrays to remain cooler under load . This is all relative , of course . If you are having issues with blower type cards , the alternative , open air cards , will only make things worse .

Blower type cooling on a GPU . Cool air in , hot air out through the rear :

 

[awkward_eagle]

It does not hurt to install blower type fan graphics cards . This is where the air gets sucked into the card and exhausted through the rear of the computer ( the card's PCIe bracket ) . This is what a workstation desires .

Maybe I misunderstood the article. I assumed that of the three fans on the Radeon VII, at least one of them exhausted air back into the case in addition to exhausting out the back as opposed to a rear exhaust only card with a single fan.

 

[Snow Tiger]

Maybe I misunderstood the article. I assumed that of the three fans on the Radeon VII, at least one of them exhausted air back into the case in addition to exhausting out the back as opposed to a rear exhaust only card with a single fan.

The Radeon VII has an open air cooler . Hot air gets pushed right back into the inside of the computer . Not much gets exhausted out of the System .


Open air type cooling on a GPU . Cool air in , hot air out everywhere :

 

[awkward_eagle]

The Radeon VII has an open air cooler . Hot air gets pushed right back into the inside of the computer . Not much gets exhausted out of the System .


Open air type cooling on a GPU . Cool air in , hot air out everywhere :

View attachment 893783

Ah. That diagram makes way more sense. So it's still recycling the air back into the system for the front fans to push out the back, but doesn't push air back out through the on-card fans. For heavy rendering it looks like rear exhaust only cards are better when stacking if you don't want to hear your systems fans rev up.

 

[AidenShaw]

It does not hurt to install blower type fan graphics cards . This is where the air gets sucked into the card and exhausted through the rear of the computer ( the card's PCIe bracket ) . This is what a workstation desires .

My GPU server cards have no fans. The chassis fans pressurize the case, and the cool air escapes by flowing over the GPU feat sink fins and out the back. Note that the GPUs have no video ports, so there's lots of airflow.
[automerge]1581535558[/automerge]

View attachment 893783

Maybe it would be more accurate to change the blue arrows to purple. The intake air is mixed with the heated air.

 

[Snow Tiger]

My GPU server cards have no fans. The chassis fans pressurize the case, and the cool air escapes by flowing over the GPU feat sink fins and out the back. Note that the GPUs have no video ports, so there's lots of airflow.

Is a feat sink what happens when a tech falls into thermally conductive quicksand ?

Seriously , do your servers have loud , small , fast system fans ? This would not work well with a personal workstation in small studio environment . The multimedia editors are already sensitive as it is ! A real server would drive them into an early grave ...

Maybe it would be more accurate to change the blue arrows to purple. The intake air is mixed with the heated air.

In all fairness , I stole the pictures . I should be ashamed but I was too lazy .

 

[AidenShaw]

Is a feat sink what happens when a tech falls into thermally conductive quicksand ?

Seriously , do your servers have loud , small , fast system fans ? This would not work well with a personal workstation in small studio environment . The multimedia editors are already sensitive as it is ! A real server would drive them into an early grave ...

Ask Bare Feats ...

The "fanless GPUs, pressurized case" is exactly what the stock MP7,1 GPUs use. That approach to cooling doesn't require it to be loud. Rack servers are optimized for density, not quiet operation.

 

[Snow Tiger]

Ask Bare Feats ...

The "fanless GPUs, pressurized case" is exactly what the stock MP7,1 GPUs use. That approach to cooling doesn't require it to be loud. Rack servers are optimized for density, not quiet operation.

Gods , I'm designing an air filter for these rigs and I didn't even know about this .

And you really don't want to know who else should have known , too ... 😬 .

More embarrassment . But the job will get done much more precisely now .

Well , I guess I'm gonna have to whip out the old manometer , eh ?

Have you ever used one of these , Aiden ?

 

[AidenShaw]

Have you ever used one of these , Aiden ?

The last one I used had a mercury column....

And yes, my 2U (44.5 mm high) servers have a lot (6) of small fans. To use the fanless GPUs, you even need to order the noisier "high performance" fans.



Also note that the airflow is left to right:

• Disks first, under the metal bar on the left side (ambient temperature in my lab is 17°)
• Fans
• CPUs and 24 RAM slots
• Power supplies, cards, and motherboard

Noise isn't an issue, my desk is about 3.6km from the servers.

 

[Snow Tiger]

Disks first, under the metal bar on the left side (ambient temperature in my lab is 17° )

Ho , Ho ! I'll send you a cardigan for your next B-day and you can be the Mr . Rodgers of IT !!! 😂