MP 1,1-5,1 5,1 Mac Pro randomly shuts down

[vilmycil]

Hello everyone,

As per thread title, my work mac started miss-behaving. Out of the blue, few weeks ago it randomly restarted. I paid no attention. Continued working. Last week happened again. Rebooted and worked. Made me sort of anxious so I started backing up more frequently. Earlier this week it shut down and got stuck in a loop. Reseting PRAM sort of helped. It didn't boot right away, after reset I had to switch of the machine and start again then it loaded. Today as I am working screens go green with vertical lines. I quickly save work and hit shut down. While I save I noticed application spat out an error smth about CPU but screen goes grey with vertical lines and computer shuts down. Same story again. No boot. Stuck in a loop. Reset pram. Hard Shut Down. And here I am posting this post. Below is crash report. Anyone can assist with reading this? I don't have a replacement machine, and cant' afford much downtime.

Running 10.14.3 (18D109)
Mac Pro (Mid 2012)
2 x 3.46 GHz 6-Core Intel Xeon
64 GB 1333 MHz DDR3
Startup disk is m2 on pcie
AMD Radeon HD 7970 3072 MB

Humongous thank you in advance to everyone who's got anything to say.

Regards,
Tomas


Anonymous UUID: 811F414D-9823-4563-E9EC-B05DF9ACD257

Fri Feb 14 15:43:25 2020

*** Panic Report ***
Panic(CPU 0, time 5510515830393): NMIPI for unresponsive processor: TLB flush timeout, TLB state:0x0
RAX: 0x0000000000000050, RBX: 0x0000000000010000, RCX: 0xffffff8011b1aae0, RDX: 0xffffff80a027211c
RSP: 0xffffffa76dd43e00, RBP: 0xffffffa76dd43e80, RSI: 0xffffffa775f602dc, RDI: 0x0000000000000000
R8: 0x00000000ffffffff, R9: 0x0000000000000020, R10: 0x0000000000020000, R11: 0x0000000000000000
R12: 0x00000000000000ff, R13: 0x0000000000070000, R14: 0x0000000000000019, R15: 0xffffff8011c90ec0
RFL: 0x0000000000000002, RIP: 0xffffff80114a83b3, CS: 0x0000000000000008, SS: 0x0000000000000010
Backtrace (CPU 0), Frame : Return Address
0xffffff8715626f70 : 0xffffff80114d980f
0xffffff8715626fd0 : 0xffffff801135be4d
0xffffffa76dd43e80 : 0xffffff80114a8f96
0xffffffa76dd43ec0 : 0xffffff80113eb085
0xffffffa76dd43f40 : 0xffffff80113eac35
0xffffffa76dd43fa0 : 0xffffff801135b0ce

BSD process name corresponding to current thread: kernel_task

Mac OS version:
18D109

Kernel version:
Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64
Kernel UUID: 1970B070-E53F-3178-83F3-1B95FA340695
Kernel slide: 0x0000000011000000
Kernel text base: 0xffffff8011200000
__HIB text base: 0xffffff8011100000
System model name: MacPro5,1 (Mac-F221BEC8)

System uptime in nanoseconds: 5510515859807
last loaded kext at 5503375475369: com.apple.driver.AppleXsanScheme 3 (addr 0xffffff7f951c1000, size 32768)
last unloaded kext at 504944732518: com.apple.driver.AppleUSBMergeNub 900.4.2 (addr 0xffffff7f93509000, size 12288)
loaded kexts:
jp.co.yamaha.driver.YamahaSteinbergUSBAudio 1.9.93
com.aja.iokit.ntv2 15.1.0
com.apple.filesystems.msdosfs 1.10
com.apple.fileutil 18.306.12
com.apple.filesystems.autofs 3.0
com.apple.driver.AGPM 110.23.46
com.apple.driver.AppleUpstreamUserClient 3.6.5
com.apple.driver.AppleMCCSControl 1.5.6
com.apple.kext.AMDFramebuffer 2.0.4
com.apple.kext.AMDRadeonX4000 2.0.4
com.apple.driver.AppleGraphicsDevicePolicy 3.28.4
com.apple.driver.AudioAUUC 1.70
com.apple.AGDCPluginDisplayMetrics 3.28.4
com.apple.driver.AppleMikeyHIDDriver 131
com.apple.driver.AppleHV 1
com.apple.driver.AppleHDA 282.10
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.AppleMikeyDriver 282.10
com.apple.iokit.IOBluetoothSerialManager 6.0.10f1
com.apple.driver.pmtelemetry 1
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleLPC 3.1
com.apple.kext.AMD7000Controller 2.0.4
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.filesystems.apfs 945.241.4
com.apple.iokit.SCSITaskUserClient 408.200.1
com.apple.driver.AppleFWOHCI 5.6.0
com.apple.driver.AppleVirtIO 2.1.3
com.apple.filesystems.hfs.kext 407.200.4
com.apple.driver.Intel82574LEthernet 2.7.2
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.driver.AirPort.Brcm4331 800.21.30
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.driver.AppleAHCIPort 329.200.2
com.apple.AppleSystemPolicy 1.0
com.apple.private.KextAudit 1.0
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleACPIButtons 6.1
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleACPIEC 6.1
com.apple.driver.AppleAPIC 1.7
com.apple.driver.AppleIntelCPUPowerManagementClient 220.0.0
com.apple.nke.applicationfirewall 190
com.apple.security.TMSafetyNet 8
com.apple.driver.AppleIntelCPUPowerManagement 220.0.0
com.apple.driver.AppleXsanScheme 3
com.apple.kext.triggers 1.0
com.apple.kext.AMDRadeonX4030HWLibs 1.0
com.apple.iokit.IOAcceleratorFamily2 404.2.2
com.apple.kext.AMDRadeonX4000HWServices 2.0.4
com.apple.driver.AppleGraphicsControl 3.28.4
com.apple.iokit.IOAVBFamily 710.1
com.apple.plugin.IOgPTPPlugin 700.7
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.driver.DspFuncLib 282.10
com.apple.kext.OSvKernDSPLib 527
com.apple.driver.AppleSSE 1.0
com.apple.iokit.IOSurface 255.1
com.apple.driver.usb.cdc.acm 5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.iokit.IONDRVSupport 530
com.apple.driver.AppleSMBusController 1.0.18d1
com.apple.AppleGPUWrangler 3.28.4
com.apple.iokit.IOVideoFamily 1.2.1
com.apple.iokit.IOStreamFamily 1.1.0
com.apple.driver.AppleHDAController 282.10
com.apple.iokit.IOHDAFamily 282.10
com.apple.driver.AppleSMBusPCI 1.0.14d1
com.apple.kext.AMDSupport 2.0.4
com.apple.AppleGraphicsDeviceControl 3.28.4
com.apple.iokit.IOGraphicsFamily 530.14
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOFireWireIP 2.3.0
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d8
com.apple.driver.CoreStorage 546.50.1
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 6.0.10f1
com.apple.iokit.IOBluetoothHostControllerUSBTransport 6.0.10f1
com.apple.iokit.IOBluetoothHostControllerTransport 6.0.10f1
com.apple.iokit.IOBluetoothFamily 6.0.10f1
com.apple.iokit.IOAHCIBlockStorage 301.200.2
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.iokit.IOAudioFamily 206.5
com.apple.vecLib.kext 1.2.0
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.iokit.IOSCSIMultimediaCommandsDevice 408.200.1
com.apple.iokit.IOBDStorageFamily 1.8
com.apple.iokit.IODVDStorageFamily 1.8
com.apple.iokit.IOCDStorageFamily 1.8
com.apple.iokit.IOFireWireFamily 4.7.3
com.apple.iokit.IOSerialFamily 11
com.apple.iokit.IOAHCISerialATAPI 267.50.1
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.iokit.IO80211Family 1200.12.2
com.apple.driver.corecapture 1.0.4
com.apple.filesystems.hfs.encodings.kext 1
com.apple.iokit.IOAHCIFamily 288
com.apple.driver.usb.AppleUSBEHCIPCI 1.2
com.apple.driver.usb.AppleUSBUHCIPCI 1.2
com.apple.driver.usb.AppleUSBUHCI 1.2
com.apple.driver.usb.AppleUSBEHCI 1.2
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOUSBFamily 900.4.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.AppleEFINVRAM 2.1
com.apple.driver.AppleEFIRuntime 2.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.security.quarantine 3
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.DiskImages 493.0.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleUSBTDM 456.230.1
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.kext.CoreTrust 1
com.apple.iokit.IOUSBMassStorageDriver 145.200.2
com.apple.iokit.IOSCSIBlockCommandsDevice 408.200.1
com.apple.iokit.IOSCSIArchitectureModelFamily 408.200.1
com.apple.iokit.IOStorageFamily 2.1
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleBusPowerController 1.0
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.IOSlaveProcessor 1
com.apple.iokit.IOReportFamily 47
com.apple.iokit.IOTimeSyncFamily 700.7
com.apple.iokit.IONetworkingFamily 3.4
com.apple.driver.AppleACPIPlatform 6.1
com.apple.driver.AppleSMC 3.1.9
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.pthread 1
com.apple.kec.corecrypto 1.0
com.apple.kec.Libm 1
panic(cpu 2 caller 0xffffff80114b4d35): "TLB invalidation IPI timeout, unresponsive CPU bitmap: 0x1, NMIPI acks: 0x0, now: 0x1, deadline: 5509262043132"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4903.241.1/osfmk/x86_64/pmap.c:2837
Backtrace (CPU 2), Frame : Return Address
0xffffffa74f22b320 : 0xffffff80113aeb0d
0xffffffa74f22b370 : 0xffffff80114e8653
0xffffffa74f22b3b0 : 0xffffff80114da07a
0xffffffa74f22b420 : 0xffffff801135bca0
0xffffffa74f22b440 : 0xffffff80113ae527
0xffffffa74f22b560 : 0xffffff80113ae373
0xffffffa74f22b5d0 : 0xffffff80114b4d35
0xffffffa74f22b670 : 0xffffff80114bbc24
0xffffffa74f22b750 : 0xffffff80114bcba7
0xffffffa74f22b7b0 : 0xffffff8011443d6c
0xffffffa74f22b900 : 0xffffff801144b452
0xffffffa74f22b940 : 0xffffff801143f1b3
0xffffffa74f22b970 : 0xffffff80113bb441
0xffffffa74f22b9c0 : 0xffffff8011902dc7
0xffffffa74f22ba20 : 0xffffff801142488d
0xffffffa74f22ba40 : 0xffffff801145bfa4
0xffffffa74f22ba70 : 0xffffff80115f75a1
0xffffffa74f22baf0 : 0xffffff80115f6d96
0xffffffa74f22bb30 : 0xffffff80115fb5c0
0xffffffa74f22bbb0 : 0xffffff8011606d15
0xffffffa74f22bc50 : 0xffffff80115fca93
0xffffffa74f22bc80 : 0xffffff80115f6349
0xffffffa74f22bcf0 : 0xffffff80115fc8ab
0xffffffa74f22bd30 : 0xffffff80118ce0d6
0xffffffa74f22bdf0 : 0xffffff80118e6c7c
0xffffffa74f22bf40 : 0xffffff80119b62bb
0xffffffa74f22bfa0 : 0xffffff801135c466

BSD process name corresponding to current thread: launchd

EOF
Model: MacPro5,1, BootROM 140.0.0.0.0, 12 processors, 6-Core Intel Xeon, 3.46 GHz, 64 GB, SMC 1.39f11
Graphics: AMD Radeon HD 7970, AMD Radeon HD 7970, PCIe
Memory Module: DIMM 1, 8 GB, DDR3 ECC, 1333 MHz, 0x80AD, 0x484D54333147523743465234412D48392020
Memory Module: DIMM 2, 8 GB, DDR3 ECC, 1333 MHz, 0x80AD, 0x484D54333147523743465234412D48392020
Memory Module: DIMM 3, 8 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342314B37304348302D5948392020
Memory Module: DIMM 4, 8 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342314B37304348302D5948392020
Memory Module: DIMM 5, 8 GB, DDR3 ECC, 1333 MHz, 0x80AD, 0x484D54333147523743465234412D48392020
Memory Module: DIMM 6, 8 GB, DDR3 ECC, 1333 MHz, 0x80AD, 0x484D54333147523743465234412D48392020
Memory Module: DIMM 7, 8 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342314B37304348302D5948392020
Memory Module: DIMM 8, 8 GB, DDR3 ECC, 1333 MHz, 0x80CE, 0x4D33393342314B37304348302D5948392020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x8E), Broadcom BCM43xx 1.0 (5.106.98.102.30)
Bluetooth: Version 6.0.10f1, 3 services, 27 devices, 1 incoming serial ports
Network Service: Ethernet 1, Ethernet, en0
Network Service: Ethernet 2, Ethernet, en1
PCI Card: AJA NTV2, Video, Slot-2
PCI Card: AMD Radeon HD 7970, Display Controller, Slot-1
PCI Card: AMD Radeon HD 7970, ATY,HamachiParent, Slot-1
PCI Card: pci1b21,612, AHCI Controller, Slot-4
PCI Card: PXS3, USB eXtensible Host Controller, Slot-3
Serial ATA Device: HL-DT-ST DVD-RW GH61N
Serial ATA Device: Samsung SSD 860 EVO M.2 1TB, 1 TB
Serial ATA Device: Samsung SSD 860 EVO 500GB, 500.11 GB
Serial ATA Device: WDC WD40EFRX-68WT0N0, 4 TB
Serial ATA Device: ST10000DM0004-1ZC101, 10 TB
USB Device: USB 2.0 Bus
USB Device: USB2.1 Hub
USB Device: KOMPLETE KONTROL M32
USB Device: Medusa
USB Device: DataTraveler 2.0
USB Device: iLok
USB Device: USB 2.0 Bus
USB Device: Keyboard Hub
USB Device: Apple Keyboard
USB Device: Hub
USB Device: USB Keyboard
USB Device: IC RECORDER
USB Device: FaderPort
USB Device: Steinberg UR28M
USB Device: USB Bus
USB Device: BRCM2046 Hub
USB Device: Bluetooth USB Host Controller
USB Device: USB Bus
USB Device: USB Bus
USB Device: Expert Wireless TB
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB Bus
USB Device: USB 3.0 Bus
USB Device: Flash Card Reader/Writer
FireWire Device: built-in_hub, Up to 800 Mb/sec
Thunderbolt Bus:

 

[th0masp]

Hmm, these reports never do much for me to clear things up.

Perhaps try running this in terminal:

log show --predicate 'eventMessage contains "Previous shutdown cause"' --last 1000h

That should give you a numeric code for every system restart in the last 40ish days. Will take a while to compile the output. Then check out this site to see what the numbers mean: https://georgegarside.com/blog/macos/shutdown-causes/
[automerge]1581697545[/automerge]
I should point out that if you happen to shut down your computer at the end of every day then its going to be quite the log and filled with regular restart entries as well as signs of trouble.

At any rate the '5's should be safe to ignore.

 

[vilmycil]

Thanks! Here's a result. Error -62 seems to be matching the days I experienced issues. I do have chrome open most of the times while working. For the time being I shall have it turned off and next week I will inspect extensions. Oddly, I did not install or change anything on the system.. which makes me thing it is hardware related issue. Any ideas? Personal experiences anyone? After all it does not reboot until I reset PRAM.

For reference here's wording from the shut down causes site:

-61	Watchdog timer detected unresponsive application, shutting down the system.	Troubleshoot for stuck application, startup item and any loading preferences. Boot to Safe Mode by holding ⇧shift on boot which disables most startup items and daemons. If the problem does not occur, try a new user account.
-62	Watchdog timer detected unresponsive application, restarting the system.	The system will automatically restart after this shutdown cause, unlike -61. Troubleshooting applies the same for cause -61. -61 is used where the system believes it cannot automatically recover, whereas -62 is used where the system determines it is likely to be resolved after restart. Hardware acceleration has been experimentally deemed a potential cause, based on kernel extensions being shown in logs. Memory management may be a potential cause, such as Google Chrome’s excessive use of RAM, which can be mitigated with various extensions that suspend tabs when not in use. While it is a bug with macOS that Google Chrome should be able to cause a -62, such mitigations might be necessary.

 

[startergo]

Try leaving only two ram modules in the machine one each CPU.

 

[th0masp]

According to what you wrote my money is on the GPU. I'd swap it first thing to see if it solves the problem.

 

[vilmycil]

I've got Sapphire Radeon RX 580 NITRO+ 8 GB GDDR5 that I can pull from a pc at home. Will have it with me on Monday and see. Will keep ram in mind too. Thanks guys.

 

[Macschrauber]

What are the Northbridge temperatures core / heatsink ?

Mac Pro Heatsink, northbridge rivet, loud fan

Machine Mac Pro 2009 - Quad core. Symptoms: Red light on processor board, no boot, no fan in CPU heat sink. Also can see red-light on CPU-A heat on backboard. The other case fans run excessively and loud. Diagnosis: Failed Heat sink fan. Also failed was the rivets on the Northbridge heat...

forums.macrumors.com

 

[vilmycil]

What are the Northbridge temperatures core / heatsink ?

Mac Pro Heatsink, northbridge rivet, loud fan

Machine Mac Pro 2009 - Quad core. Symptoms: Red light on processor board, no boot, no fan in CPU heat sink. Also can see red-light on CPU-A heat on backboard. The other case fans run excessively and loud. Diagnosis: Failed Heat sink fan. Also failed was the rivets on the Northbridge heat...

forums.macrumors.com

Last crash was under same load as this screenshot taken just now.

 

[Macschrauber]

Ok, no abnormal temperature differences.

 

[vilmycil]

Quick update. System failed again today, and nothing would bring it back. Switched graphics cards, and it started as usual. The annoying part is it's Radeon RX 580 8192 MB, not only it's a complete overkill for my line of work, it also is so huge I can't have all pcie slots loaded. Any suggestions how to overcome this would be very useful. In the meantime I am waiting for MSI RADEON RX 560 AERO ITX card which will remain in this mac permanently.

 

[flowrider]

I've had my machine since 2/06 - Have not experienced a panic or a shutdown. Running with seven internal SSDs. 580X replaced with an RX5700XT. 96GB (twelve 8GB modules).

Lou

 

[tommy chen]

I've had my machine since 2/06 - Have not experienced a panic or a shutdown. Running with seven internal SSDs. 580X replaced with an RX5700XT. 96GB (twelve 8GB modules).

Lou

this here is about the 5,1 not the 7,1

 

[flowrider]

^^^^

Lou

 

[vilmycil]

Anyone interested new video card has resolved the issue. System has been running solid since February, large loads of work etc.

Here's a related question. Every end of day once the system is shut down, I use mains switches to turn off all of the remaining equipment including the extension that feeds the mac. The odd thing is that every morning when I come in and turn on the power that feeds the mac, the computer starts right away, I don't have to press a button etc. It's been happening for a long time, ignored it since it's almost convenient in a way, but now thinking perhaps I should look into this.

So, I had a look ta the shut down cause log again and I can see cause case 0, power disconnected. Is that ok? Is the boot up related to this? Do I need a new battery? In the life of owning computers never needed to replace one so don't know. Any suggestions? It's not a macbook, we're talking 5,1 cheesegrater.

 

[KeesMacPro]

Here's a related question. Every end of day once the system is shut down, I use mains switches to turn off all of the remaining equipment including the extension that feeds the mac. The odd thing is that every morning when I come in and turn on the power that feeds the mac, the computer starts right away, I don't have to press a button etc. It's been happening for a long time, ignored it since it's almost convenient in a way, but now thinking perhaps I should look into this.

So, I had a look ta the shut down cause log again and I can see cause case 0, power disconnected. Is that ok? Is the boot up related to this? Do I need a new battery? In the life of owning computers never needed to replace one so don't know. Any suggestions? It's not a macbook, we're talking 5,1 cheesegrater.

About your questions:
-Power on: check the settings in preferences (the lightbulb): deactivate "automatic power on after power failure"

-Battery: It's recommended to replace an ~10 years old battery. It's a BR2032 . AFAIK Panasonic is the only available ,some people replace it by a CR2032 , but personally I'd not recommend it, the battery is very close to the GPU and the CR2032 is not resistant to higher Temps.

 

[vilmycil]

About your questions:
-Power on: check the settings in preferences (the lightbulb): deactivate "automatic power on after power failure"

-Battery: It's recommended to replace an ~10 years old battery. It's a BR2032 . AFAIK Panasonic is the only available ,some people replace it by a CR2032 , but personally I'd not recommend it, the battery is very close to the GPU and the CR2032 is not resistant to higher Temps.

Thanks!

Automatic startup after power failure appears to be off.

Will get a new battery just in case.