Help! Frequent Mac Pro Kernel Panic...

[DesignJoe]

Hi,

I am receiving frequent Kernel Panic's on my late 2007 Mac Pro. This began happening sporatically when using Snow Leopard (10.6.x) and has continued after upgrading to Lion (10.7.4). They seem to be getting progressively worse and I was hoping that someone familar with Diagnostic Reports would be able to make some sense of the following...

Thu Jul 12 20:24:55 2012
Machine-check capabilities (cpu 1) 0x0000000000000006:
family: 6 model: 15 stepping: 6 microcode: 198
Intel(R) Xeon(R) CPU 5150 @ 2.66GHz
6 error-reporting banks
Machine-check status 0x0000000000000005:
restart IP valid
machine-check in progress
MCA error-reporting registers:
IA32_MC0_STATUS(0x401): 0x1000000000000000 invalid
IA32_MC1_STATUS(0x405): 0x0000000000000000 invalid
IA32_MC2_STATUS(0x409): 0x0000000000000000 invalid
IA32_MC3_STATUS(0x40d): 0x0020000000000000 invalid
IA32_MC4_STATUS(0x411): 0x0000000000000011 invalid
IA32_MC5_STATUS(0x415): 0xb200001802000e0f valid
MCA error code: 0x0e0f
Model specific error code: 0x0200
Other information: 0x00000018
Status bits:
Processor context corrupt
Error enabled
Uncorrected error
panic(cpu 1 caller 0x2cf3cc): "Machine Check at 0x020ddf68, trapno:0x12, err:0x0," "registers:\n" "CR0: 0x8001003b, CR2: 0x03291ff8, CR3: 0x00100000, CR4: 0x00000660\n" "EAX: 0x00000000, EBX: 0x0928c000, ECX: 0x00000001, EDX: 0x00000000\n" "ESP: 0x00fbbd10, EBP: 0x00fbbd48, ESI: 0x00000000, EDI: 0x00000108\n" "EFL: 0x00000046, EIP: 0x020ddf68\n"@/SourceCache/xnu/xnu-1699.26.8/osfmk/i386/trap_native.c:258
Backtrace (CPU 1), Frame : Return Address (4 potential args on stack)
0x67a0ef38 : 0x2203de (0x6afa8c 0x67a0ef58 0x229fb0 0x0)
0x67a0ef68 : 0x2cf3cc (0x6bd020 0x6bd154 0x20ddf68 0x12)
0x67a0f0d8 : 0x2cf409 (0x67a0f110 0xde 0x6bd154 0x1)
0x67a0f0f8 : 0x2e5e32 (0x67a0f110 0x0 0x0 0x0)
0xfbbd48 : 0x20d59df (0x679a6100 0x1 0x928c000 0x0)
0xfbbde8 : 0x20d7c73 (0xffffffff 0x7fffffff 0x0 0x0)
0xfbbe78 : 0x2d0cde (0xffffffff 0x7fffffff 0xfbbe98 0x2cd4ca)
0xfbbe98 : 0x22d3e5 (0x679be000 0x12a74ee9 0xa965 0x679bede0)
0xfbbed8 : 0x22e1c0 (0x92b6be8 0x679be000 0x0 0x0)
0xfbbf28 : 0x22f6c2 (0x92b6c04 0x0 0xfbbf48 0x2)
0xfbbf88 : 0x22f7f0 (0x6622d8 0x91ec7c0 0x0 0x2c72b6)
0xfbbfa8 : 0x662339 (0x6622d8 0x91ec7c0 0x91ec7dc 0x92b6be8)
0xfbbfc8 : 0x2c7d0c (0x91ec7c0 0x0 0x10 0x9dc5310)
Kernel Extensions in backtrace:
com.apple.driver.AppleIntelCPUPowerManagement(193.0)[5B5B4311-7A87-41B8-AC17-AE 08B259E210]@0x20d3000->0x20fbfff


BSD process name corresponding to current thread: kernel_task


Mac OS version:
11E53


Kernel version:
Darwin Kernel Version 11.4.0: Mon Apr 9 19:33:05 PDT 2012; root:xnu-1699.26.8~1/RELEASE_I386
Kernel UUID: DC2420E0-4451-376C-8A70-74301C98CE6B
System model name: MacPro1,1 (Mac-F4208DC8)


System uptime in nanoseconds: 186251570207831
last loaded kext at 175732993561526: com.apple.filesystems.smbfs 1.7.2 (addr 0x1c3d000, size 249856)
last unloaded kext at 4537682734432: com.apple.driver.AppleUSBCDC 4.1.17 (addr 0xf23000, size 16384)
loaded kexts:
com.vmware.kext.vmnet 3.1.4
com.vmware.kext.vmioplug 3.1.4
com.vmware.kext.vmci 3.1.4
com.vmware.kext.vmx86 3.1.4
com.manycamllc.driver.ManyCamDriver 0.0.9
com.apple.filesystems.smbfs 1.7.2
com.apple.filesystems.afpfs 9.8.1
com.apple.nke.asp_tcp 6.0.1
com.apple.driver.AppleUSBCDC 4.1.17
com.apple.driver.AppleUSBEthernetHost 2.3.0
com.apple.driver.AppleHWSensor 1.9.5d0
com.apple.filesystems.autofs 3.0
com.apple.driver.AppleBluetoothMultitouch 70.12
com.apple.driver.AppleHDA 2.2.0f3
com.apple.driver.AppleUpstreamUserClient 3.5.9
com.apple.driver.AppleMCCSControl 1.0.26
com.apple.driver.AudioAUUC 1.59
com.apple.driver.AppleUSBAudio 2.8.5f4
com.apple.iokit.IOUserEthernet 1.0.0d1
com.apple.iokit.IOBluetoothSerialManager 4.0.5f11
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AudioIPCDriver 1.2.2
com.apple.driver.AppleMCEDriver 1.1.9
com.apple.driver.ACPI_SMC_PlatformPlugin 5.0.0d0
com.apple.GeForce7xxx 7.0.0
com.apple.driver.AppleLPC 1.5.8
com.apple.driver.CSRUSBBluetoothHCIController 4.0.5f11
com.apple.iokit.SCSITaskUserClient 3.2.0
com.apple.driver.XsanFilter 404
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.BootCache 33
com.apple.iokit.IOAHCIBlockStorage 2.0.3
com.apple.driver.AppleFWOHCI 4.8.9
com.apple.driver.AppleAHCIPort 2.3.0
com.apple.driver.AirPortBrcm43224 501.36.15
com.apple.driver.AppleIntelPIIXATA 2.5.1
com.apple.driver.AppleUSBHub 4.5.0
com.apple.driver.AppleIntel8254XEthernet 2.1.3b1
com.apple.driver.AppleUSBEHCI 4.5.8
com.apple.driver.AppleEFINVRAM 1.5.0
com.apple.driver.AppleUSBUHCI 4.4.5
com.apple.driver.AppleACPIButtons 1.5
com.apple.driver.AppleRTC 1.5
com.apple.driver.AppleHPET 1.6
com.apple.driver.AppleSMBIOS 1.8
com.apple.driver.AppleACPIEC 1.5
com.apple.driver.AppleAPIC 1.5
com.apple.driver.AppleIntelCPUPowerManagementClient 193.0.0
com.apple.nke.applicationfirewall 3.2.30
com.apple.security.quarantine 1.3
com.apple.driver.AppleIntelCPUPowerManagement 193.0.0
com.apple.security.SecureRemotePassword 1.0
com.apple.iokit.IOSCSIBlockCommandsDevice 3.2.0
com.apple.iokit.IOUSBMassStorageClass 3.0.1
com.apple.kext.triggers 1.0
com.apple.driver.AppleMultitouchDriver 231.4
com.apple.driver.IOBluetoothHIDDriver 4.0.5f11
com.apple.driver.DspFuncLib 2.2.0f3
com.apple.iokit.IOFireWireIP 2.2.4
com.apple.iokit.IOSurface 80.0.2
com.apple.iokit.IOSerialFamily 10.0.5
com.apple.iokit.IOAudioFamily 1.8.6fc17
com.apple.kext.OSvKernDSPLib 1.3
com.apple.driver.AppleHDAController 2.2.0f3
com.apple.iokit.IOHDAFamily 2.2.0f3
com.apple.driver.ApplePolicyControl 3.0.16
com.apple.driver.AppleGraphicsControl 3.0.16
com.apple.driver.AppleSMC 3.1.3d8
com.apple.driver.IOPlatformPluginLegacy 5.0.0d0
com.apple.driver.IOPlatformPluginFamily 5.1.0d17
com.apple.nvidia.nv40hal.G7xxx 7.0.0
com.apple.NVDAResman.G7xxx 7.0.0
com.apple.iokit.IONDRVSupport 2.3.2
com.apple.iokit.IOGraphicsFamily 2.3.2
com.apple.driver.AppleUSBBluetoothHCIController 4.0.5f11
com.apple.iokit.IOBluetoothFamily 4.0.5f11
com.apple.driver.AppleUSBMergeNub 4.5.3
com.apple.iokit.IOUSBHIDDriver 4.4.5
com.apple.driver.AppleUSBComposite 4.5.8
com.apple.iokit.IOSCSIMultimediaCommandsDevice 3.2.0
com.apple.iokit.IOBDStorageFamily 1.6
com.apple.iokit.IODVDStorageFamily 1.7
com.apple.iokit.IOCDStorageFamily 1.7
com.apple.iokit.IOATAPIProtocolTransport 3.0.0
com.apple.iokit.IOSCSIArchitectureModelFamily 3.2.0
com.apple.iokit.IOFireWireFamily 4.4.5
com.apple.iokit.IOAHCIFamily 2.0.8
com.apple.iokit.IO80211Family 420.3
com.apple.iokit.IOATAFamily 2.5.1
com.apple.iokit.IONetworkingFamily 2.1
com.apple.iokit.IOUSBUserClient 4.5.8
com.apple.iokit.IOUSBFamily 4.5.8
com.apple.driver.AppleEFIRuntime 1.5.0
com.apple.iokit.IOHIDFamily 1.7.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.security.sandbox 177.5
com.apple.kext.AppleMatch 1.0.0d1
com.apple.security.TMSafetyNet 7
com.apple.driver.DiskImages 331.6
com.apple.iokit.IOStorageFamily 1.7.1
com.apple.driver.AppleKeyStore 28.18
com.apple.driver.AppleACPIPlatform 1.5
com.apple.iokit.IOPCIFamily 2.6.8
com.apple.iokit.IOACPIFamily 1.4​

Upon request, I can provide many others, if it will be helpful. I have already taken the following steps (and probably a few others) to no avail.

• Apple Hardware Test - No Errors (x2)
• Replace Hard Disk
• Clean Install of Snow Leopard (and later – upgrade to Lion)
• Disconnect all periphrials (except USB keyboard, Blutooth Magic Mouse and two DVI monitors)
• TechTool Deluxe
• Reseated RAM (Originally installed at the Apple Store)
• Fix Permissions
• Repair Disk

After each of these, the problem has continued, leaving me to believe that it must be a hardware issue. Unfortunately, these logs are (mostly) greek to me. The only error that I've been able to consistently see are that the same MCA registers are report invalid in each log.

I should also note that the Kernel Panics are very sporatic and I have been unable to identify and single application or task that I am performing when they occur. I have gone up to 10 days between Kernel Panics and occassionally they will occur two-three times in an 8-hour work day. For example, the following recent crashes were logged:

2012-07-12 - 20:24:57
2012-07-10 - 16:12:58
2012-07-10 - 11:31:21
2012-07-09 - 22:20:39
2012-07-09 - 12:32:25
2012-07-09 - 12:01:00
2012-07-08 - 18:13:32
2012-06-30 - 00:36:34
2012-06-29 - 14:21:57
2012-06-29 - 13:38:44
2012-06-21 - 13:44:59
2012-06-21 - 12:21:38
2012-06-21 - 11:00:19
2012-06-18 - 12:15:40​

Thanks, in advance for any suggestions or guidance!

Best,

~Joe

 

[mischief]

Seen this enough times....

MacPros are very power thirsty and require VERY good UPS equipment, particularly in offices or homes that have a lot of appliances or have AC units. The consistency of the error location (the Power manager) backs this up. Get a new APS UPS with power conditioning, true sine wave in the 800 to 1800 watt size. If the problem persists it may be your RAM.... but bad building power is MUCH more common and will never be reproduceable by an AppleCare provider.

 

[DesignJoe]

Get a new APS UPS with power conditioning, true sine wave in the 800 to 1800 watt size.

Thanks for the tip! I went through the rest of the logs and found "Kernel Extensions in backtrace: com.apple.driver.AppleIntelCPUPowerManagement" in some (but not quite half) of the other Kenel Panic logs available. Could power inconsistencies still be the issue?

Currently the Mac Pro is running on an APC BX1300LCD (http://bit.ly/M6dvZl) with brand new battery.

As luck would have it, I actually have a new-in-box Smart UPS C 1500VA LCD 120V in stock (http://bit.ly/M6d29m). Would this be adequate to meet the power needs & conditioning for the Mac Pro?

Thanks again for lending your expertise!

~Joe

----------

Also, just for giggles (very sad, nervous giggles ), I am attaching some of my most recent panic logs.

Thanks!

~Joe

 

[mischief]

Thanks for the tip! I went through the rest of the logs and found "Kernel Extensions in backtrace: com.apple.driver.AppleIntelCPUPowerManagement" in some (but not quite half) of the other Kenel Panic logs available. Could power inconsistencies still be the issue?

Currently the Mac Pro is running on an APC BX1300LCD (http://bit.ly/M6dvZl) with brand new battery.

As luck would have it, I actually have a new-in-box Smart UPS C 1500VA LCD 120V in stock (http://bit.ly/M6d29m). Would this be adequate to meet the power needs & conditioning for the Mac Pro?

Thanks again for lending your expertise!

~Joe

----------

Also, just for giggles (very sad, nervous giggles ), I am attaching some of my most recent panic logs.

Thanks!

~Joe

Interesting. I've seen the BX series work adequately but since you have the other one, yeah... it's a bit higher spec and has the true sine wave output. Give it a shot. At the very least you'll have removed the issue as a possibility. The next thing to think about is running Apple Hardware Test (should be on one of the original grey restore disks). That will go through and test everything per the basic suite. Sometimes there's even higher end versions of the memory and processor tests included where you can see them. I'd run those too if you have the time.

Have you installed any third party RAM or unseated the Processor tray from the logic board at any point?

----------

Looks like in that last KP you were having issues with the primary CPU making sense of SMB. Make sure all your software is up to date. As Apple has upgraded the OS more and more 32 bit and Universal Binary based software is getting buggy enough that if you're not up to date things like this happen. Get everything updated. (at least the point upgrades, do the research to see if you need the paid upgrades)

 

[DesignJoe]

Interesting. I've seen the BX series work adequately but since you have the other one, yeah... it's a bit higher spec and has the true sine wave output. Give it a shot. At the very least you'll have removed the issue as a possibility.

I'll give it a try over the weekend and see how it goes.

The next thing to think about is running Apple Hardware Test

I have run the 'Extended Apple Hardware Tests' a couple of times and it reports no problems.

Have you installed any third party RAM or unseated the Processor tray from the logic board at any point?

Nope. The RAM in the system (5GB) was installed at the Apple Store at the time of purchase. Besides installing a new hard disk, everything is factory.

Looks like in that last KP you were having issues with the primary CPU making sense of SMB. Make sure all your software is up to date.

OS X Lion is fully updated via Software Update. I also just used AppFresh to analyze my apps and I now have almost everything updated (with only a couple of exception on paid software that I only use occasionally.

Finally, a couple other bits of info that may be helpful...

1. Usually, when the KP occurs the system reboots and comes back up (restoring the applications that were open at the time of KP). It seems more likely to crash again shortly thereafter if I don't Quit all apps and do a clean reboot.
2. Also, for the first time ever yesterday a KP occurred and when the system started rebooting, it hung at a solid (blank) white screen (no apple logo or pinwheel as would be typical). I powered off and turned it back on to the same result. Finally, after several attempts it booted normally once again.

Thanks again... I've been pulling my hair out over this one for quite a while now and I was getting nowhere quick. I'll let you know how it goes!

Thanks!

~Joe

 

[DesignJoe]

Well, so much for the hope that the new UPS would resolve the issue. I've had two crashes since switching. I've attached the latest KP Log for review.

Thanks, in advance, for any additional ideas! I'm really at a loss!

~Joe

 

[DesignJoe]

And ANOTHER KP moments ago. I'm getting quite desperate to find the cause, since it is severely hampering productivity. Log attached.

Thanks!

~Joe

 

[DesignJoe]

Any thoughts? Anyone? Thanks, in advance!

~Joe

 

[brn44]

Designjoe,
Did you get your KP figured out?
I was having that problem some time ago. I can't remember the details but I found out it was caused by a third party software. Remove it & all was well.
The author of the software didn't follow the rules.