Become a MacRumors Supporter for $25/year with no ads, private forums, and more!
Status
The first post of this thread is a WikiPost and can be edited by anyone with the appropiate permissions. Your edits will be public.
Status
Not open for further replies.

VitaminK

macrumors member
Original poster
Dec 28, 2019
56
25
macOS Big Sur 11.3 Beta 3 has broken Mac Pro's early-2009 to mid-2012 ability to boot reliably into macOS.

It's still possible to boot 11.3/11.4, but the process requires multiple restarts and a connection has been made between PCIe devices, especially storage related devices and USB controller cards, causing panics at boot. While most people report NVMe crashes, the issue is not NVMe related and PCIe SATA cards or AHCI PCIe M.2 devices, including Apple 12+16 AHCI PCIe models, also have the same crashes. It's probably an unknown race condition somewhere early in the kernel that is causing the crashes.

Some Mac Pros are less affected with the 11.3.1/11.4/11.5bx crashes and seems that having fewer PCIe devices installed improve the rate of successful boots. Other non-storage related PCIe devices also seem to contribute to the crashes like the older AirPort Extreme models - at this moment the best success rate of an AirPort Extreme model seems to be with the BCM943602CDPAX AirPort Extreme + bluetooth 4.1 combo card, but this card have unrelated issues when you boot Mojave from BigSur.

Please check if your NVRAM volume garbage collection is working before trying to test 11.3.1/11.4/11.5bx, if you have a Mac Pro with the VSS stores full, you could brick it with the constant crashes at boot - several people already have, be warned.


The crashes still continue to happen with the current 11.4 (20F71) release version and with the developers beta release, 11.5b2 (20G5033c).

To the users that do not have issues while booting 11.3, test at least 10 reboots before reporting successful then please list any PCIe cards, PCIe storage and the model of the AirPort Extreme you may have installed.


For detailed panic logs see here:


Note: MacPro4,1 and MacPro5,1 are not the only machines affected. These symptoms can be replicated less frequently by @khronokernel on MacPro3,1, iMac8,1, iMac11,2, MacBook7,1 and MacBookPro8,3 as well as by ASentientBot on a Penryn Hackintosh. Therefore this race condition is not limited by the CPU architecture and instead by the CPU's speed and number of PCI devices present for device setup

Big Sur 11.2.3 direct download link:​


Big Sur 11.2.3 direct download from Apple servers: http://swcdn.apple.com/content/down...i7fezrmvu4vuab80m0e8a5ll/InstallAssistant.pkg
 
Last edited:

Syncretic

Contributor
Apr 22, 2019
296
1,257
Can you (or someone) post a copy of the panic report? It's always helpful to understand the details of what's happening, even if the problem appears to be insoluble.
 
  • Like
Reactions: cdf

VitaminK

macrumors member
Original poster
Dec 28, 2019
56
25
Maybe a tad, but I wouldn't want anyone walking into the issues ill had today. The issue isn't limited to NVME drivers, more a combination of PCI cards causing freezing during boot noticeable in verbose mode, the issue has been all over the place for me.
 
Last edited:

goMac

Contributor
Apr 15, 2004
7,406
1,347
Technically macOS 11 has never supported older Mac Pros. If you’re running it on a 5,1 or older things are probably going to break...

5,1s are no longer in their test set. I wouldn’t count on them fixing anything. They have no reason to fix it unless it’s also affecting 7,1s.
 

startergo

macrumors 68040
Sep 20, 2018
3,875
1,753
Technically macOS 11 has never supported older Mac Pros. If you’re running it on a 5,1 or older things are probably going to break...

5,1s are no longer in their test set. I wouldn’t count on them fixing anything. They have no reason to fix it unless it’s also affecting 7,1s.
Up to 11.2 the cMP was perfectly running BS on a NVME PCIE drive. The only thing needed was -no_compat_check boot argument. I can confirm the issue with the NVME drivers. I will try to update the NVME driver inside the firmware from the MP6,1 firmware.
 

startergo

macrumors 68040
Sep 20, 2018
3,875
1,753

cdf

macrumors 68000
Jul 27, 2012
1,916
1,956
Indeed. Although I haven't investigated the issue, it does seem rather widespread.
 

cdf

macrumors 68000
Jul 27, 2012
1,916
1,956
Weird that APFSJumpStart EFI module inside the MP5,1 EFI firmware is not loading the correct/updated APFS EFI module from the Big Sur EFI partition…
Is that what's going on here? Perhaps enabling EnableJumpstart in OC can get the correct module to load...
 

Macschrauber

macrumors 68000
Dec 27, 2015
1,810
795
Germany
11.3 Beta even crashes on a AHCI Apple Blade, btw

Also I noticed that using a Kepler GPU without UGA Bootscreen Rom does not crash when the Apple AHCI Blade was in.

So that's a second data point for PCIe Bus Problems.

A Buddy reported that using a USB3 PCIe Card also crashed 11.3 Beta 4 and 5.

So maybe it's not NVMe but a problem with talking to the PCI Bus at all?
 
  • Like
Reactions: VitaminK

VitaminK

macrumors member
Original poster
Dec 28, 2019
56
25
11.3 beta 5 will boot from an NVME, ill had more success with booting after an initial restart from a working Mac OS or let the system kernel panic restart decrease the chance of a halt from assert failed,
 
  • Like
Reactions: TECK

startergo

macrumors 68040
Sep 20, 2018
3,875
1,753
Code:
panic(cpu 20 caller 0xffffff800d9bee36): Kernel trap at 0xffffff801000f109, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000000, CR3: 0x0000000011c35000, CR4: 0x00000000000226e0
RAX: 0x0000000000000100, RBX: 0x0000000000000000, RCX: 0x0000000000000000, RDX: 0x0000000000000009
RSP: 0xffffffa212c1bd80, RBP: 0xffffffa212c1bdb0, RSI: 0x0000000905243fd3, RDI: 0xffffff94206c4800
R8:  0xffffff9420796640, R9:  0xffffff8754905580, R10: 0x0000000000000001, R11: 0xffffff942092d280
R12: 0xffffff942092d280, R13: 0xffffff801000dfee, R14: 0x0000000905243fd3, R15: 0xffffff94206c4800
RFL: 0x0000000000010246, RIP: 0xffffff801000f109, CS:  0x0000000000000008, SS:  0x0000000000000000
Fault CR2: 0x0000000000000000, Error code: 0x0000000000000000, Fault CPU: 0x14, PL: 0, VF: 1

Backtrace (CPU 20), Frame : Return Address
0xffffffa212c1b7a0 : 0xffffff800d887f3d mach_kernel : _handle_debugger_trap + 0x3fd
0xffffffa212c1b7f0 : 0xffffff800d9ce493 mach_kernel : _kdp_i386_trap + 0x143
0xffffffa212c1b830 : 0xffffff800d9bea6a mach_kernel : _kernel_trap + 0x55a
0xffffffa212c1b880 : 0xffffff800d82ca2f mach_kernel : _return_from_trap + 0xff
0xffffffa212c1b8a0 : 0xffffff800d88775d mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffa212c1b9c0 : 0xffffff800d887a53 mach_kernel : _panic_trap_to_debugger + 0x273
0xffffffa212c1ba30 : 0xffffff800e095c9a mach_kernel : _panic + 0x54
0xffffffa212c1baa0 : 0xffffff800d9bee36 mach_kernel : _sync_iss_to_iks + 0x2c6
0xffffffa212c1bc20 : 0xffffff800d9beb1d mach_kernel : _kernel_trap + 0x60d
0xffffffa212c1bc70 : 0xffffff800d82ca2f mach_kernel : _return_from_trap + 0xff
0xffffffa212c1bc90 : 0xffffff801000f109 com.apple.iokit.IONVMeFamily : __ZN16IONVMeController18GetOveragedRequestEy + 0x27
0xffffffa212c1bdb0 : 0xffffff801000e04f com.apple.iokit.IONVMeFamily : __ZN16IONVMeController15CheckForTimeoutEP18IOTimerEventSource + 0x61
0xffffffa212c1bde0 : 0xffffff800dfeddb5 mach_kernel : __ZN18IOTimerEventSource15timeoutSignaledEPvS0_ + 0xa5
0xffffffa212c1be50 : 0xffffff800dfedcb6 mach_kernel : __ZN18IOTimerEventSource17timeoutAndReleaseEPvS0_ + 0xc6
0xffffffa212c1be80 : 0xffffff800d8ce315 mach_kernel : _thread_call_delayed_timer + 0x4a5
0xffffffa212c1bef0 : 0xffffff800d8cf284 mach_kernel : _thread_call_delayed_timer + 0x1414
0xffffffa212c1bfa0 : 0xffffff800d82c13e mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         com.apple.iokit.IONVMeFamily(2.1)[204D1783-899F-353F-84CB-D7F6862AE265]@0xffffff8010001000->0xffffff801002afff
            dependency: com.apple.driver.AppleEFINVRAM(2.1)[055F51C0-3FE9-31C1-8CC8-2ADC456E3C28]@0xffffff800ecf1000->0xffffff800ecfafff
            dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[08C786C4-D093-3AE9-A378-7A716759967F]@0xffffff800eeab000->0xffffff800eec0fff
            dependency: com.apple.iokit.IOPCIFamily(2.9)[368F5229-F604-3F4F-B877-05DAB5072243]@0xffffff80102c0000->0xffffff80102e8fff
            dependency: com.apple.iokit.IOReportFamily(47)[21571942-1055-35FE-935F-544ADB4A43A5]@0xffffff80102f7000->0xffffff80102f9fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[063C627D-49F0-356D-9B37-EE3D587BECC2]@0xffffff80103c3000->0xffffff80103d4fff

Process name corresponding to current thread: kernel_task
Boot args: -v -no_compat_check keepsyms=1 debug=0x144 no32exec=0

Mac OS version:
Not yet set

Kernel version:
Darwin Kernel Version 20.4.0: Tue Feb 23 23:43:36 PST 2021; root:xnu-7195.100.354.131.1~3/RELEASE_X86_64
Kernel UUID: 638E017A-1AC0-30AB-BB38-C4C34BEF8D3A
KernelCache slide: 0x000000000d600000
KernelCache base:  0xffffff800d800000
Kernel slide:      0x000000000d610000
Kernel text base:  0xffffff800d810000
__HIB  text base: 0xffffff800d700000
System model name: MacPro5,1 (Mac-7BA5B2D9E42DDD94)
System shutdown begun: NO
Panic diags file unavailable, panic occurred prior to initialization
Hibernation exit count: 0

System uptime in nanoseconds: 39862752695
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000000094ade257a
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x0000003216a3d58c 0x0000000000000000
Above is the panic log and AppleNvme assert failed just before the panic:
1616711558085.png

I also tried updating the latest NVME and APFS jump start drivers from MP61.scap
 

MisterAndrew

macrumors 68030
Sep 15, 2015
2,530
1,979
Portland, Ore.
It seems weird to me to complain about a compatibility issue with a non-supported system. Big Sur isn’t suppose to work on 5,1 and older. You need a 6,1 or newer for Big Sur.
 
  • Like
Reactions: G5isAlive

startergo

macrumors 68040
Sep 20, 2018
3,875
1,753
Big Sur isn’t suppose to work on 5,1 and older.
The accurate statement is :
is not designed with respect to cMP and older. As it was demonstrated many times before the only thing, which was needed to run Big Sur was _no_compat_check boot argument. Big Sur does not support USB1 and some older wifi cards. Other than that everything worked fine.
 

MisterAndrew

macrumors 68030
Sep 15, 2015
2,530
1,979
Portland, Ore.
The accurate statement is :
is not designed with respect to cMP and older. As it was demonstrated many times before the only thing, which was needed to run Big Sur was _no_compat_check boot argument. Big Sur does not support USB1 and some older wifi cards. Other than that everything worked fine.
Maybe it would run fine, but I'm sure there are more complicated compatibility issues than that.
 

VitaminK

macrumors member
Original poster
Dec 28, 2019
56
25
Isn’t a maybe.. If a computer has a compatible CPU typically Mac OS will boot older unsupported Mac’s especially.

As our Mac’s have native hardware Mac OS is looking like SMC, an EC , and correct DSDT & firmware. Very rarely Apple has fully dropped an unsupported Mac small parts stay for awhile like audio layouts, USB mapping , Ethernet , power management , Sata , Cd/DVD burning. Apple so far has dropped WiFi/Bluetooth, original Non metal GPUs from Mac Pro 5,1..
 
Last edited:

startergo

macrumors 68040
Sep 20, 2018
3,875
1,753
To get back BS working I had to create a new BS 11.2.3 partition, then from within BS install it to the unbootable partition. From the bootpicker with bootable USB installer it said it cannot "downgrade" BS. Hopefully that will be fixed in the latest major BS release.
Also Safari for later beta cannot run on earlier released version (if reinstalled), so I had to manually copy the contents of the Safari package from the freshly installed Big Sur partition to the old Safari package and
/Library/Apple/System/Library/CoreServices/SafariSupport.bundle
 
Last edited:

startergo

macrumors 68040
Sep 20, 2018
3,875
1,753
What is interesting is that the Assert failed is not the reason for the kernel panic. Here is the Bootlog_Kernel of Catalina:
Code:
2021-03-27 07:15:10.67 kernel[0]: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2021-03-27 07:15:10.68 kernel[0]: (IONVMeFamily) ReleaseIDNode
2021-03-27 07:15:10.68 kernel[0]: (IONVMeFamily) file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-470.100.17/IONVMeController.cpp
2021-03-27 07:15:10.69 kernel[0]: (IONVMeFamily) line: 5478
2021-03-27 07:15:10.69 kernel[0]: (IONVMeFamily)
2021-03-27 07:15:10.69 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:10.69 kernel[0]: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2021-03-27 07:15:10.69 kernel[0]: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2021-03-27 07:15:10.69 kernel[0]: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2021-03-27 07:15:10.69 kernel[0]: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) ReleaseIDNode
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) ReleaseIDNode
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-470.100.17/IONVMeController.cpp
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-470.100.17/IONVMeController.cpp
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) line: 5478
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) ReleaseIDNode
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) ReleaseIDNode
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-470.100.17/IONVMeController.cpp
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-470.100.17/IONVMeController.cpp
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) line: 5478
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) line: 5478
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily)
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily)
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily)
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:10.70 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:10.71 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:10.71 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:10.71 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:10.71 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:10.98 kernel[0]: (IONVMeFamily) line: 5478
2021-03-27 07:15:11.05 kernel[0]: (IONVMeFamily)
2021-03-27 07:15:11.05 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:11.06 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64
2021-03-27 07:15:11.06 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
2021-03-27 07:15:11.06 kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64
 

VitaminK

macrumors member
Original poster
Dec 28, 2019
56
25
I don’t believe the issue has todo with an NVME any longer but PCI cards them self become an issue. Apple has focused heavily in 11.3 improving 7,1 Mac Pros these changes could be the issue during boot but after Beta 5 our Pros still have full compatibility.

for reference I use a Sonnet NVME, Titan ridge, Sonnet Asmedia 1142, and Vega FE
 
Status
Not open for further replies.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.