MP 6,1 Mac Pro 2013 Sequoia SN850X NVMe kernel panic during sleep

[MrMagicMadMax]

Hi,
I have a problem with my Mac Pro 6,1. First some background Info:
I have a total of three Mac Pros. One is a 5,1 (flashed) running Monterey with OpenCore which I had for 9 years now and wanted to replace so I created a Time Machine Backup of it. I also have two 6,1s. One is a D500 with its stock Quad-Core and upgraded 64GB of Samsung RAM and a stock 250GB Apple SSD. This one is running a fresh Install of Sequoia with OpenCore and stock settings except for the disabled bootpicker.
The third one is the problematic one. It's a D700 in which I installed a 12-Core Xeon (E5-2697 v2). It also has 64GB of upgraded RAM and a WD Black SN850X 1TB SSD.
What I first did was a new install of Sequoia, then reverted the root patches, restored the Time Machine backup and then installed the root patches again. When I put the machine asleep during the first night, I woke up to it with running fan, solid white power light and a yellow LED shining out the bottom side of it. It must be the CPU_PROCHOT or MEM_EVENT because of its location and color. It didn't respond to anything and had to be hard reset. When I started it again, it showed the kernel panic screen and continued to boot. The kernel panic report showed "3rd party NVMe controller. Fatal error. Delete IO submission queue.". After that it froze two times after just a few minutes of using it with a kernel panic reporting "3rd party NVMe controller. Loss of MMIO space. Write.". This (freeze during use) never happened again though. Testing some more revealed, that it wakes and dark wakes during sleep for different reasons and eventually get stuck not returning to sleep at which point it mostly shows the "Loss of MMIO space" error and very rarely the "Delete IO submission queue" error. Most of the times, the yellow LED shines through the bottom vents. One weird thing is, that this doesn't happen consistently after e set period time. Sometimes it happens after just an hour, sometimes it takes 10 hours. This makes troubleshooting very difficult.
I have iStat installed and sometimes when I manually woke the Mac, I could see in the graphs that it had been active for some periods of time when it was supposed to sleep as there was temperature data and even more interesting, almost all temps were higher after some breaks than before and this is also something I noticed. Sometimes it has a solid white light and I can feel it getting warm but the fan doesn't spin which explains the higher temperature and maybe also the yellow LED because that indicates overtemperature in RAM or CPU.
I also tried disabling "3rd party NVMe PM" in OpenCore to avoid loading NVMeFix.kext but that didn't change anything.
To narrow it down, I checked the other Mac Pro and found out that the wakes and dark wakes during sleep are normal. This one sleeps just fine with no issues. I used a cheap generic M.2 adapter from amazon until this point and thought it may be the problem and ordered the long Sintech one. Unfortunately it didn't change anything. I then noticed that the D700 Mac Pro had an older firmware from Big Sur which was known to be problematic with NVMe SSDs. Unfortunately, updating it to the latest 481 didn't fix it either.
I then swapped the SSDs of both machines and both slept fine for 6:45 hours when I stopped the test. I now know that it could have taken longer for the issues to arise.
I swapped the SSDs back and created a new partition on the SN850X and installed a fresh copy of Monterey. It slept fine for 6:20 hours when I stopped the test however it never woke or dark woke during that time so maybe it just didn't have any chances to fail.
I then installed a fresh copy of Sequoia with stock OpenCore settings on this partition to rule out my time machine backup. This time it slept fine for full 10 hours when it finally froze again and showed the yellow light. My next step would be to disable "3rd party NVMe PM" again but since I already tried that with my first install, I don't have high hopes.
I could also try swapping SSDs again and installing Monterey again to conduct longer tests.

Does anybody have any idea what is going on here?

 

[MrMagicMadMax]

I've done some more testing:
Disabled "3rd party NVMe PM" in OpenCore on the SN850X with the fresh Sequoia install and put it asleep in the D700 Mac Pro again and this time also unplugged everything apart from the power cord. It kernel panicked and showed the yellow light again after 3 hours.
I then swapped the SSDs again between both Mac Pros. So the D700 Mac Pro now runs the stock 250GB Apple SSD with the fresh Sequoia Install and stock OpenCore and the D500 Mac Pro runs the SN850X with the fresh Sequoia install and "3rd party NVMe PM" disabled. I unplugged everything apart from power from both and both are connected to the same WiFi.
Both have been sleeping completely fine for 17 hours now. I have no idea what could potentially be causing these issues. Why is the D500 working fine with the exact same SSD that causes kernel panics during sleep in the D700?? But at the same time, the D700 is fine with a stock SSD??
What I can eliminate for sure is the RAM as I swapped it and the issue didn't follow.
The only differences between these machines are:
- D700 vs. D500 GPUs
- 12-Core vs. 4-Core CPU
- The D700 has a Magic Mouse 2 connected via Bluetooth, while the D500 doesn't (Moving the mouse to the other machine will be my next test)

Also I didn't screw in the SSDs for this test. The Sintech adapter and SN850X aren't under any stress when screwed in though. The only thing I could imagine is that the SN850X isn't properly grounded anymore without the screw but I can't see why that would improve anything

 

[MrMagicMadMax]

I did another test with the SN850X inside the D500 Mac Pro (screwed in this time) and with the Magic Mouse 2 connected and again the OEM SSD in the D700 Mac Pro. Both machines slept fine for over 30 hours when I stopped the test.
So I conclude that the SN850X is 1) not damaged in any way and 2) compatible with the Mac Pro 6,1 in general.
But I still can't wrap my head around why the D700 Mac Pro has these issues but works fine with the Apple SSD.
One final thing I'm trying now is a deep NVRAM reset (hold ⌘+⌥+P+R until the 5th chime) but I doubt that it will change anything.

 

[MrMagicMadMax]

Interesting! Something actually changed after the deep NVRAM reset.
I had it sleeping again without anything attached and it slept fine for 5 hours when it suddenly rebooted and gave me an MCA Error:

I put it asleep again and it slept fine for over 13 hours.
In the pmset logs I found some more evidence about what happened:

"0x30000100000001F: EFI/bootrom Failure after last point of entry to sleep" suggests that has indeed something to do with the Bootrom. I previously had a problematic Bootrom from Big Sur (as this Mac was never updated beyond Big Sur) and updated straight to the most recent 481 Bootrom by doing a fresh install of Monterey on the stock Apple SSD.
Googling this error suggests that it can also be caused by NVMe drives being a little to slow to wake up sometimes but I then I don't get why this same SSD works fine in the other Mac Pro.
Is there any way to reflash the Bootrom or something like that?

 

[MrMagicMadMax]

I swapped the D700 and D500 GPUs between the two machines today and installed a fresh copy of Sonoma as that runs much better on the 6,1.
The D700 with the SN850X again showed some weird behavior already. I noticed that it had woken up from sleep (power button no longer pulsing) and they always did that from time to time but after about 15 minutes it still didn't go back to sleep so I tried to wake it and it worked but the sensors revealed some weird stuff.
The Mac dark woke at 13:14 "due to RTC/Maintenance but never went back to sleep. The last log line is at 13:16:

All the messages between 13:14 and 13:31 look normal. At 13:36 I woke the machine and as you can see, it says "DarkWake to FullWake" so it was stuck in DarkWake. Maybe it would have gone back to sleep as there were messages from 13:31 but i doubt it and it still doesn't look normal at all.
What's also interesting is that at first after the Dark Wake it did stuff and drew quiet a lot of power heating up everything inside the Mac to a point where the fan kicked in (which it usually doesn't during Dark Wakes). The CPU heated up to 79°C and both GPUs to 73°C before the fan finally started:

But what caused the heat buildup? Apparently the CPU and both GPUs were very active.
Here you can see the CPU (now the Quad-Core) going straight to over 30W and staying there for around 7-8 minutes and the dropping to around 17W until I woke it up.

Even more interesting are the GPUs. Usually GPU 1 is not used at all unless some application is specifically configures to use it for computing and GPU 2 is used for everything else like the displays. Both GPUs together drew a whopping 85-110W continuously during that first 7-8 minute period and then dropped to around 30-33W which seems to be normal during a dark wake as can be seen at around 12:56 where a more normal, short Dark Wake occured:

During the first 7-8 minute period, both GPU's Core voltage went up to 1,1V and the current to 16,5A. After that time it went down to 0,83V and 1,4A so compared to that it was 1460% higher.
Unfortunately no logs show what caused this high CPU and GPU usage. Does anybody know what could be causing this or how I could find it out?

Edit: so apparently com.apple.mediaanalysisd.filesystem.video was causing the high CPU and GPU usage. Weird because there's nothing on this fresh install. Or might it be processing the other volume on the disk with all my data on it?