MP 1,1-5,1 MacPro 5,1 stuck booting after Catalina -> Monterey upgrade

[hbanko]

Hi Guys,

I was running Catalina on my 2010 dual CPU MacPro 5,1 without any issues. As gfx card I use an RX570 and booting off a 1TB NVMe.

Unfortunately Fusion360 refuses to work with Catalina now and I was forced to update to a later OS. From my quick research it seemsed that Monterey "was the sweet spot" and would also resolve my issue. So here's what I did:

* Download the OC app and install (update) OC again to the same NVMe drive
* Use the app to download Monterey
* Creat the thumb drive as installation media with the app

The I attempted to install it out of Catalina which looks like it completed. But the loading bar after first boot stops at 75% and is not moving further. Cold reboot and I can see the bar moving quickly to over 50$ and then the screen blanks and the bar is then at 10%.

I tried Verbose boot mode and I can see plenty of message passing over the screen and then there is a flash and I see again this 10% bar and thats it. My feeling is, that its actually completing the boot.

Made another installation attempt and booted off the thumbdrive this time. Exactly the same behaviour.

I got an Mojave installation on a different drive in my Mac and tried to boot it by picking throigh the OC bootloader menu. But after completing the boot the loading bar disappears and there is suddenly no more signal displayed on the screen. It goes into powersafe and turns off.

A PRAM reset did not change anything for either OS.

Any suggestion from here what to try next?

 

[Macschrauber]

Test your Mojave installation without OpenCore. Remove all drives with OC and Monterey for this test.

 

[tsialex]

You wrote about OC, but from what you wrote seems you are running OCLP, a very different beast. OC does not have an app.

You are starting with dosdude1 Catalina and trying to upgrade it to Monterey? This usually won't work, at least not with several hardware configs. Start with vanilla Mojave, create the Monterey USB installer, re-boot, select the OCLP Monterey via the BootPicker, then install to an empty disk. Btw, Catalina is not supported by OCLP.

You are changing GPUs while installing? You can't change GPUs with OCLP without removing the root patches.

 

[hbanko]

I used Open Core before to install Catalina. But I can't remeber if this app existed at that time. The installation I did for Catalina came with the boot menu - a feature from my understanding you would only get with OpenCore. Before Catalina on different systems I used dosdude1 that I think just replaced boot.bin (not Open Core?)

The GFX card is the same as it was in Catalina - no hardware change. There is a ton of software installed on this Mac and is the main system I use for my small business and hobby. Having to install everything from scratch would be an absolute nightmare. But backups do exist!

Right now I am giving it another go but enabling this time the NVMe boot option.

 

[tsialex]

You are mixing OC with OCLP, this is asking for trouble. Even more so with an ancient OC instance booting a recent OCLP installer.

You best chance to make this work is creating a Mojave rescue disk, remove OC from the ESP of the Catalina disk, add -no_compat_check to the boot args, then use the OCLP Monterey installer.

If you RX 570 is not flashed, you gonna need EnableGop injected to the GPU or the BootROM.

 

[hbanko]

Ok, I am confused.

So we got Open Core -> The big leap and enabler for more recent OSX versions on old Macs

Then therte is Open Core Legacy Patcher -> Which got nothing to do with Open Core and is a totally different thing?

If I search for Open Core I pretty much always end up with Open Core Legacy Patcher and Open Core is on the same GitHub?

I don't recall that I followed these instructions as below:

Getting started with OpenCore | OpenCore Install Guide

dortania.github.io

Thats like a bigger project to make an installer.

 

[hbanko]

I installed a updated OCLP on my boot drive and then installed from the USB drive. Any components left of the old install other than the OS I am not aware of?

 

[tsialex]

OC and OCLP are not the same thing. OpenCore is a boot loader, OpenCore Legacy Patcher (OCLP) while utilizing OC as a bootloader is a way to make unsupported Macs to run current macOS releases, this involves macOS root patching and lot's of other tweaks that OpenCore does not provide and are not developed by the same people.

OpenCore project on GitHub

OCLP project on GitHub

 

[tsialex]

I installed a updated OCLP on my boot drive and then installed from the USB drive. Any components left of the old install other than the OS I am not aware of?

OC/OCLP install loads of things inside the EFI System Partition (ESP) of your disk.

If you did root patching, there even more inside the macOS.

 

[tsialex]

Whatever method, OC or OCLP, this stands as the way that you can succeed:

You best chance to make this work is creating a Mojave rescue disk, remove OC from the ESP of the Catalina disk, add -no_compat_check to the boot args, then use the OCLP Monterey installer.

If your RX 570 is not flashed, you gonna need EnableGop injected to the GPU or the BootROM.

 

[hbanko]

ok, thnaks for your help. Before starting from sratch I'll give it another go tomorrow with EnableGop and the NVMe boot option enabled.

Anything I can check / change out of the recovery via the console?

 

[hbanko]

Followed the advice and started from a blank disk (NVMe). I used an USB NVMe adapter on a different Mac and select host mode. Picked MacPro 5,1 and set options as mention here. Installed OC to the virgin NVMe.

After that I installed the NVMe in the MacPro 5,1 and installed Monterey from the USB thumdrive (made with OCLP) app.

The one extra and unnecessary PCI card was a USB 3.0 that I have removed as well.

Setup:
MacPro 5,1 (dual cpu)
NVMe SD card with a single 256GB NVMe installed
RX570 GFX Card
Nothing Else

Result after the installation:
MacPro 5,1 gets stuck the same way as before. In verbose I can see it chucking out a gazilion of messages and seemingly completing the boot. The verbose screen disappears and I see the Apple Logo with the Bar. The Bar is maybe a 5%. Thats where the other attempt go stuck with my Catalain that I upgraded.

There is one difference now. The Mac reboot after 30 seconds beeing stuck on the Apple Logo + Bar. In other words - reboot loop instead of frozen.

According to the success reports here, this should be straight forward and not really a challenge. What am I doing wrong?

 

[tsialex]

Why you didn't install Mojave to your Mac Pro from a createinstallmedia USB installer, made sure that the Mac Pro is working correctly, then after that you tried Monterey?

 

[hbanko]

Well I have been using it for 2 years in this configuration with Catalina. Nothing changed other than the OS. There is no doubt that it works.

But might be close to have it resolved. The installation on the first blank NVMe was stuck for 15 minutes and after rebooting itself 3 times the installation continued and completed.

So I wiped in confidence my actual system NVMe drive and the install did not get stuck or have been rebooting unsually. It went straight throguh. Currently restoring my Timemachine backup and hopefully tomorrow morning I am back in business.

But wonder if I would have succeded without touching the bootloader/EFI and would have just updated to Monterey like I am on a natively supported Mac.

 

[tsialex]

Well I have been using it for 2 years in this configuration with Catalina. Nothing changed other than the OS. There is no doubt that it works.

Sorry to get you a reality check, but you could pretty much trashed the Mac Pro BootROM NVRAM volume with the several unsuccessful installs. People in the same situation bricked the Mac Pro, every week we have a post or a thread about it.