Become a MacRumors Supporter for $25/year with no ads, private forums, and more!
Status
The first post of this thread is a WikiPost and can be edited by anyone with the appropiate permissions. Your edits will be public.

amstel78

macrumors 6502
Aug 12, 2018
447
161
Make sure to use the correct command:nvram 4D1FDA02-38C7-4A6A-9CC6-4BCCA8B30102:opencore-version
Oops! Thanks, CDF. I mistyped the earlier string. All is good now.
1613318260260.png
 
Last edited:

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
The same thing happened to me recently. I could no longer read the opencore-version nvram variable, even if ExposeSensitiveData was properly set.

When I booted natively into Mojave to diagnose the issue, I realized that something was wrong with the NVRAM. Perhaps it was just a coincidence, but I could no longer bless other volumes. I did a 4-time NVRAM reset, which finally allowed me to boot into recovery. I disabled SIP, rebooted and promptly restored a clean backup of my BootROM. Afterwards, everything was fine.

We've been warned many times by @tsialex about the eventual fate of the SPI flash memory. If you haven't done so already, back up your BootROM! I also have a replacement chip ready to go.

View attachment 1729956
There is a extremely simple way (simple here as in not need to know how the NVRAM works/check free space indicators/validate checksums and etc) to check if the garbage collection failed and the need to reflash with the clean dump:

  • Dump the BootROM with ROMTool
  • Open the dump with UEFITool NE 0.58
  • Go to EFISystemNvDataFvGUID, open it
    VSS_44781 - EFISystemNvDataFvGUID.png

    VSS_44781 - EFISystemNvDataFvGUID.png
  • Go to the first VSS store, open it
    VSS_44781 - EFISystemNvDataFvGUID - VSS.png
  • Click Free space, it's after the last variable/VSS entry:
    VSS_44781 - EFISystemNvDataFvGUID - VSS - Free Space.png
  • Check on the right panel the Full size:
    VSS_44781 - EFISystemNvDataFvGUID - VSS - Free Space - Full size.png
A clean reconstructed never booted image have the Free Space Full Size as 65448 - this is for a fully empty store:
VSS_reconstructed.EMPTY.png


A normal working single CPU Mac Pro with 3 DIMMs have the Free Space Full Size usually around 45000 to 40000 - this is for a healthy working dump:
VSS_44781 - EFISystemNvDataFvGUID - VSS - Free Space - Full size.png


A normal working dual CPU Mac Pro with 8 DIMMs have the Free space Full size around 35000 to 30000 - this is for a healthy working dump:
VSS_34808.png


A Mac Pro that the garbage collection is not working anymore will have less than 1/3 of Free Space Full Size available, less than 22000 bytes available. Any less than this and you usually start having problems.

This one has just 8921 and already corrupted the NVRAM volume:
VSS_8921.png


Where is the second VSS store?
VSS_8921_No 2nd VSS.png


I personally don't wait for the garbage collection to fail, I have a recurrent appointment on my calendar to flash the cleaned BootROM image every 3 months. Since starting doing it, I never had a brick or any NVRAM problems - even with all my crazy tests that bricked so much times my backplanes in the past.
 
Last edited:

amstel78

macrumors 6502
Aug 12, 2018
447
161
@tsialex thanks for the post above. For someone not well versed in this, how does one go about by cleaning the BootROM image? I was thinking of just cleaning NVRAM 3, perhaps 4 times then dumping that and keeping the saved binary file for backup.
 

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
@tsialex thanks for the post above. For someone not well versed in this, how does one go about by cleaning the BootROM image? I was thinking of just cleaning NVRAM 3, perhaps 4 times then dumping that and keeping the saved binary file for backup.
If the NVRAM garbage collection already failed, cleaning the NVRAM will do nothing - only the secondary VSS store will be cleaned. If yours is functional yet, wait for the 5th chime and dump it.

For dumps where the garbage collection is not working and you don't have a backup dump, a reconstruction will be needed.
 
  • Like
Reactions: JeDiGM and amstel78

amstel78

macrumors 6502
Aug 12, 2018
447
161
If the NVRAM garbage collection already failed, cleaning the NVRAM will do nothing - only the secondary VSS store will be cleaned. If yours is functional yet, wait for the 5th chime and dump it.

For dumps where the garbage collection is not working and you don't have a backup dump, a reconstruction will be needed.
This is what my dump shows. Looks like I'm bound to start having some troubles down the road?
1613320878912.png


So do I just hold down Command-Opt-P-R until I hear the 5th chime, boot into non-OC MacOS, dump the BootROM and compare? If I get closer to 30000-35000, I should be good?
 

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
This is what my dump shows. Looks like I'm bound to start having some troubles down the road?
View attachment 1730050
You probably already are overusing the last NAND sectors where the first VSS store is stored inside the SPI flash memory. This will cause SPI failure.
So do I just hold down Command-Opt-P-R until I hear the 5th chime, boot into non-OC MacOS, dump the BootROM and compare? If I get closer to 30000-35000, I should be good?
Always dump or flash from a vanilla macOS install - dumps, and the System Information reports, from OC installs are not reliable.

Garbage collection needs space to work. If the circular log lost it's control of the head and tails, usually happens when there are less than a third of available space, you will overuse the NAND cells and have problems with the volume itself.

MP4,1/MP5,1 was designed for another era of NVRAM usage, back in 2008 the NVRAM use was very sparse, today is heavily used for all sort of things.
 

Enricote

macrumors member
Oct 7, 2018
99
60
Madrid, Spain
There is a extremely simple way (simple here as in not need to now how the NVRAM works/check free space indicators/validate checksums and etc) to check if the garbage collection failed and the need to reflash with the clean dump:

A clean reconstructed never booted image have the Free Space Full Size as 65448 - this is for a fully empty store:
View attachment 1730012

A normal working single CPU Mac Pro with 3 DIMMs have the Free Space Full Size usually have around 45000 to 40000 free space - this is for a healthy working dump:
View attachment 1730011

A normal working dual CPU Mac Pro with 8 DIMMs have the Free space Full size around 35000 to 30000 free space - this is for a healthy working dump:
View attachment 1730024

A Mac Pro that the garbage collection is not working anymore will have less than 1/3 of Free Space Full Size available, less than 22000 bytes available. Any less than this and you usually start having problems.

This one has just 8921 and already corrupted the NVRAM volume:
View attachment 1730020

Where is the second VSS store?
View attachment 1730021

I personally don't wait for the garbage collection to fail, I flash the cleaned BootROM image every 3 months. Since starting doing it, I never had a brick or any NVRAM problems - even with all my crazy tests that bricked so much times my backplanes in the past.
Thank you @tsialex as always! Your knowledge overwhelms me!
I have fixed my problem just reflashing BootROM using the reconstructed one that you made it for me.
Now everything is ok again
 

Attachments

  • Screenshot 2021-02-14 at 17.46.37.png
    Screenshot 2021-02-14 at 17.46.37.png
    275.8 KB · Views: 74
Last edited:
  • Like
Reactions: zoltm and tsialex

amstel78

macrumors 6502
Aug 12, 2018
447
161
You probably already are overusing the last NAND sectors where the first VSS store is stored inside the SPI flash memory. This will cause SPI failure.

Always dump or flash from a vanilla macOS install - dumps, and the System Information reports, from OC installs are not reliable.

Garbage collection needs space to work. If the circular log lost it's control of the head and tails, usually happens when there are less than a third of available space, you will overuse the NAND cells and have problems with the volume itself.

MP4,1/MP5,1 was designed for another era of NVRAM usage, back in 2008 the NVRAM use was very sparse, today is heavily used for all sort of things.
Zapped NVRAM (waited for the 5th chime) and then booted into recovery. Disabled SIP then restarted into vanilla Mojave. Dumped the BootROM and this is the result:
1613322734409.png


I take it the above is a good sign?

Will be interesting to see how this changes one I re-bless OpenCore. That said, I thought setting WriteFlash to <false/> prevented OC from writing to NVRAM, instead storing variables into RAM?

I will tuck this ROM dump away for safe keeping. OT: What should the free space size be for a clean MP3,1 with 2 processors and 8 DIMMs?
 

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
Zapped NVRAM (waited for the 5th chime) and then booted into recovery. Disabled SIP then restarted into vanilla Mojave. Dumped the BootROM and this is the result:
View attachment 1730062

I take it the above is a good sign?

Will be interesting to see how this changes one I re-bless OpenCore. That said, I thought setting WriteFlash to <false/> prevented OC from writing to NVRAM, instead storing variables into RAM?

I will tuck this ROM dump away for safe keeping. OT: What should the free space size be for a clean MP3,1 with 2 processors and 8 DIMMs?
It’s good enough, but you don’t have much headroom.

OC eve with protection enabled still need some variables inside the NVRAM.

MP3,1 have a separate flash memory for the NVRAM, it’s a very different BootROM architecture. You can’t clean it, btw.
Differently from a MP4,1/5,1, a failed NVRAM SPI with a MP3,1 still boots, but you can’t bless for example.
 

amstel78

macrumors 6502
Aug 12, 2018
447
161
It’s good enough, but you don’t have much headroom.
MP3,1 have a separate flash memory for the NVRAM, it’s a very different BootROM architecture. You can’t clean it, btw.
Differently from a MP4,1/5,1, a failed NVRAM SPI with a MP3,1 still boots, but you can’t bless for example.
Thanks again for sharing your knowledge with me. Much appreciated.

Last couple of questions. How can I confirm garbage collection is working correctly? And since it appears zapping NVRAM in my 5,1 cleans things up to an acceptable level, would you suggest I do this every several months or so? I'd like to keep the SPI flash working as long as possible.
 

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
Thanks again for sharing your knowledge with me. Much appreciated.

Last couple of questions. How can I confirm garbage collection is working correctly? And since it appears zapping NVRAM in my 5,1 cleans things up to an acceptable level, would you suggest I do this every several months or so? I'd like to keep the SPI flash working as long as possible.
It’s too complex to really know if a NVRAM volume is healthy, you have to know a lot about Apple flavor of the EFI implementation to evaluate it.

Track the space available, it’s the simplest way to see if it’s still doing it’s job.

Like I wrote, I flash my never booted image every 3 months, I have a recurrent appointment in my calendar.
 

amstel78

macrumors 6502
Aug 12, 2018
447
161
It’s too complex too really know if a NVRAM volume is healthy, you have to know a lot about Apple flavor of the EFI implementation to evaluate it.

Track the space available, it’s the simplest way to see if it’s still doing it’s job.

Like I wrote, I flash my never booted image every 3 months, I have a recurrent appointment in my calendar.
Understood. Unfortunately I don't have a "never-booted" ROM backup, so will have to flash based on what I have today - or zap until the 5th chime is heard (just to clear space).
 

Macschrauber

macrumors 68000
Dec 27, 2015
1,622
661
Germany
Sometimes, when starting with a clean nvram the 4 times nvram reset cleans an exploding stream. Dont ask me why, it happens.

started with a clean nvram, something went nuts with a bootloader, 1st VSS was almost full after only a few boots. In that case the deep nvram reset helped. Of course I have the binwalks and dumps but this is clearly visible.

maybe it was cleaned because of the previous perfect empty stream. I dont know :)

3E31FEC9-CD88-446D-AE7E-0E9DA5243913.png
 

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
Sometimes, when starting with a clean nvram the 4 times nvram reset cleans an exploding stream. Dont ask me why, it happens.

started with a clean nvram, something went nuts with a bootloader, 1st VSS was almost full after only a few boots. In that case the deep nvram reset helped. Of course I have the binwalks and dumps but this is clearly visible.

maybe it was cleaned because of the previous perfect empty stream. I dont know :)

View attachment 1730147
When the garbage collection is still working, the circular log is backed up to the second store and the first one is reset.

If the circular log lost the control of the head/tail, just the second store is cleaned and the 1st stays as is, that's why some dumps have a crazy 1st VSS store and a pristine 2nd VSS recently cleaned.
 

Macschrauber

macrumors 68000
Dec 27, 2015
1,622
661
Germany
When the garbage collection is still working, the circular log is backed up to the second store and the first one is reset.

If the circular log lost the control of the head/tail, just the second store is cleaned and the 1st stays as is, that's why some dumps have a crazy 1st VSS store and a pristine 2nd VSS recently cleaned.

yes, it only works with prestine streams. I tried it with messed streams and with luck the deep nvram reset helped to free up a little but not even close to the results when starting with a clean stream.

So second it that everyone should have at least a rom backup or even better a cleaned up firmware to refresh.
 
  • Like
Reactions: tsialex

Enricote

macrumors member
Oct 7, 2018
99
60
Madrid, Spain
There is a extremely simple way (simple here as in not need to know how the NVRAM works/check free space indicators/validate checksums and etc) to check if the garbage collection failed and the need to reflash with the clean dump:

A clean reconstructed never booted image have the Free Space Full Size as 65448 - this is for a fully empty store:
View attachment 1730012

A normal working single CPU Mac Pro with 3 DIMMs have the Free Space Full Size usually have around 45000 to 40000 free space - this is for a healthy working dump:
View attachment 1730011

A normal working dual CPU Mac Pro with 8 DIMMs have the Free space Full size around 35000 to 30000 free space - this is for a healthy working dump:
View attachment 1730024

A Mac Pro that the garbage collection is not working anymore will have less than 1/3 of Free Space Full Size available, less than 22000 bytes available. Any less than this and you usually start having problems.

This one has just 8921 and already corrupted the NVRAM volume:
View attachment 1730020

Where is the second VSS store?
View attachment 1730021

I personally don't wait for the garbage collection to fail, I flash the cleaned BootROM image every 3 months. Since starting doing it, I never had a brick or any NVRAM problems - even with all my crazy tests that bricked so much times my backplanes in the past.
This is the Free Space of my dumped BootROM before I reflashed today because of my issue: 14376. This copy was a clean/reconstructed one exactly on November 16th ( 3 months ago ). I am not a very busy computer engineer... I am just a user... I have really done many changes in Opencore in the lasts few weeks.... that is true....
 

Attachments

  • Screenshot-2021-02-14-at-20.50.43.jpg
    Screenshot-2021-02-14-at-20.50.43.jpg
    392 KB · Views: 72
Last edited:
  • Like
Reactions: tsialex

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
This is the Free Space of my dumped SROM before I reflashed today because of my issue: FreeSpace 14376. This copy was a clean/reconstructed one exactly on November 16th ( 3 months ago ). I am not a very busy computer engineer... I am just a user... I have really done many changes in Opencore in the lasts few weeks.... that is true....
Yes, when you are testing with OpenCore configs, trying to find the golden config for your Mac Pro, it's better to flash the backup/cleaned dump more frequently. After you get it stabilised, no need to be as aggressive with the schedule.

One thing that is clear, no one should start with OC tests without knowing the status of the NVRAM free space. Like I always say, the MP5,1 NVRAM is tiny for todays standards and it's deadly for someone with a already compromised NVRAM, even reseting the NVRAM 4-times continuously before anything.

While the MP4,1/5,1 project stood the test of time, the NVRAM is the Achilles heel of it.
 
  • Like
Reactions: Enricote

amstel78

macrumors 6502
Aug 12, 2018
447
161
@tsialex just asking again in case you didn't see it in my other post, but what effect does the "WriteFlash" variable in OC do as far as NVRAM is concerned? I was under the impression that having that set to "false" would mitigate the number of writes OC does to NVRAM, instead loading to system RAM?
 

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
@tsialex just asking again in case you didn't see it in my other post, but what effect does the "WriteFlash" variable in OC do as far as NVRAM is concerned? I was under the impression that having that set to "false" would mitigate the number of writes OC does to NVRAM, instead loading to system RAM?
I've saw it and answered already on post #7010.

Yes, when WriteFlash=false OC writes a lot less, but several variables are stored inside the NVRAM by OC for it to work.

It's easy to prove, flash the generic upgrade image MP51.fd from 10.14.6 MAS full installer (where the NVRAM is just FFs), then enable OC, use your Mac Pro, then do some reboots, use some more and finally dump it. Open the dump with HexFiend and look between 0x120000 and 0x12FFFF, it's where the 1st store of the NVRAM is.

Code:
Install\ macOS\ Mojave/Install\ macOS\ Mojave.app/Contents/Resources/Firmware/MP51.fd

Edit:

Never do this without a backup dump of your BootROM previously saved.

The generic image don't have any serials (there are 7 different hardwareIDs that make your Mac Pro unique, none is present inside the MP51.fd).
 
Last edited:
  • Like
Reactions: amstel78

amstel78

macrumors 6502
Aug 12, 2018
447
161
I've saw it and answered already on post #7010.

Yes, when WriteFlash=false OC writes a lot less, but several variables are stored inside the NVRAM by OC for it to work.

It's easy to prove, flash the generic upgrade image MP51.fd from 10.14.6 MAS full installer (where the NVRAM is just FFs), then enable OC, use your Mac Pro, then do some reboots, use some more and finally dump it. Open the dump with HexFiend and look between 0x120000 and 0x12FFFF, it's where the 1st store of the NVRAM is.

Code:
Install\ macOS\ Mojave/Install\ macOS\ Mojave.app/Contents/Resources/Firmware/MP51.fd
Got it. Thanks for the clarification.
 

eVasilis

macrumors 6502
Jan 13, 2010
373
172
For dumps where the garbage collection is not working and you don't have a backup dump, a reconstruction will be needed.
Hi there!

How does one go about reconstructing the bootrom? Will re-applying the 144.0.0.0 one do the trick?

Thanks!
 

tsialex

macrumors G3
Jun 13, 2016
9,735
10,458
Hi there!

How does one go about reconstructing the bootrom?

Thanks!
PM sent.

Will re-applying the 144.0.0.0 one do the trick?

Thanks!
No, see why:

  • First you can't re-flash the same firmware version or downgrade it. efiflasher checks the EFI version before doing anything.
  • Second, the generic image doesn't even have a NVRAM volume, just FFs where the NVRAM is stored. Apple efiflasher completely ignores the NVRAM area of the BootROM image and keep it as is inside the SPI flash memory.

Btw, not just the NVRAM area, but the boot loader, the Firmware Recuperation module and the BootBlock. Just the EFI component of the BootROM image is ever upgraded.
 
Last edited:
  • Like
Reactions: Vvglyy Wzxit

permanentmacdabbler

macrumors newbie
Feb 11, 2021
25
3
MP3,1 have a separate flash memory for the NVRAM, it’s a very different BootROM architecture. You can’t clean it, btw.
Differently from a MP4,1/5,1, a failed NVRAM SPI with a MP3,1 still boots, but you can’t bless for example.
Does that flash memory difference make a 3,1 any better or worse than a 4,1/5,1 when it comes to OpenCore NVRAM write risks? Would you use a 3,1 over a 4,1 for testing for example?
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.