Mojave Security Update 2021-004

[katbel]

Anyone installed it yet?
Is it all running smooth ?

 

[Hombre53]

All good so far, no hiccups.

 

[madrich]

All good so far, no hiccups.

Ditto.

 

[katbel]

Thanks!

 

[jbarley]

Installed smooth and easy, sorta reminds me of days long past, you know when "Apple just works"

 

[Apple_Glen_UK]

Yep, updated my MacBook Pro and all seems ok 👍

 

[thawk9455]

We're having kerberos issues with machines that have updated. Haven't got it figured out yet...

 

[neirbowj]

We're having kerberos issues with machines that have updated. Haven't got it figured out yet...

From what I can tell, the problem has to do with the KCM credential cache. After roughly two hours of uptime, any program that attempts to interact with KCM hangs indefinitely, including system applications (e.g. Ticket Viewer.app), third-party applications linked against system Heimdal libraries, and applications linked against MIT Kerberos libraries (e.g. via MacPorts). Rebooting restores functionality, but not permanently.

 

[neirbowj]

A work-around is to use a different credential cache.

# /etc/krb5.conf
[libdefaults]
    # Heimdal
    default_cc_name = FILE:/tmp/krb5.ccache
    # MIT
    default_ccache_name = FILE:/tmp/krb5.ccache

Or set it as an environment variable.

export KRB5CCNAME=FILE:/tmp/krb5.ccache

Ticket Viewer.app specifically appears to be immune to both of these, but all the other apps I've tried respond well.

 

[Martin Berthiaume]

Hello, we are experiencing the same kerberos issues.

A work-around is to use a different credential cache.

# /etc/krb5.conf
[libdefaults]
    # Heimdal
    default_cc_name = FILE:/tmp/krb5.ccache
    # MIT
    default_ccache_name = FILE:/tmp/krb5.ccache

Or set it as an environment variable.

export KRB5CCNAME=FILE:/tmp/krb5.ccache

Ticket Viewer.app specifically appears to be immune to both of these, but all the other apps I've tried respond well.

We tried your workaround but it didn't work or we didn't applied it properly. Most probably the second option.
Could you please give us more detailed steps about your workaround?

Thanks!

Martin

 

[neirbowj]

Hello, we are experiencing the same kerberos issues.

We tried your workaround but it didn't work or we didn't applied it properly. Most probably the second option.
Could you please give us more detailed steps about your workaround?

Martin:

To apply the first option, edit the configuration file indicated (/etc/krb5.conf) and ensure that at least the Heimdal (system) flavor of the default credential cache variable is set in the libdefaults section. Also, check for the existence of any of the platform-specific configuration files for Heimdal or MIT that might conflict; mainly /Library/Preferences/edu.mit.Kerberos and/or ~/Library/Preferences/edu.mit.Kerberos, which use the same format as the platform-independent file. I am a little foggy on how multiple files interact (i.e. the order in which they are evaluated), and it seems unlikely that a conflict between them is the reason this is not working for you. The first option is the best general solution.

To apply the second option, run the export command in Terminal.app (bash or zsh shell), or find a variation that works for your circumstances. The second option is a good alternative to try if the first option fails, because environment variables tend to override configuration files, but it suffers from challenges of impermanence and ancestry constraints.

 

[croaker_1]

Hello, we are experiencing the same kerberos issues.

We tried your workaround but it didn't work or we didn't applied it properly. Most probably the second option.
Could you please give us more detailed steps about your workaround?

Changing the credential cache hasn't completely solved it for me but it is a step in the right direction. I definitely agree that it is a cache issue.

Also, check for the existence of any of the platform-specific configuration files for Heimdal or MIT that might conflict; mainly /Library/Preferences/edu.mit.Kerberos and/or ~/Library/Preferences/edu.mit.Kerberos, which use the same format as the platform-independent file. I am a little foggy on how multiple files interact (i.e. the order in which they are evaluated), and it seems unlikely that a conflict between them is the reason this is not working for you. The first option is the best general solution.

Looking at the logs, the processing order of Kerberos preferences seems to be User Library, Library and then /etc/krb5.conf (actually it looks like User, User, Library, User, krb5.conf but it probably just weird logging) if this is helpful to anyone.

Also, check out this to enable debug logging for Heimdal.

tl;dr

sudo defaults write /Library/Preferences/com.apple.Kerberos logging -dict-add krb5 '0-/SYSLOG:'
sudo defaults write /Library/Preferences/com.apple.Kerberos logging -dict-add digest-service '0-/SYSLOG'
sudo defaults write /Library/Preferences/com.apple.Kerberos logging -dict-add kcm '0-/SYSLOG:'
sudo defaults write /Library/Preferences/com.apple.Kerberos logging -dict-add kdc '0-/SYSLOG:'
defaults write com.apple.MITKerberosShim EnableDebugging -bool true
defaults write com.apple.GSS DebugLevel -int 10

If I get any further I will try to post an update.

 

[croaker_1]

Following up from my previous post. After doing some (a lot of) reading about Heimdal Kerberos, reports on the #MacAdmin Slack and a lot of trial and error here is what I have discovered. The problem is definitely being caused by the credentials cache, more specifically the Heimdal kcm (credential cache server) and seems limited to machines bound to a directory (possibly only Active Directory) before Security Update 2021-004 got installed. It may also affect machines bound after the Security Update but I have not tested that. I have not yet figured out exactly what is happening but the end result is that updated machines are no longer able to access the credential cache; which consequently prevents any ticket action from taking place. I could only find two lines of interest form the logs and you can see that they relate directly to cache config:

The fix (well, really just a stop-gap measure until I can discover the actual cause) is to basically tell the OS to ignore cached Kerberos credentials for authorization (as well as screensaver). A shout out to @jojo on #MacAdmins for confirming my suspicions. To do this you will need to edit two system files (yours should look similar to these):

/etc/pam.d/authorization
    # authorization: auth account
    auth       optional       pam_krb5.so use_first_pass use_kcminit
    auth       optional       pam_ntlm.so use_first_pass
    auth       required       pam_opendirectory.so use_first_pass nullok
    account    required       pam_opendirectory.so

/etc/pam.d/screensaver
    # screensaver: auth account
    auth       optional       pam_krb5.so use_first_pass use_kcminit
    auth       required       pam_opendirectory.so use_first_pass nullok
    account    required       pam_opendirectory.so
    account    sufficient     pam_self.so
    account    required       pam_group.so no_warn group=admin,wheel fail_safe
    account    required       pam_group.so no_warn deny group=admin,wheel ruser fail_safe

Using your favorite text editor (sudo) remove use_kcminit from each file.

For good measure I also killed related processes:

sudo pkill coreauthd
sudo pkill kcm
sudo pkill kdc

In my testing this does not prevent remote users (those without connectivity to the domain controller) from logging into/unlocking their machines but I urge you to do your own testing before rolling it out.

I've also written a no-frills script to make it easy to deploy to users:

#!/bin/bash
sed -i '' "s/use_kcminit//" "/etc/pam.d/authorization"
sed -i '' "s/use_kcminit//" "/etc/pam.d/screensaver"
pkill coreauthd
pkill kcm
pkill kdc

 

[jz0309]

installed yesterday plus Safari 14.1 (again) - no issues here

 

[Martin Berthiaume]

The problem is definitely being caused by the credentials cache, more specifically the Heimdal kcm (credential cache server) and seems limited to machines bound to a directory (possibly only Active Directory) before Security Update 2021-004 got installed. It may also affect machines bound after the Security Update but I have not tested that.

I can confirm that our macs are bound to AD. We did try to unbound and rebind, before and after security update 2021-004, but it didn't fix the issue.

 

[Glideslope]

No issues. Just updated. Very smooth.

 

[Martin Berthiaume]

I've also written a no-frills script to make it easy to deploy to users:

#!/bin/bash
sed -i '' "s/use_kcminit//" "/etc/pam.d/authorization"
sed -i '' "s/use_kcminit//" "/etc/pam.d/screensaver"
pkill coreauthd
pkill kcm
pkill kdc

Your script seems to fix the issues. Is it temporary or permanent? Do we need to run it at every login?

Thanks a lot!
Martin

 

[streets]

Thanks, Martin that script worked for me. Looks like it needed a restart. I can now run the kinit command to pull down a Kerberos ticket.

 

[Martin Berthiaume]

Your script seems to fix the issues. Is it temporary or permanent? Do we need to run it at every login?

Thanks a lot!
Martin

I answered my own questions. After runnning the script once, it fixed the issue permanently and continue to work after a reboot.

@streets You can thank croaker not me

@croaker_1 Thanks a lot!
Martin

 

[BrotherJerome]

Just installed. Went quickly. Everything seems fine.

 

[harlekyn]

Following up from my previous post. After doing some (a lot of) reading about Heimdal Kerberos, reports on the #MacAdmin Slack and a lot of trial and error here is what I have discovered. The problem is definitely being caused by the credentials cache, more specifically the Heimdal kcm (credential cache server) and seems limited to machines bound to a directory (possibly only Active Directory) before Security Update 2021-004 got installed. It may also affect machines bound after the Security Update but I have not tested that. I have not yet figured out exactly what is happening but the end result is that updated machines are no longer able to access the credential cache; which consequently prevents any ticket action from taking place. I could only find two lines of interest form the logs and you can see that they relate directly to cache config:

View attachment 1782147

The fix (well, really just a stop-gap measure until I can discover the actual cause) is to basically tell the OS to ignore cached Kerberos credentials for authorization (as well as screensaver). A shout out to @jojo on #MacAdmins for confirming my suspicions. To do this you will need to edit two system files (yours should look similar to these):

/etc/pam.d/authorization
    # authorization: auth account
    auth       optional       pam_krb5.so use_first_pass use_kcminit
    auth       optional       pam_ntlm.so use_first_pass
    auth       required       pam_opendirectory.so use_first_pass nullok
    account    required       pam_opendirectory.so

/etc/pam.d/screensaver
    # screensaver: auth account
    auth       optional       pam_krb5.so use_first_pass use_kcminit
    auth       required       pam_opendirectory.so use_first_pass nullok
    account    required       pam_opendirectory.so
    account    sufficient     pam_self.so
    account    required       pam_group.so no_warn group=admin,wheel fail_safe
    account    required       pam_group.so no_warn deny group=admin,wheel ruser fail_safe

Using your favorite text editor (sudo) remove use_kcminit from each file.

For good measure I also killed related processes:

sudo pkill coreauthd
sudo pkill kcm
sudo pkill kdc

In my testing this does not prevent remote users (those without connectivity to the domain controller) from logging into/unlocking their machines but I urge you to do your own testing before rolling it out.

I've also written a no-frills script to make it easy to deploy to users:

#!/bin/bash
sed -i '' "s/use_kcminit//" "/etc/pam.d/authorization"
sed -i '' "s/use_kcminit//" "/etc/pam.d/screensaver"
pkill coreauthd
pkill kcm
pkill kdc

Thanks, that did the trick for me as well!

 

[croaker_1]

Your script seems to fix the issues. Is it temporary or permanent? Do we need to run it at every login?

Thanks a lot!
Martin

So glad it worked for you. It is looking good on my side as well...over half of around 300 Mojave machines have already been patched so far.

As you discovered, changes to pam.d config should should remain permanent as long as a future update doesn't decided it needs rewrite them all.

Thanks, Martin that script worked for me. Looks like it needed a restart. I can now run the kinit command to pull down a Kerberos ticket.

That's interesting. I will check if our users are needing to restart to make it work. In the machines I tested on I only had to kill the kdc and kcm processes (coreauthd was just because I figured it couldn't hurt). If I discover that a restart is definitely required I'll see if there are other processes that need to be killed.

 

[MacBird]

Google Drive Desktop is still broken.

 

[Martin Berthiaume]

That's interesting. I will check if our users are needing to restart to make it work. In the machines I tested on I only had to kill the kdc and kcm processes (coreauthd was just because I figured it couldn't hurt). If I discover that a restart is definitely required I'll see if there are other processes that need to be killed.

No restart was required on our side. We pushed the script through our JAMF cloud. We have over 500 Mojave Macs.

You made my day! Thanks again!
Martin

 

[Patrice Brousseau]

@Martin Berthiaume and @croaker_1 : any impact for home users? I would guess the bug affects Business or School environments but just wanna be sure...

Thanks,

Patrice