Google Keystone update breaks Macs’ ability to boot if System Integrity Protection is disabled

On the evening of Monday, September 23rd, a number of film and TV editors started reporting that their workstations were not rebooting successfully. The problem was initially blamed on the Media Composer software sold by Avid.

Here's our latest update to the @Avid user community from CEO @JeffJRosica and CPO Dana Ruzicka with some helpful advice for our creative tools users running @MediaComposer on Mac Pro.

More info on OS recovery ▶️ https://t.co/DsUk7qdf39https://t.co/NKt6yMHvSZ

— Avid (@Avid) September 24, 2019

On September 24th, more instances were reported and it became clear that this was not an issue restricted to Macs with Media Composer installed. After extensive checking and testing, the folks in the MacAdmins Slack were able to narrow down the issue to an update to Google’s Keystone software, which Google uses to update Google Chrome and other Google products on macOS.

The now-pulled Keystone update attempts to remove the /var symlink, which is usually protected by Apple’s System Integrity Protection (SIP) security feature.

On Macs where SIP was disabled, this protection did not apply and the Keystone update was able to remove the /var symlink. This symlink is not a directory itself, but points to another directory (/private/var) which contains software necessary for the operating system to boot and function correctly, so removing the /var symlink rendered the affected Macs unbootable.

As mentioned previously, Google has pulled the problematic Keystone update and has published instructions on how to remediate affected Macs. For more details, please see below the jump.

Updated System Integrity Protection status reporting script

After writing a Casper Extension Attribute script to report on the status of System Integrity Protection, I realized that I hadn’t accounted for reporting SIP’s custom configurations. These are configurations where SIP is enabled, but one or more of SIP’s protections or restrictions has been disabled. I’ve now updated the script to also report on the following SIP configurations:

• Kext Signing: disabled
• Filesystem Protections: disabled
• NVRAM Protections: disabled
• Debugging Restrictions: disabled
• DTrace Restrictions: disabled

For more details, please see below the jump.

Casper Extension Attribute script to report System Integrity Protection status

To follow up on Apple’s changing the output of csrutil status when System Integrity Protection (SIP) is disabled, I’ve posted the following Casper extension attribute to my GitHub repo to report on whether SIP is enabled or disabled.

	#!/bin/bash
	osvers_major=$(sw_vers -productVersion | awk -F. '{print $1}')
	osvers_minor=$(sw_vers -productVersion | awk -F. '{print $2}')
	# Checks to see if the OS on the Mac is 10.x.x. If it is not, the
	# following message is displayed without quotes:
	#
	# "Unknown Version Of Mac OS X"
	if [[ ${osvers_major} -ne 10 ]]; then
	echo "<result>Unknown Version of Mac OS X</result>"
	fi
	# Checks to see if the OS on the Mac is 10.11.x or higher.
	# If it is not, the following message is displayed without quotes:
	#
	# "System Integrity Protection Not Available For" followed by the version of OS X.
	if [[ ${osvers_major} -eq 10 ]] && [[ ${osvers_minor} -lt 11 ]]; then
	echo "<result>System Integrity Protection Not Available For `sw_vers -productVersion`</result>"
	fi
	if [[ ${osvers_major} -eq 10 ]] && [[ ${osvers_minor} -ge 11 ]]; then
	# Checks System Integrity Protection status on Macs
	# running 10.11.x or higher
	SIP_status=`/usr/bin/csrutil status | awk '/status/ {print $5}' | sed 's/\.$//'`
	if [ $SIP_status = "disabled" ]; then
	result=Disabled
	elif [ $SIP_status = "enabled" ]; then
	result=Active
	fi
	echo "<result>$result</result>"
	fi

view raw

gistfile1.txt

hosted with ❤ by GitHub

The script has the following functions:

If the Mac is running 10.10.x or earlier

The script reports System Integrity Protection Not Available For and then reports the relevant version of OS X. For example, the script returns the following output on a Mac running OS X 10.10.5:

System Integrity Protection Not Available For 10.10.5

If the Mac is running 10.11.x or later

This script uses csrutil status to check SIP’s status. If System Integrity Protection is disabled, the script returns the following output:

Disabled

If System Integrity Protection is enabled, the script returns the following output:

Active

For those interested, the script is available on my GitHub repo:

https://github.com/rtrouton/rtrouton_scripts/tree/master/rtrouton_scripts/Casper_Extension_Attributes/check_system_integrity_protection_status

System Integrity Protection’s csrutil status’ message change for OS X 10.11.2

In order to check whether System Integrity Protection (SIP) is enabled or disabled on a Mac running OS X El Capitan, you can use the csrutil command to report on SIP’s current status. For example, to learn if SIP is enabled or disabled, run the following command:

csrutil status

This command can be run without root privileges and will tell you if SIP is on or off.

If SIP is enabled on 10.11.0 or higher, you should receive the following message:

System Integrity Protection: enabled

If SIP is disabled on OS X 10.11.0 or 10.11.1, you may receive a confusing message which indicates that SIP is enabled, followed by a list of individual SIP functions which are disabled. If all functions listed are showing as being disabled, SIP is actually completely disabled; it’s just confusingly worded.

It appears that Apple has updated the status message on OS X 10.11.2 to make it much more clear when SIP is disabled. On 10.11.2, if SIP is disabled, you now should receive the following message:

System Integrity Protection: disabled

Configuring System Integrity Protection without booting to Recovery HD

One interesting part of Apple’s developer documentation for System Integrity Protection (SIP) is the note shown below, indicating that it’s possible to configure SIP for environments that can’t access Recovery.

When I followed up with Apple about this, I was told that this meant I could configure it using NetBoot, using a NetBoot set that included the needed Recovery environment.

The example used was leveraging a new option in System Image Utility to create a package-only installation NetBoot set.

This new type of NetBoot set is is designed to install only scripts, configuration profiles and packages as opposed to installing an OS. For more details, see below the jump.

System Integrity Protection – Adding another layer to Apple’s security model

As part of the release of OS X El Capitan, Apple has added a new layer named System Integrity Protection (SIP) to its security model. To understand how System Integrity Protection fits in, let’s first take a look at Apple’s security model as it existed as of OS X Yosemite.

 

Gatekeeper

Gatekeeper is one of the outer lines of defense. It allows users to restrict which sources they can install applications from, with the general idea being that malware will not be from an allowed source.

 

Sandboxing

OS X also uses sandboxing extensively. A sandbox typically provides a tightly controlled set of resources for programs to run in. Network access, the ability to inspect the host system, or reading from input devices is usually disallowed or heavily restricted.

 

POSIX permissions

OS X uses the Unix permissions model as defined by POSIX, which governs which users and groups can access which files and directories. If a particular user account requests access to a particular file or directory and does not have the necessary rights, that account is refused access.

 

Keychains

The innermost layer of defense are keychains. Keychains are very specialized databases which are designed for the storing of secrets, like passwords, private keys, PIN numbers, and then controlling access to those secrets. To help protect these secrets, keychains are encrypted.

 

There’s an issue with this model though and it’s been there for decades. It pre-exists OS X and even pre-exists Apple as a company. That issue is found in the POSIX permissions layer.

Root

 

Root is the superuser for a Unix system and the Unix permissions model is designed around the assumption that root has access to everything. Apple has not ignored this issue and has put some controls in place to limit the actual root user. These controls include disabling the root user account, discouraging its use, and providing ways to access elevated or root privileges using other means.

However, the root user account is still present and still can do anything on the system.

 

System Integrity Protection

To limit what the superuser can do and add another layer to OS X’s security model, Apple has developed SIP and deployed it as part of OS X El Capitan. SIP is designed to limit the power of root and to protect the system even from the superuser. For more details, see below the jump.

System Integrity Protection and resetting NVRAM

OS X El Capitan’s new System Integrity Protection (SIP) security feature stores its active security configuration in NVRAM. This allows SIP’s configuration to persist across OS installs, but this design choice also means that resetting NVRAM will cause SIP to reset as well. In my testing, this reset will result in the following SIP configuration:

• SIP will be enabled with all protections in place
• No entries will be set in the SIP NetBoot whitelist

Resetting the NVRAM, otherwise known as a PRAM reset or PRAM zap, has been a standard part of the Mac troubleshooting toolkit for a long time and is performed by pressing and holding down the Option, Command (⌘), P, and R keyboard keys at startup.

For shops that do not plan to change SIP’s default configuration or set a NetBoot whitelist, NVRAM resets causing SIP’s configuration to also reset should not affect normal operations.

However, for those shops who will need to maintain a NetBoot whitelist or a custom SIP configuration, I would advise education where needed about this change and how it affects SIP configuration in your environment.

System Integrity Protection and the end of XProtect management for browser plug-ins

OS X El Capitan adds a new security feature named System Integrity Protection (SIP). Among other things, SIP prevents parties other than Apple from adding, deleting or modifying directories and files stored in certain directories:

• /bin
• /sbin
• /usr
• /System

Apple has indicated that the following directories are available for developers to access:

• /usr/local
• /Applications
• /Library
• ~/Library

All directories in /usr except for /usr/local are protected by SIP.

SIP’s protection of /System affects XProtect’s XProtect.plist and XProtect.meta.plist configuration files as they are stored in the following location inside /System:

/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.plist

/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.meta.plist

As the XProtect configuration files will be locked against editing on OS X El Capitan, this means that they can no longer be managed to allow older versions of the Flash and Java browser plug-ins to run.

If your shop includes a mission-critical system that requires using older Flash or Java browser plug-ins, I recommend working with your vendor and/or in-house developers to find out:

1. If the use of the Java and/or Flash browser plug-ins can be discontinued.
2. If their use can’t be discontinued, if the system in question can be updated to support the latest versions of these plug-ins and continue to be compatible as new versions of the Java and/or Flash browser plug-ins are released.

Update – 9-14-2015: Josh Dyson has pointed out that there is a way to allow older plug-ins to access specific sites.

By adding the needed sites to a whitelist in Safari and setting those specific sites to Allow Always, those sites’ functions will be accessible with the older browser plug-in even if XProtect would otherwise block the use of the plug-in. Websites not included in the whitelist would still have the use of the plug-in blocked.

Apple has provided a KBase article showing how to manage Safari plug-in options, including how to whitelist websites, using a configuration profile. It’s available via the link below:

https://support.apple.com/HT202947

NetBooting and System Integrity Protection

Apple took an unusual step this week and released a knowledgebase (KBase) article that refers to an as-yet unreleased operating system:

Prepare for NetBoot, NetInstall, and NetRestore requirements in OS X El Capitan

I can only praise the decision to create it. The content covered affects a number of enterprise Mac environments and gives the Mac admins who support those environments time to prepare for an important change which may affect them.

That said, the KBase article itself is confusingly written and also includes an error. For more details, see below the jump.

Uninstalling macOS system extensions

With the ongoing change from kernel extensions to system extensions, one new thing Mac admins will need to learn is how to uninstall system extensions. Fortunately, Apple has provided a tool as of macOS Catalina that assists with this: systemextensionsctl

If you run the systemextensionsctl command by itself, you should get the following information about usage:

systemextensionsctl: usage:
	systemextensionsctl developer [on|off]
	systemextensionsctl list [category]
	systemextensionsctl reset  - reset all System Extensions state
	systemextensionsctl uninstall  ; can also accept '-' for teamID

The last verb, uninstall, is what allows us to remove system extensions. For more details, please see below the jump.