Automating AutoPkg runs with autopkg-conductor

About two weeks ago, I noticed I had an SSL error cropping up with one of my AutoPkg recipes:

[Errno socket error] EOF occurred in violation of protocol (_ssl.c:590)

When I investigated what it meant, I wound up at this lengthy issue opened for Python’s requests module. In the end, it seemed to boil down to four issues:

1. I was running AutoPkg on macOS Sierra 10.12.6.
2. The recipe I was running used a processor which called Python’s urllib2 library.
3. Python’s urllib2 library was calling the OS’s installed version of OpenSSL to connect to a server using TLSv1.2 .
4. The version of OpenSSL included with 10.12.6 does not support TLSv1.2 for the urllib2 library.

When I looked into the situation on macOS High Sierra 10.13.5, Apple had addressed the problem by replacing OpenSSL with LibreSSL. Among other improvements, LibreSSL allowed Python’s urllib2 library to be able to connect to servers using TLSv1.2. Problem solved!

Until I ran into another problem.

I had been using AutoPkgr as my way of managing AutoPkg and scheduling AutoPkg runs. However, when I set up AutoPkgr on a 10.13.5 VM and scheduled my AutoPkg nightly run, nothing happened except my CPU spiked to 100% and AutoPkgr locked up with the pinwheel of patience.

OK, maybe it was something with my VM. No problem, set up a new macOS 10.13.5 VM.

Same problem.

Maybe it was because I was trying to run the VM on VMware’s ESXi? Set up a new VM running in VMware Fusion. Same problem.

Maybe AutoPkgr was getting confused by Apple File System? I set up a 10.13.5 VM which used an HFS+ boot volume. Same problem, replicated on both ESXi and Fusion.

No matter what I tried, trying to run recipes using AutoPkgr on macOS 10.13.x resulted in the following:

• The VM’s CPU spiking to 100%
• AutoPkgr locking up with the pinwheel of patience
• My AutoPkg recipes not running

I was able to eliminate AutoPkg itself as being the issue, as running recipes from the command line using AutoPkg worked fine. With that information in mind, I decided to see if I could replicate what I most liked about using AutoPkgr into another form. In the end, my needs boiled down to three:

1. I wanted to be able to run a list of AutoPkg recipes on a scheduled basis. These recipes would be .jss recipes for uploading to a Jamf Pro server.
2. I wanted to be able to post information about those AutoPkg recipes to a Slack channel
3. I wanted all the error messages from an AutoPkg run, but I didn’t care about all the information that came from a successful AutoPkg run.

With that, I decided to draw on some earlier work done by Sean Kaiser, a colleague who had written a script for managing AutoPkg in the pre-AutoPkgr days. For more details, please see below the jump.

Sean’s solution relies on a script and LaunchDaemon running on a Mac, where it runs hourly and is set up to only send him emails if the AutoPkg logs are different from previous runs. The email notifications are a diff against the previous logs, so only the true differences get sent.

For those interested, Sean’s script is available from here:

https://github.com/seankaiser/automation-scripts/tree/master/autopkg

I was more focused on a once-daily run, so I didn’t want to use the diff methodology. After some more research, I found that my colleague Graham Pugh had written pretty much exactly what I needed: An AutoPkg post-processor named Slacker which could be used with an AutoPkg recipe list of .jss recipes to post the results to a Slack channel.

I forked a copy of the Slacker post-processor and (with Graham’s help) made some edits to it to have the output appear exactly the way I wanted it to.

New package message:

No new package message:

Along with the Slacker post-processor, I also found a script for sending multiline output to a Slack channel. This would allow me to send the complete error log from an AutoPkg run to a specified Slack webhook.

	#!/bin/bash
	tail -n0 -F "$1" | while read LINE; do
	(echo "$LINE" | grep -e "$3") && curl -X POST –silent –data-urlencode \
	"payload={\"text\": \"$(echo $LINE | sed "s/\"/'/g")\"}" "$2";
	done

view raw

tail-slack.sh

hosted with ❤ by GitHub

Using all of this, I wrote a script named autopkg-conductor which is designed to do the following:

1. Detect a list of AutoPkg recipes at a defined location and verify that the list is readable.
2. If the AutoPkg recipe list is readable and available, run the following actions:

A. Verify that AutoPkg is installed.
B. Update all available AutoPkg repos with the latest recipes.
C. Run the AutoPkg recipes in the list.

The AutoPkg run has all actions logged to ~/Library/Logs, with the logfiles being named autopkg-run-for- followed by the date.

If the optional slack_post_processor and slack_webhook variables are both populated, any AutoPkg .jss recipes should have their output sent to the Slack webhook specified in the slack_webhook variable.

If only the slack_webhook variable is populated, all output from the AutoPkg run is sent to the Slack channel. No filtering is applied, everything is sent.

If neither the slack_post_processor or slack_webhook variables are populated, no information is sent to Slack. All AutoPkg run information will be in the logs stored in ~/Library Logs.

For scheduled runs, I recommend the following:

1. Set up a user account named autopkg to run AutoPkg in.
2. Copy the autopkg-conductor script to /usr/local/bin/autopkg-conductor.sh and set the autopkg-conductor.sh script to be executable.
3. Set up a LaunchDaemon to run /usr/local/bin/autopkg-conductor.sh at a pre-determined time or interval.

For this example, the LaunchDaemon shown below will run /usr/local/bin/autopkg-conductor.sh as the autopkg user once a day at 2:00 AM.

	<?xml version="1.0" encoding="UTF-8"?>
	<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"&gt;
	<plist version="1.0">
	<dict>
	<key>AbandonProcessGroup</key>
	<true/>
	<key>EnvironmentVariables</key>
	<dict>
	<key>PATH</key>
	<string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
	</dict>
	<key>Label</key>
	<string>com.github.autopkg-nightly-run</string>
	<key>ProgramArguments</key>
	<array>
	<string>/usr/local/bin/autopkg-conductor.sh</string>
	</array>
	<key>RunAtLoad</key>
	<false/>
	<key>StartCalendarInterval</key>
	<array>
	<dict>
	<key>Hour</key>
	<integer>2</integer>
	<key>Minute</key>
	<integer>0</integer>
	</dict>
	</array>
	<key>UserName</key>
	<string>autopkg</string>
	</dict>
	</plist>

view raw

com.github.autopkg-conductor-nightly-run.plist

hosted with ❤ by GitHub

The autopkg-conductor script is available below. It’s also available from GitHub using the following link:

https://github.com/rtrouton/autopkg-conductor

	#!/bin/bash
	# AutoPkg automation script
	# Adjust the following variables for your particular configuration.
	#
	# autopkg_user_account – This should be the user account you're running AutoPkg in.
	# autopkg_user_account_home – This should be the home folder location of the AutoPkg user account
	#
	# Note: The home folder location is currently set to be automatically discovered
	# using the autopkg_user_account variable.
	#
	# recipe_list – This is the location of the plain text file being used to store
	# your list of AutoPkg recipes. For more information about this list, please see
	# the link below:
	#
	# https://github.com/autopkg/autopkg/wiki/Running-Multiple-Recipes
	#
	# log_location – This should be the location and name of the AutoPkg run logs.
	#
	# Note: The location is currently set to be automatically discovered
	# using the autopkg_user_account_home variable.
	autopkg_user_account="username_goes_here"
	autopkg_user_account_home=$(/usr/bin/dscl . -read /Users/"$autopkg_user_account" NFSHomeDirectory | awk '{print $2}')
	recipe_list="/path/to/recipe_list.txt"
	log_location="$autopkg_user_account_home/Library/Logs/autopkg-run-for-$(date +%Y-%m-%d-%H%M%S).log"
	# If you're using JSSImporter, the URL of your Jamf Pro server should be populated
	# into the jamfpro_server variable automatically.
	#
	# If you're not using JSSImporter, this variable will return nothing and that's OK.
	jamfpro_server=$(/usr/bin/defaults read "$autopkg_user_account_home"/Library/Preferences/com.github.autopkg JSS_URL)
	# Optional variables
	# To use the Slacker post-processor, you'll need to use either Graham Pugh's or my
	# fork of Graham's. For information on Graham's, please see the following post:
	#
	# http://grahampugh.github.io/2017/12/22/slack-for-autopkg-jssimporter.html
	#
	# To use mine, please add my AutoPkg repo by running the following command:
	#
	# autopkg repo-add rtrouton-recipes
	#
	# If using Graham's, the slack_post_processor variable should look like this:
	# slack_post_processor="com.github.grahampugh.recipes.postprocessors/Slacker"
	#
	# If using mine, the slack_post_processor variable should look like this:
	# slack_post_processor="com.github.rtrouton.recipes.postprocessors/Slacker"
	slack_post_processor=""
	# If you're sending the results of your AutoPkg run to Slack, you'll need to set up
	# a Slack webhook to receive the information being sent by the script.
	# If you need help with configuring a Slack webhook, please see the links below:
	#
	# https://api.slack.com/incoming-webhooks
	# https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack
	#
	# Once a Slack webhook is available, the slack_webhook variable should look similar
	# to this:
	# slack_webhook="https://hooks.slack.com/services/XXXXXXXXX/YYYYYYYYY/ZZZZZZZZZZ"
	slack_webhook=""
	# don't change anything below this line
	# Set script exit status
	exit_error=0
	# Define logger behavior
	ScriptLogging(){
	DATE=$(date +%Y-%m-%d\ %H:%M:%S)
	LOG="$log_location"
	echo "$DATE" " $1" >> $LOG
	}
	# Function for sending multi-line output to a Slack webhook. Original script from here:
	#
	# http://blog.getpostman.com/2015/12/23/stream-any-log-file-to-slack-using-curl/
	SendToSlack(){
	cat "$1" | while read LINE; do
	(echo "$LINE" | grep -e "$3") && curl -X POST –silent –data-urlencode "payload={\"text\": \"$(echo $LINE | sed "s/\"/'/g")\"}" "$2";
	done
	}
	# If the AutoPkg run's log file is not available, create it
	if [[ ! -r "$log_location" ]]; then
	touch "$log_location"
	fi
	# If the AutoPkg recipe list is missing or unreadable, stop the script with an error.
	if [[ ! -r "$recipe_list" ]]; then
	ScriptLogging "Error Detected. Unable to start AutoPkg run."
	echo "" > /tmp/autopkg_error.out
	if [[ "$jamfpro_server" = "" ]]; then
	echo "AutoPkg run failed" >> /tmp/autopkg_error.out
	else
	echo "AutoPkg run for $jamfpro_server failed" >> /tmp/autopkg_error.out
	fi
	echo "$recipe_list is missing or unreadable. Fix immediately." >> /tmp/autopkg_error.out
	echo "" > /tmp/autopkg.out
	# If a Slack webhook is configured, send the error log to Slack.
	if [[ ! -z "$slack_webhook" ]]; then
	SendToSlack /tmp/autopkg_error.out ${slack_webhook}
	fi
	cat /tmp/autopkg_error.out >> "$log_location"
	ScriptLogging "Finished AutoPkg run"
	exit_error=1
	fi
	# If the the AutoPkg recipe list is readable and AutoPkg is installed,
	# run the recipes stored in the recipe list.
	if [[ -x /usr/local/bin/autopkg ]] && [[ -r "$recipe_list" ]]; then
	ScriptLogging "AutoPkg installed at $(which autopkg)"
	ScriptLogging "Recipe list located at $recipe_list and is readable."
	echo "" > /tmp/autopkg.out
	if [[ "$jamfpro_server" = "" ]]; then
	echo "Starting AutoPkg run" >> /tmp/autopkg_error.out
	else
	echo "Starting AutoPkg run for $jamfpro_server" >> /tmp/autopkg.out
	fi
	echo "" >> /tmp/autopkg.out
	echo "" > /tmp/autopkg_error.out
	echo "Error log for AutoPkg run" >> /tmp/autopkg_error.out
	echo "" >> /tmp/autopkg_error.out
	/usr/local/bin/autopkg repo-update all 2>&1 >> /tmp/autopkg.out 2>>/tmp/autopkg_error.out
	cat /tmp/autopkg.out >> "$log_location" && cat /tmp/autopkg_error.out >> "$log_location"
	if [[ ! -z "$slack_webhook" ]]; then
	if [[ ! -z "$slack_post_processor" ]]; then
	# If both the Slacker post-processor and a Slack webhook are configured, the .jss
	# recipes should have their outputs posted to Slack using the post-processor, while
	# all other standard output should go to /tmp/autopkg.out. All standard error output
	# should go to /tmp/autopkg_error.out
	/usr/local/bin/autopkg run –recipe-list="$recipe_list" –post="$slack_post_processor" –key webhook_url="$slack_webhook" >> /tmp/autopkg.out 2>>/tmp/autopkg_error.out
	else
	# If only using a Slack webhook, all standard output should go to /tmp/autopkg.out.
	# All standard error output should go to /tmp/autopkg_error.out.
	/usr/local/bin/autopkg run –recipe-list="$recipe_list" >> /tmp/autopkg.out 2>>/tmp/autopkg_error.out
	fi
	else
	# If a Slack webhook is not configured, all standard output should go to /tmp/autopkg.out.
	# All standard error output should go to /tmp/autopkg_error.out.
	/usr/local/bin/autopkg run –recipe-list="$recipe_list" >> /tmp/autopkg.out 2>>/tmp/autopkg_error.out
	fi
	if [[ "$jamfpro_server" = "" ]]; then
	echo "Finished with AutoPkg run" >> /tmp/autopkg.out
	else
	echo "Finished with AutoPkg run for $jamfpro_server" >> /tmp/autopkg.out
	fi
	echo "" >> /tmp/autopkg.out && echo "" >> /tmp/autopkg_error.out
	cat /tmp/autopkg.out >> "$log_location"
	cat /tmp/autopkg_error.out >> "$log_location"
	ScriptLogging "Finished AutoPkg run"
	echo "" >> /tmp/autopkg_error.out
	echo "End of error log for AutoPkg run" >> /tmp/autopkg_error.out
	echo "" >> /tmp/autopkg_error.out
	if [[ -z "$slack_post_processor" ]] && [[ ! -z "$slack_webhook" ]]; then
	# If the Slacker post-processor is not configured but we do have a Slack webhook
	# set up, all standard output should be sent to Slack.
	ScriptLogging "Sending AutoPkg output log to Slack"
	SendToSlack /tmp/autopkg.out ${slack_webhook}
	ScriptLogging "Sent AutoPkg output log to $slack_webhook."
	fi
	if [[ ! -z "$slack_webhook" ]]; then
	# If using a Slack webhook, at the end of the AutoPkg run all standard
	# error output logged to /tmp/autopkg_error.out should be output to Slack,
	# using the SendToSlack function.
	ScriptLogging "Sending AutoPkg error log to Slack"
	SendToSlack /tmp/autopkg_error.out ${slack_webhook}
	ScriptLogging "Sent autopkg log to $slack_webhook. Ending run."
	fi
	fi
	exit "$exit_error"

view raw

autopkg-conductor.sh

hosted with ❤ by GitHub