Fixing Casper’s MySQL database with mysqlcheck
After some VMware host issues that were out of my control, the RHEL VMs that my production and test Casper servers are hosted on were unexpectedly rebooted a couple of times. When I checked the VMs afterwards, everything appeared to be OK. I figured that I had been fortunate, until my Casper test server sent me the nightly “Successful backup of the Casper database” email and my production server didn’t.
When I checked the directory when my production server stores its backups of the Casper database, there wasn’t a backup from the night before. I immediately launched the JAMF Database Utility application and had it make a backup of the production database. A task which normally would take 10 minutes or so now took 40 minutes.
To lighten the load on the database, I went into the JSS and had it manually flush all but the last week’s worth of logs (I normally retain 30 days of logs.)
Once the log flush had completed, I then rebooted the box. On reboot, the JSS initialized and then hung halfway through the JSS startup process.
Really not good.
For the details of how I fixed this, see below the jump.
Casper’s JSS server software comes in two parts, a Tomcat web application and a MySQL database named jamfsoftware. As far as I could tell, the Tomcat part was OK but was having problems communicating with Casper’s jamfsoftware MySQL database.
To ensure that my Casper-managed Macs were not trying to update the database while I was working on fixing it, I stopped the Tomcat web application by running the following command as root:
I then restarted MySQL with the following command:
After the MySQL restart, I ran the following mysqlcheck command to repair the database:
mysqlcheck -repair jamfsoftware -u root -p
In this case, mysqlcheck was set to repair all of the tables in the jamfsoftware database.
90 minutes later, the repair process had finished checking and repairing the database. I then ran optimization with mysqlcheck using the following command:
mysqlcheck -o jamfsoftware -u root -p
Since the database tables had just been repaired, optimization finished within seconds. I then rebooted the server. I could have just restarted the Tomcat web application, but I wanted to make sure everything worked on restart.
On restart, the Casper service started up normally. Later that night, I once again received the nightly “Successful backup of the Casper database” email from both my test and production servers.