Home > Casper, Java, JSS, Linux, Scripting > Monitoring the Casper JSS Tomcat on Red Hat Linux

Monitoring the Casper JSS Tomcat on Red Hat Linux

December 14, 2011 Leave a comment Go to comments

In my Casper setup, Casper’s JSS depends on a Jamf-installed Tomcat 7 installation on both my Casper production and Casper test servers, both of which are hosted on Red Hat Enterprise Linux 6.x VM servers.

To make sure that Tomcat is restarted automatically in case of a problem, a set of scripts has been installed with an accompanying crontab entry to check Tomcat to make sure it’s running. If not, an email with diagnostic information is sent then Tomcat is restarted. See below the jump for the scripts and the root crontab entry I’m using.

Script location:

The scripts are stored in /scripts on both my Casper production and Casper test servers. I created this directory, you can use whatever directory you prefer as long as you update it in the scripts.

Crontab:

The following entry has been added to the root crontab


*/1 * * * * /scripts/tomcat_check.sh 2>&1 >> /dev/null

Scripts:

The scripts are available here on my GitHub repo.

tomcat_check.sh – monitors Tomcat by checking to see if port 8080 on localhost is active. If not, tomcat_check.sh triggers /scripts/tomcat_report.sh to send an report via email. After the email is sent, Tomcat is then stopped and started using the startup scripts for the Casper Tomcat installation.

#!/bin/bash
# Services Restarter - Automatically restart tomcat if it dies
/bin/netstat -ln | /bin/grep ":8080 " | /usr/bin/wc -l | /bin/awk '{if ($1 == 0) system("/scripts/tomcat_report.sh; /etc/rc.d/init.d/jamf.tomcat7 stop; /etc/rc.d/init.d/jamf.tomcat7 start") }'

tomcat_report.sh – Sends an report via email. The emailed report includes the last 60 lines of JAMFSoftwareServer.log, a vmstat sample and load infomation.


#!/bin/sh
PATH=/bin:/usr/bin:/sbin:/usr/sbin export PATH

# Define the recipient.
RECIP="email@address.here"

# That should be it for the necessary configuration part. The rest can be pretty much as-is.
NAME=`hostname`
LOGS="/tmp/tomcat-restart.txt"
HWLOGDATE=$(printf "`date "+%a %h %e"` \n")
SEND="tomcat_restart@`hostname`"

# Now begin writing the daily report.
echo "From: Tomcat Restart Report " > $LOGS
echo "To: $RECIP" >> $LOGS
echo "Subject: $NAME Tomcat Restart Report" - `date` >> $LOGS

ADDY=`ifconfig eth1 |grep "inet addr" |awk '{print $2}' |awk -F: '{print $2}'`

# Give an introduction.
echo "***********************************************************************" >> $LOGS
echo "***** Hi. You're receiving this because Tomcat restarted" >> $LOGS
echo "***** This is the report for `date "+%a %h %e"`. " >> $LOGS
echo "***** Report is for `hostname` ($ADDY). " >> $LOGS
echo "***********************************************************************" >> $LOGS
echo " " >> $LOGS
echo " " >> $LOGS

# Check the uptime, so we can notice any unexpected and automatic reboots.
echo "Uptime" >> $LOGS
echo "------" >> $LOGS
echo `uptime` >> $LOGS
echo " " >> $LOGS
echo " " >> $LOGS

# Check to see how much space we have left on the volumes.
echo "FREE SPACE" >> $LOGS
echo "----------" >> $LOGS
df -khl >> $LOGS
echo " " >> $LOGS
echo " " >> $LOGS

# This looks at who's connected at the time of this report's generation.
# It's probably not too interesting. We have to filter for this host's IP
# address because it might be a syslog server and the logs would be cluttered.
echo "CURRENTLY ESTABLISHED CONNECTIONS" >> $LOGS
echo "---------------------------------" >> $LOGS
netstat -an | grep -i "established" | grep $ADDY >> $LOGS
echo " " >> $LOGS
echo " " >> $LOGS

# This looks at who's connected via SSH or at the console at the time of this report's generation.
echo "CURRENTLY ESTABLISHED SSH AND CONSOLE CONNECTIONS" >> $LOGS
echo "-------------------------------------------------" >> $LOGS
who >> $LOGS
echo " " >> $LOGS
echo " " >> $LOGS

# This reports on the virtual memory stats at the time of the stoppage
echo "REPORT VIRTUAL MEMORY STATISTICS" >> $LOGS
echo "--------------------------------" >> $LOGS
vmstat 1 10 >> $LOGS
echo " " >> $LOGS
echo " " >> $LOGS

# This tails /usr/local/jss/logs/JAMFSoftwareServer.log and hopefully catches the error
echo "LAST 60 LINES OF THE JSS SERVER LOG" >> $LOGS
echo "--------------------------------" >> $LOGS
tail -60 /usr/local/jss/logs/JAMFSoftwareServer.log >> $LOGS
echo " " >> $LOGS
echo " " >> $LOGS

# Read LOGS, pipe it to sendmail and fire off an email.
cat $LOGS | sendmail -f $RECIP -t

# Get rid of the files.
rm $LOGS

Sample email report output:


***********************************************************************
***** Hi. You're receiving this because Tomcat restarted
***** This is the report for Tue Dec 13.
***** Report is for jss.server.name (ip.address.here).
***********************************************************************

Uptime
------
22:01:01 up 4:31, 2 users, load average: 0.03, 0.02, 0.00

FREE SPACE
----------
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 20G 4.7G 15G 25% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 97M 43M 50M 46% /boot
/dev/sda4 4.0G 271M 3.5G 8% /opt
/dev/mapper/vg_data-lvol0
504G 39G 441G 9% /data

CURRENTLY ESTABLISHED CONNECTIONS
---------------------------------
tcp 0 0 ip.address.here:22 ip.address.here:60654 ESTABLISHED
tcp 0 0 ip.address.here:49254 ip.address.here:389 ESTABLISHED
tcp 0 0 ip.address.here:49251 ip.address.here:389 ESTABLISHED
tcp 0 0 ip.address.here:49250 ip.address.here:389 ESTABLISHED
tcp 0 0 ip.address.here:22 ip.address.here:57903 ESTABLISHED
tcp 0 0 ip.address.here:49253 ip.address.here:389 ESTABLISHED
tcp 0 0 ip.address.here:22 ip.address.here:57906 ESTABLISHED
tcp 0 0 ip.address.here:49252 ip.address.here:389 ESTABLISHED

CURRENTLY ESTABLISHED SSH AND CONSOLE CONNECTIONS
-------------------------------------------------
root pts/0 Dec 13 21:57 (ip.address.here)

REPORT VIRTUAL MEMORY STATISTICS
--------------------------------
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 4578176 34628 3060384 0 0 94 12 79 94 1 0 99 0 0
0 0 0 4578100 34628 3060388 0 0 0 0 57 41 0 0 100 0 0
0 0 0 4578100 34628 3060388 0 0 0 0 57 66 0 0 100 0 0
0 0 0 4578100 34628 3060388 0 0 0 0 27 35 0 0 100 0 0
0 0 0 4578100 34628 3060388 0 0 0 0 34 35 0 0 100 0 0
0 0 0 4578100 34628 3060388 0 0 0 0 49 45 0 0 100 0 0
0 0 0 4578092 34636 3060388 0 0 0 52 51 60 0 0 100 0 0
0 0 0 4578100 34636 3060388 0 0 0 0 60 49 0 0 100 0 0
0 0 0 4578092 34636 3060388 0 0 0 0 42 43 0 0 100 0 0
0 0 0 4578100 34636 3060388 0 0 0 0 50 58 0 1 100 0 0

LAST 60 LINES OF THE JSS SERVER LOG
--------------------------------
2011-12-13 19:31:13,650 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,651 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,652 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,652 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,653 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,654 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,655 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,657 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,659 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,661 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,662 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,671 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,686 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupPurchasingInfo.LookupPurchasingInfo).
2011-12-13 19:31:13,687 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.Attachment.lookupAttachmentList).
2011-12-13 19:31:13,727 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterIPhone.EnterIPhone).
2011-12-13 19:31:13,728 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterIPhone.EnterIPhone).
2011-12-13 19:31:13,730 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterPurchasingInformation.EnterPurchasingInformation).
2011-12-13 19:31:13,732 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupLocation.LookupLocation).
2011-12-13 19:31:13,784 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupIPhone.LookupIPhoneByUdid).
2011-12-13 19:31:13,785 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupIPhone.LookupIPhoneDetails).
2011-12-13 19:31:13,787 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupLocation.LookupLocation).
2011-12-13 19:31:13,788 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupIPhoneApplication.LookupIPhoneApplications).
2011-12-13 19:31:13,791 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupPurchasingInfo.LookupPurchasingInfo).
2011-12-13 19:31:13,792 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.Attachment.lookupAttachmentList).
2011-12-13 19:31:13,793 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupIPhoneProfilesAndCerts.LookupIPhoneAssets).
2011-12-13 19:31:13,794 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupIPhoneProfilesAndCerts.LookupIPhoneAssets).
2011-12-13 19:31:13,794 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupIPhoneProfilesAndCerts.LookupIPhoneAssets).
2011-12-13 19:31:13,795 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,796 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,797 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,798 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,799 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,799 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,800 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupReportID).
2011-12-13 19:31:13,801 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,802 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,803 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,803 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,804 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,805 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupComputerInfo.LookupComputerInfo).
2011-12-13 19:31:13,806 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupPurchasingInfo.LookupPurchasingInfo).
2011-12-13 19:31:13,807 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.Attachment.lookupAttachmentList).
2011-12-13 19:31:13,827 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterIPhone.EnterIPhone).
2011-12-13 19:31:13,829 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterIPhone.EnterIPhone).
2011-12-13 19:31:13,830 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterPurchasingInformation.EnterPurchasingInformation).
2011-12-13 19:31:13,832 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.LookupLocation.LookupLocation).
2011-12-13 19:31:13,833 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterLocation.EnterLocation).
2011-12-13 19:31:13,834 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterLocationChange.EnterLocationChangeIPhone).
2011-12-13 19:31:13,835 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterSoftware.EnterIPhoneApplication).
2011-12-13 19:31:13,889 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterApplicationUsageLog.clearLog).
2011-12-13 19:31:13,891 [WARN ] [DataSource ] - WARNING: This session (ID: F5FA7CAF12077B829E308FC5A8FD9B4C) has obtained 2 connections (Calling Function: com.jamfsoftware.EnterApplicationUsageLog.EnterApplicationUsageLogs).
2011-12-13 21:58:31,958 [WARN ] [SmartComputerGroupMonitor] - SmartComputerGroupMonitor aborted.
2011-12-13 21:58:31,959 [WARN ] [LogReaper ] - LogReaper aborted: java.lang.InterruptedException: sleep interrupted
2011-12-13 21:59:16,638 [INFO ] [InitializeServer ] - Initializing JSS...
2011-12-13 21:59:17,241 [INFO ] [VerifyDatabaseSchema ] - Verifying Database Schema...
2011-12-13 21:59:18,123 [INFO ] [VerifyDatabaseSchema ] - Verify Database Schema complete.
2011-12-13 21:59:20,500 [INFO ] [Init ] - CA is present. No need to recreate.
2011-12-13 21:59:23,526 [INFO ] [InitializeServer ] - Finished JSS Initialization.
2011-12-13 22:00:33,929 [WARN ] [SmartComputerGroupMonitor] - SmartComputerGroupMonitor aborted.
2011-12-13 22:00:33,930 [WARN ] [LogReaper ] - LogReaper aborted: java.lang.InterruptedException: sleep interrupted

Categories: Casper, Java, JSS, Linux, Scripting
  1. No comments yet.
  1. No trackbacks yet.

Leave a comment