Overview

Troubleshooting Methodology

Configuration Steps

Example: GroupWise Plugin

Example: Sentinel Plugin

Conclusion

Installation Instructions

Attachments

Overview

Supportconfig included in supportutils package has plugin support. This feature allows you to write your own script or program, have supportconfig run it and include it's output as a file with the supportconfig tar ball. This article will include a methodology for writing plugins, installation instructions and two example plugin scripts.

The included supportconfig-plugins package contains all the plugins mentioned in this article and any additional plugins that may be added in the future. Download Now

Troubleshooting Methodology

If you are writing a plugin to help troubleshoot an application or service, I have found it helpful to include some basic information. The information you include should help you determine if the application is installed, configured and running properly. I check the system to answer the following basic questions:

1. Is the service installed?

• If it's installed as an RPM package(s), do the RPM packages pass validation?

• Is the service configured to start at boot up?

• Is the service currently running?

• What is the state of it's current running processes?

• What network ports is the service listening on?

• Gather all the configuration files.

• Gather important log files.

• Anything else that makes sense.

Configuration Steps

In a nutshell, just copy your script to the /opt/supportconfig/plugins directory and run supportconfig. The detailed steps:

1. Download and install the latest supportutils package.

• Create /opt/supportconfig/plugins directory.

• Write your plugin. It should send it's output to stdout and stderr. Likewise, it should be able to run as a stand alone program with execute permissions. Supportconfig supports any kind of executible in the plugin directory.

• Copy your plugin to the /opt/supportconfig/plugins directory.

• Make sure your plugin has execute permission set.

• Run the supportconfig command.

• Extract the supportconfig tar ball in /var/log.

• Supportconfig saves your script and it's output to the plugin-.txt file.

• If you don't want supportconfig to run plugins, just delete or rename the /opt/supportconfig/plugins directory, or run supportconfig -p.

• The supportconfig-plugin home page is located at: http://en.opensuse.org/SC-Plugins
  

Example: GroupWise Plugin

Novell GroupWise is collaboration and productivity software. The GroupWise Supportconfig Plugin gathers important troubleshooting information about GroupWise Agents, Monitor, GWIA and WebAccess.

#!/bin/bash
#############################################################
# Name:        GroupWise Supportconfig Plugin
# Description: Gathers important troubleshooting information
#              about GroupWise Agents, Monitor, GWIA and 
#              WebAccess.
#############################################################

SVER=1.00
LOG_LINES=500   #0 means include the entire file
LOG_DIR=/var/log/novell/groupwise
CONF_DIR=/etc/opt/novell/groupwise
CONF_SHARE=/opt/novell/groupwise/agents/share

section_header() {
   echo
   echo "   --------------------------------------------------------- "
   printf "#-| %-55s |-#\n" "$*"
   echo "   --------------------------------------------------------- "
}

cat_log() {
   echo "#-> $1"
   sed -e '/^[[:space:]]*#/d' -e '/^[[:space:]]*;/d' -e '/^[[:space:]]*\/\//d' -e '/^$/d' $1
   echo
}

tail_log() {
   echo "#-> $1"
   if [ $LOG_LINES -eq 0 ]; then
      cat $1
   else
      tail -${LOG_LINES} $1
   fi
   echo
}

validate_rpm() {
   echo "#-> Validating $THISRPM"
   if rpm -V $THISRPM; then
      echo "    Status: Passed"
   else
      echo "    Status: WARNING"
   fi
   echo
}

section_header GroupWise Supportconfig Plugin, v${SVER}
section_header RPM List
if RPMLIST=$(rpm -qa | egrep -i "groupwise" | sort); then
   echo "$RPMLIST"
else
   echo "ERROR: GroupWise RPM Packages not installed"
   echo
   exit 1
fi

section_header Checking GroupWise Daemons
for i in grpwise grpwise-ma
do
   if [ -f /etc/init.d/$i ]; then
      chkconfig -l $i
      /etc/init.d/$i status
      echo
   fi
done

section_header Running Processes
ps axwwo user,pid,ppid,%cpu,%mem,vsz,rss,stat,time,cmd | head -1
ps axwwo user,pid,ppid,%cpu,%mem,vsz,rss,stat,time,cmd | grep -E "gwmta|gwpoa|gwinter|gwdva|gwia|gwmon" | grep -v grep

section_header Network Port Status
netstat -nlp | grep -E "gwmta|gwpoa|gwinter|gwdva|gwia|gwmon"

section_header Configuration Files
GWHA=${CONF_DIR}/gwha.conf
cat_log $GWHA
if [ -f $GWHA ]; then
   for i in $(\ls -A1 $CONF_SHARE)
   do
      SFILE=$CONF_SHARE/$i
      test -f $SFILE -a -s $SFILE && cat_log $CONF_SHARE/$i 
   done
else
   echo "ERROR: File not found"
   echo "File:  $GWHA"
   echo
fi

section_header Validating RPM List
for THISRPM in $RPMLIST
do
   validate_rpm $THISRPM
done

section_header Output Files: Last $LOG_LINES Lines
for OUTFILE in $(find $LOG_DIR -type f | grep output)
do
   tail_log $OUTFILE
done

section_header Log Files: Last $LOG_LINES Lines
for GDIR in $(\ls -A1 $LOG_DIR | sort )
do
   test "$GDIR" == ".mta" && continue
   GLOG="$LOG_DIR/$GDIR"
   if [ -d $GLOG ]; then
      tail_log $(find $GLOG -type f | sort | grep -E '[[:digit:]]$' | tail -1)
   fi
done

Example: Sentinel Plugin

Sentinel from Novell is a comprehensive security information and event management (SIEM) solution. The Sentinel Supportconfig Plugin gathers troubleshooting information directly related to Sentinel.

#!/bin/bash
###############################################################
# Name:        Sentinel Supportconfig Plugin
# Description: Gathers troubleshooting information directly 
#              related to Sentinel.
###############################################################

SVER=1.00
LOG_LINES=500

section_header() {
   echo
   echo "   --------------------------------------------------------- "
   printf "#-| %-55s |-#\n" "$*"
   echo "   --------------------------------------------------------- "
}

cat_log() {
   echo "#-> $1"
   cat $1
   echo
}

tail_log() {
   echo "#-> $1"
   tail -${LOG_LINES} $1
   echo
}

section_header Sentinel Supportconfig Plugin, v${SVER}
section_header Sentinel Environment Variables for root
if ! env | grep ^ESEC_; then
   echo "ERROR: Sentinel environment variables not set"
   echo
fi

if [ -f /etc/init.d/sentinel ]; then
   section_header System Daemon Status
   chkconfig -l sentinel
   /etc/init.d/sentinel status

   section_header Network Port Status
   echo "# netstat -nap | egrep \"10012|10013|10014\""
   netstat -nap | egrep "10012|10013|10014"
fi

SHOME=$ESEC_HOME
if [ -n "$SHOME" ]; then
   section_header Sentinel Broker Connections
   echo "# $SHOME/bin/list_broker_connections.sh 127.0.0.1 10012"
   $SHOME/bin/list_broker_connections.sh 127.0.0.1 10012

   section_header JAR File Version Compatibility
   for SJAR in $(find $SHOME/lib -type f | grep jar$ | sort)
   do
      printf "%-60s - %s\n" "$SJAR" "$($SHOME/bin/versionreader.sh $SJAR)"
   done

   section_header Configuration Files
   for CLOG in $(find $SHOME/config -type f | sort)
   do
      cat_log $CLOG
   done 

   section_header Log Files: Last $LOG_LINES Lines
   for SLOG in $(find $SHOME/log -type f | grep 0.log$ | sort)
   do
      tail_log $SLOG
   done 

   section_header 3rd Party Log Files: Last $LOG_LINES Lines
   for TLOG in $(find $SHOME/3rdparty -type f | grep log$ | grep esec | sort)
   do
      tail_log $TLOG
   done 

   section_header Installation Logs: Last $LOG_LINES Lines
   for ILOG in $(find /opt/novell/sentinel6/install_log/ -maxdepth 1 -type f | sort)
   do
      tail_log $ILOG
   done

   section_header Cached Event Data Files
   echo "# find $SHOME/data/events -type f | xargs ls -l"
   find $SHOME/data/events -type f | xargs ls -l
fi

Installation Instructions

1. Download and install the latest supportutils package.

• Download and install the latest supportconfig-plugins package.

• Run the supportconfig command.

• The supportconfig tar ball is saved in /var/log.

• Extract the tar ball and look at the plugin text file(s). For example, plugin-groupwise.txt.

Conclusion

The supportconfig included in the supportutils package is extensible with plugin support. Plugins are any executible program or script, which in turn saves all stdout and stderr output to a log file that is included with the other supportconfig files. The resulting tar ball can be used internally or sent to Novell with your service request information. Plugin support allows you to include any kind of information you deem important to gather. Both the GroupWise and Sentinel supportconfig plugins have been included as an example, and for functional troubleshooting purposes.