eventhandler not working!

[Jasmine Chua]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone,

I have a problem in getting my monitoring server to run the event-handler to 
restart a program remotely. I have already made sure that eventhandlers are 
enabled in the main config file. Below are the steps I take to do it.  

Help anyone? 
Thanks .. any help will be truly appreciated!

In the nrpe.cfg on the remote server, i have added the following command:
command[check-program]=/usr/local/nagios/libexec/restart_program

On the remote server hostABC:

# restart_program script
#!/bin/sh
#
# Event handler script for restarting program on hostABC

  echocmd="/bin/echo"
  killcmd="/bin/kill"
  callprogram="export DISPLAY=:0 ; program &" 

        # Checks if program 
        # exists for killing
        check=`ps ax | grep program | grep -v grep | head -1`

        # Grabs the process id of the program
        process=`$echocmd $check | cut -f 1 -d ' '`

        if [ "$process" != "" ] ; then
                $killcmd $process
        fi
        su admin -c "$callprogram" &

        echo "Program restarted!"

exit 0


The restart_program script works manually when tested ... and actually restart 
the program. However, I cant get Nagios monitoring server to restart the 
script remotely. It returns the appropriate output from my restart_program 
script but does not actually restart the program.  

When testing the plugins on the monitoring server:

./check_nrpe xxx.xxx.xxx.xxx -c check-program
Program restarted!
./call_program CRITICAL HARD 3
Program restarted!

In the log file on the monitoring server:

SERVICE ALERT: hostABC;Memory;CRITICAL;HARD;5;CRITICAL - total %MEM for 
process ProgramA : 40.1
SERVICE EVENT HANDLER: hostABC;Memory;CRITICAL;HARD;5;call_program

But, the program on the remote server is not restarted at all. 

My service template on the monitoring sever:

define service{
        name                            ABC-service
        active_checks_enabled           1       
        passive_checks_enabled          0    
        parallelize_check               1       
        obsess_over_service             0      
        check_freshness                 0      
        notifications_enabled           0      
        event_handler_enabled           1       
        flap_detection_enabled          1       
        process_perf_data               0       
        retain_status_information       1       
        retain_nonstatus_information    1     

        register                        0       
        }

define service{
        use                             ABC-service
        host_name                       hostABC
        service_description             Memory
        check_command                   check_nrpe!check-program
        max_check_attempts              5
        event_handler                   call_program
        normal_check_interval           5
        retry_check_interval            1
        check_period                    24x7
        notification_interval           0
        notification_period             24x7
        notification_options            w,u,c,r
        contact_groups                  admin
}

My command template on the monitoring server:

# 'call_program' command definition
define command{
        command_name    call_program
        command_line    $USER2$/call_program $SERVICESTATE$ $STATETYPE$ 
$SERVICEATTEMPT$
        }

On the monitoring server:
In the /usr/local/nagios/libexec/eventhandlers/ directory:

#!/bin/sh
# 'call_program' script
# Event handler script for calling hostABC NRPE daemon to restart program
#
# Note: This script will only call to restart program if the memory service is
# retried 3 times (in a "soft" state) or if the memory service
# falls into a "hard" error state
#

# Arguments:
# $1 = service_state
# $2 = state_type
# $3 = service_attempt

hostABC="xxx.xxx.xxx.xxx"
checknrpedir="/usr/local/nagios/libexec"

# What state is the memory service in?

case "$1" in
        OK)
                # There is no need to do anything.
                # The service is OK.

        ;;
        WARNING)
                # There is no need to do anything still.
                # The service is just in WARNING state.

        ;;
        UNKNOWN)
                # No need for anything because we dont really know
                # the exact problem.

        ;;
        CRITICAL)
                # The memory is heavy. There is a need to restart program.

                # See the state type if it is soft or hard?
                case "$2" in

                        SOFT)

                                # Determine the number of service attempts.
                                # Only restart program if its the 3rd attempt
                                # to confirm that it is really critical

 case "$3" in

                                        3)

                                                $checknrpedir/check_nrpe 
$hostABC -c "check-program"

                                        ;;

                                esac
                        ;;
                        HARD)
                                        # If the check has tried 4 times and
                                        # the check still fails it will fall
                                        # into hard state. But normally the
                                        # 3rd try should restart program and
                                        # solve the memory problem.

                                                $checknrpedir/check_nrpe 
$hostABC -c "check-program"

                        ;;
                esac
        ;;
esac
exit 0


- -- 
Jasmine Chua
Security Engineer, SecureCiRT (A SBU of Z-Vance Pte Ltd)
http://www.securecirt.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE94j6DNgvTa7Hj2AURAnJJAKCgMJh8NhdCShQw4ow0z6Ah0IZytQCcCA+2
NTjL58I+z0MxwBDB4hi22PE=
=KgtH
-----END PGP SIGNATURE-----



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf