Problems with Distributed Monitoring

[Bill Omer]

> I'm doing many thousands of distributed service checks but no host
> checks. I'll see if I can help.

Hey Marc, thanks for the help!

> > On the distributed server, I'm using:
> >
> > ochp_command=submit_check_result
> > ocsp_command=submit_check_result
> > obsess_over_service=1
> > obsess_over_hosts=1
> >
>
> Also set log_event_handlers=1 while your troubleshooting. That'll let
> you know if Nagios is attempting to call your ochp/ocsp commands.

That was already enabled.

<snip>

I've created two scripts, one called submit_service_check and
submit_host_check.  Here are the two sections for these scripts:

define command{
        command_name    submit_service_check
        command_line    /usr/lib/nagios/plugins/submit_service_check
$HOSTNAME$ '$SERVICEDESC$' $SERVICESTATEID$ '$SERVICEOUTPUT$'
        }


define command{
        command_name    submit_host_check
        command_line    /usr/lib/nagios/plugins/submit_host_check
$HOSTNAME$ '$HOSTSTATE$' '$HOSTOUTPUT$'
        }


In my nagios.cfg, I have:
ochp_command=submit_host_check
ocsp_command=submit_service_check


The good bits of my submit_host/service scripts are:

/usr/bin/printf "%s\t%s\t%s\t%s\n" "$1" "$2" "$return_code" "$4" |
/usr/lib/nagios/plugins/send_nsca -H <remote ip> -c
/usr/lib/nagios/send_nsca.cfg

The $4 shouldn't matter when a host result is sent, it should just be ignored.



> > Finnaly, I can run /usr/lib/nagios/plugins/submit_check_result
> > manually and it replies with:
> > 1 data packet(s) sent to host successfully.
> >
> > However I never see anything in /var/log/nagios/nagios.log, even
> > though I am using  log_passive_checks=1
>
> Enable debug in nsca.cfg on your master and watch /var/log/messages
> while you submit. Is nsca running as a deamon or in (x)inetd mode?
> Verify that you have the correct command_file specified in nsca.cfg and
> that the user that nsca is running as has permissions to write to the
> named pipe.
>


I'm running nsca as a daemon.  I've enabled debugg and I've also added
a logger command to my two scripts, so I can see when it's being ran
in /var/log/messages.

I can run this manually and it actually works!
/usr/lib/nagios/plugins/submit_service_check AnnapolisFW UP PING "OK -
Packet loss = 0%, RTA = 79.41 ms"

On the centeral server, I see the following in /var/log/messages:
Dec 18 16:03:45 ip-208-109-28-219 nsca[10438]: SERVICE CHECK -> Host
Name: 'AnnapolisFW', Service Description: 'UP', Return Code: '2',
Output: 'OK - Packet loss = 0%, RTA = 79.41 ms'
Dec 18 16:03:45 ip-208-109-28-219 nsca[10438]: End of connection...
Dec 18 09:03:46 ip-208-109-28-219 nagios: EXTERNAL COMMAND:
PROCESS_SERVICE_CHECK_RESULT;AnnapolisFW;UP;2;OK - Packet loss = 0%,
RTA = 79.41 ms



When I put nsca in to debug mode, I found a typeo in the passwords I
picked.  So now I can submit results and they work just fine.
However, when these scripts are being ran via nagios, for some reason
I'm not seeing anything show up in my /var/log/messages on the central
server.

On the distributed monitoring server, I see in /var/log/messages the
logger messages from where nagios is actually executing the script,
but I just dont see anything else.


I did however solve my own problem.   I wrapped the
submit_service_check script with logger and it logged the output from
send_nsca, which just so happened to have the key problem in it.
Apparently the nsca config file I was using only had read permissions
for root.  A quick chmod a+r send_nsca.cfg  and now, all is good.

Thanks for the help Marc.  You troubleshooting assistances was exactly
what I needed to get this knocked out.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null