Nagios + Logsurfer or Swatch?

[Jim Mozley]

Emmett Hogan wrote:
> Hi Folks,
> 
> I need to be able to scan log files looking for several different 
> strings, and sometimes the number of times a string appears is 
> important.  Often times, this is on a rather large log file, so the 
> existing "check_log" script would tend to load the machine quite a bit 
> since it can only handle one string at a time, plus it really cannot 
> handle the "alarm if this_string occurs more than 10 times in 10 
> minutes" in the log.
> 
> I was just wondering if anyone had written a more robust check_log 
> extension, or possibly come up with some logsurfer or swatch integration 
> checks.
> 
> ...or is it time to "roll my own"? :-)

Below is something I put on this list previously. At someone else's 
suggestion I submitted this as a FAQ, but I don't think it ever made it 
to the web site. Using SEC seems a better bet than swatch (although this 
si what I talk about below). Please look thorough the archives for "SEC" 
in the subject, esepcially useful are submissions by Stanley Hopcroft.

Jim Mozley


____________________________________________________________

Better late than never I hope. The following is a setup for using 
syslog-ng, swatch and a script to pass passive events into Nagios. This 
is based on getting syslog messages from a set of network devices, 
although the same principle would apply to hosts. I've tried to make the 
example fairly generic, for instance we generate the nagios config for 
this automatically from device configurations, so please don't 
copy/paste this without understanding what's going on.

1. Define services in Nagios. For instance if you want to monitor MPLS 
messages for each network device define an MPLS service as below.

#Service definition template
define service{
         name                            mpls-service-template
         active_checks_enabled           no
         passive_checks_enabled          1
         parallelize_check               1
         obsess_over_service             0
         check_freshness                 0
         freshness_threshold             28800
         notifications_enabled           1
         event_handler_enabled           1
         flap_detection_enabled          1
         process_perf_data               1
         retain_status_information       1
         retain_nonstatus_information    1
         register                        0
         is_volatile                     1
         check_period                    none
         max_check_attempts              1
         normal_check_interval           5
         retry_check_interval            1
         contact_groups                  network-admins
         notification_interval           120
         notification_period             24x7
         notification_options            w,c,r
         }

define service{
         use             mpls-service-template         ; template
         host_name       host1
         service_description             mpls
         check_command   ""
         }


2. Configure syslog-ng. Once you have got your config file ready you 
will need to disable the native syslog daemon and start syslog-ng.

syslog-ng config; this is for Solaris the source may need to change for 
linux/BSD:

options {
         keep_hostname(off);
         long_hostnames(off);
         sync(1);
         log_fifo_size(2048);
         bad_hostname("%");
         };

source all {
         sun-stream("/dev/log" door("/etc/.syslog_door"));
         internal();
         udp();
};


# Put each day's log in a separate file within a directory for
# each host.
destination d_hosts {
         file("/var/log/hosts/$HOST/$HOST-$YEAR$MONTH$DAY"
         owner(root) group(syslog) perm(0660) dir_perm(0750) 
create_dirs(yes));
};


# To send messages to swatch
destination d_swatch {
         program("/usr/local/bin/swatch --config-file=/etc/swatchrc 
--read-pipe=\
"cat /dev/fd/0\"");
};


# log all messages in a directory per host
log {
         source(all);
         destination(d_hosts);
};

# send all logs to swatch
log {
         source(all);
         destination(d_swatch);
};



3. Install and configure swatch.

Sample line from swatchrc

watchfor /pattern-i-want-to-match/
         exec /path/to/my/script.pl $*

So for instance if you are looking out for MPLS messages and your 
devices include "MPLS" in these syslog message match the pattern /MPLS/

4. Create the script

This should build a passive command based on the contents of the syslog 
message.

For instance it might contain element such as:

if ( $syslogmsg =~ /down/ ) {
     $nagios_code = 2; # Critical
}

You will need to extract the hostname from the syslog message (and 
possibly the service unless this is hardcoded - it is in this example).

In the end you want to build a message such as:

my $cmd = "[$epoch] 
PROCESS_SERVICE_CHECK_RESULT;$host;$service;$nagios_code;$msg";

And fire it into nagios:

my $echo = '/usr/bin/echo';
my $pipe = '/usr/local/nagios/var/rw/nagios.cmd';

system "$echo \"$cmd\" >> $pipe";

(As an aside I tried the perl way of doing this rather than a system 
command and had a problem on Solaris I couldn't resolve.)

I would recommend you take a look at Al Toby's module on CPAN for 
passing commands to Nagios.

There are also some shell script examples of passing Nagios commands 
supplied with the distribution.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null