Eventlog

[Russell Adams]

Don't get me wrong about Nagios, I use it to monitor all the disks,
processes, and nics on my hosts, but it doesn't quite fit into the
logging scheme. Nagios is very good at operational monitoring, but I
use Cricket for trend data on hosts and networks, and syslog-ng with
Logmucher for system logs.

Having used several log parsing packages, I found LogSentry was quite
good, but its now unavailable. Cisco bought Psionic Software
(www.psionic.com) and all their software (LogSentry, PortSentry,
HostSentry) is offline, though originally GPL'ed. :P

About 8 months ago when I setup my central syslog host, I decided on
syslog-ng with a multi-host directory hierarchy
(/var/log/HOSTS/hostname/year/month/day/loglevel). I also evaluated
Logmuncher at that time. I've found I prefer Logmuncher's flexibility
with my setup. A minor modification to Logmuncher to support the
multi-host directory hierarchy and I've used it ever since. It works
much like LogSentry, having dictionaries of regexp statements that
match patterns in syslog messages to ignore, warn about by default, or
immediately notify the admin as critical. As it stands, I have a
common dictionary across my hosts, and then host specific dictionaries
for ignore, warn, and critical. It sends email reports, and is
configured to send critical emails to my pager via email. Logmuncher
runs only on my central host at 5 minute intervals.

I haven't encountered any recurring critical messages. A bad disk
block once it has occurred could go on the ignore list I suppose, with
any further errors being caught by the default warning trap. Most of
my recurring logs are informational logging messages that can be
explicitly ignored. So long as the regexp is specific enough about
what to ignore, any change in the message should result in the message
becoming a warning by default, which is quite desireable.

Once I put my systems on my central log host, it took about 3 days of
wading through the spam to get my pattern dictionaries correct, with
minimal updates ever since.

One tool I've been searching for but haven't found is something to
help me identify duplicate log messages for purposes of tabulating
them and putting them into Logmuncher. Multiple messages where only
the PID changes get quite frustrating. If I could feed a file into a
tool that I could tell "return all messages that match 95% as one
message with the ambiguous parts *'d out", it would greatly speed up
the time required to make pattern files. Right now I tend to remove
PID data by columns and then doing "sort | uniq -c | sort -rn" to see
common messages, but its still quite spammy.

Archiving the log file using the multi-host directory hierarchy I
setup can be a pain, but on the bright side log rotation is
automatic. I've created a shell script to perform archives on data
older than 2 months. This also limits the number of offset files that
have to be searched.

If I've not sufficiently elaborated, I'll be happy to answer any
further questions.

Russell

On Wed, Apr 02, 2003 at 01:40:38PM -0600, Carroll, Jim P [Contractor] wrote:
> Very interesting.  Thought provoking, even.
> 
> We're using the Perl version of check_log here for checking for 'warning'
> type msgs, as well as a tweaked version of same checking for 'critical' type
> msgs.  It's been pretty good, but has been perplexing at times how best to
> handle things, not the least of which deals with the issue of central vs.
> host-based syslog scrubbing.  One issue is how to stop repeated
> notifications once you're aware of the problem, eg, a bad disk block.  At
> the individual host level, I've set check_log.pl to be volatile, since it
> fit the description (and since the non-volatile behaviour was problematic).
> 
> I'd be interested in hearing how you dealt with issues such as this.
> 
> This relates to Nagios, and yet it doesn't.  If noone else on the list jumps
> in before the end of the day, maybe you and I can take this off-list.
> 
> jc
> 
> 
> > -----Original Message-----
> > From: Russell Adams [mailto:RLAdams at Kelsey-Seybold.com]
> > Sent: Wednesday, April 02, 2003 10:06 AM
> > To: nagios-users at lists.sourceforge.net
> > Subject: Re: [Nagios-users] Eventlog
> > 
> > 
> > -= Sorry, here's the list copy. :P =-
> > 
> > I use EventReporter, http://www.eventreporter.com/en/ .
> > 
> > Its cheap ($50/host before bulk discounts), and I've never had a
> > problem with it.
> > 
> > Quite frankly, I don't use "check_log". Monitoring my system logs is a
> > job for my central syslog server, running syslog-ng, with email/paging
> > reports by LogMuncher (a distant relative of LogSentry, but thats not
> > available anymore). See http://www.campin.net/newlogcheck.html for
> > suggestions.
> > 
> > Russell
> > 
> > On Wed, Apr 02, 2003 at 10:53:56AM +0200, Johannes Dagemark wrote:
> > > Hello
> > > 
> > > I would like to monitor the eventlog on a couple of w2k 
> > boxes. The idea so far
> > > is to install a program on the w2k boxes that sends the 
> > entries in the eventlog
> > > to a syslogserver and then use the check_log plugin to 
> > check for certain
> > > problems.
> > > 
> > > Can anyone recommend a good software (small, simple and 
> > free) that can send
> > > entries in the eventlog to syslog or is there someone that 
> > has a better idea?
> > > 
> > > Regards
> > > 
> > > Johannes Dagemark
> > > 
> > > 
> > > 
> > > -------------------------------------------------------
> > > This SF.net email is sponsored by: ValueWeb: 
> > > Dedicated Hosting for just $79/mo with 500 GB of bandwidth! 
> > > No other company gives more support or power for your 
> > dedicated server
> > > http://click.atdmt.com/AFF/go/sdnxxaff00300020aff/direct/01/
> > > _______________________________________________
> > > Nagios-users mailing list
> > > Nagios-users at lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/nagios-users
> > > ::: Please include Nagios version, plugin version (-v) and 
> > OS when reporting any issue. 
> > > ::: Messages without supporting info will risk being sent 
> > to /dev/null
> > 
> > 
> > -------------------------------------------------------
> > This SF.net email is sponsored by: ValueWeb: 
> > Dedicated Hosting for just $79/mo with 500 GB of bandwidth! 
> > No other company gives more support or power for your dedicated server
> > http://click.atdmt.com/AFF/go/sdnxxaff00300020aff/direct/01/
> > _______________________________________________
> > Nagios-users mailing list
> > Nagios-users at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nagios-users
> > ::: Please include Nagios version, plugin version (-v) and OS 
> > when reporting any issue. 
> > ::: Messages without supporting info will risk being sent to /dev/null
> > 


-------------------------------------------------------
This SF.net email is sponsored by: ValueWeb: 
Dedicated Hosting for just $79/mo with 500 GB of bandwidth! 
No other company gives more support or power for your dedicated server
http://click.atdmt.com/AFF/go/sdnxxaff00300020aff/direct/01/
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null