Sequence of Service versus Flapping checks/notifications

[Matthew Richardson]

I have spotted today a couple of cases where a service notification has
been received immediately followed by a FLAPPINGSTART notification when
using 3.0b4.  These struck me as being not quite what one might logically
expect.

For example:-

|[03-10-2007 18:01:24] SERVICE NOTIFICATION: smstest1;walkers_smtp-ky;smtp;FLAPPINGSTART (OK);notify-service-by-email;SMTP OK - 6.036 sec. response time
|[03-10-2007 18:01:24] SERVICE FLAPPING ALERT: walkers_smtp-ky;smtp;STARTED; Service appears to have started flapping (23.0% change >= 20.0% threshold)
|[03-10-2007 18:01:23] SERVICE NOTIFICATION: smstest1;walkers_smtp-ky;smtp;OK;notify-service-by-email;SMTP OK - 6.036 sec. response time
|[03-10-2007 18:01:23] SERVICE ALERT: walkers_smtp-ky;smtp;OK;HARD;3;SMTP OK - 6.036 sec. response time

|[03-10-2007 19:03:34] SERVICE NOTIFICATION: smstest1;jtc_rich-jtc01;ospf_jsy-qr-jtc01;FLAPPINGSTART (OK);notify-service-by-email;OSPF OK - Full adjacency
|[03-10-2007 19:03:34] SERVICE FLAPPING ALERT: jtc_rich-jtc01;ospf_jsy-qr-jtc01;STARTED; Service appears to have started flapping (23.9% change >= 20.0% threshold)
|[03-10-2007 19:03:33] SERVICE NOTIFICATION: smstest1;jtc_rich-jtc01;ospf_jsy-qr-jtc01;OK;notify-service-by-email;OSPF OK - Full adjacency
|[03-10-2007 19:03:33] SERVICE ALERT: jtc_rich-jtc01;ospf_jsy-qr-jtc01;OK;HARD;3;OSPF OK - Full adjacency

>From what I can see, this seems to occur only when a service moves from a
HARD non-OK state into an OK state at the same time as the flapping
threshold is reached.  I have not noticed any when transition from an OK
state to non-OK.

It occurs to me that it might be preferable to turn the logic around such
that the flapping checks and notifications are done prior to reporting any
hard change of service state.  If so, then only the FLAPPINGSTART
notifications would be issued in each of the examples above.

Best wishes,
Matthew

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/