Limiting absolute floods of emails/sms

[Andreas Ericsson]

Nathan Oyler wrote:
>>Nathan Oyler wrote:
>>
>>>I wanted to know if anyone had some tips in limiting floods of
> 
> emails or
> 
>>>sms messages.
>>>
>>>Right now we use parent child relationships, but with 400 hosts
> 
> there
> 
>>>are times when the oddest things happen which will cause a huge
> 
> amount
> 
>>>of flooding.
>>>
>>>Is there a way to cap the amount of notifications that come out at a
>>>time?
>>>
>>
>>Not really, except disabling notifications when the shit hits the fan.
>>If you're talking about notifications that nobody cares about you need
>>to tweak your thresholds, notification_options or
> 
> notification_periods.
> 
> It's notifications that people care about, it's really the amount of
> them. 
> 
> Example, today we've had an issue where several machines from one
> location to another have experienced packet losses.
> 
> Each one causes a warning or critical notification, and a recovery.
> That's a lot of messages while we work on correcting the problem at one
> time. I could schedule downtime for that location but that is a pretty
> harsh reaction to the problem.
> 
> It would be preferable to do something like catch all messages every 15
> minutes, and then a condensed version of what's going on. One
> notification email which lists each issue for the last 15 minutes? Maybe
> if there have been 5 emails or more in the last 15 minutes, or some
> other criteria.
> 
> To do something like this, would one want to edit the command for emails
> to go out, and change it to call a script which would hold the
> information as to what was happening, and deal with it in that respect.
> 

Yes.

> Would this be a bad idea in your mind? 
> 

That depends on the implementation. A neat and fairly nice way of doing 
it would be to shoot them into a daemon which could keep track of the 
number of notifications for each user. When it exceeds a given threshold 
it could put them in a database of some sort and wait a given amount of 
time before sending any more. Attached to the database you could 
ofcourse have a small php-script to view and delete pending 
notifications before they're sent. It would also be quite cool if the 
little daemon could notify you (once) of the fact that it's blocking 
notifications for the pre-determined period of time before sending them on.

Sounds like a fun project. Let me know if you need any contract work 
done. I'll get you a good pricetag on it.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null