Hosts and services not sending mail

[Gary Every]

I'm pretty sure I've got everything set up correctly, as yesterday I was
getting notifications sent out, and today there are none going out.

I've added some services that I knew would go critical, and started watching
nagios.log. Here is a snippet from yesterdays log

[1179271433] EXTERNAL COMMAND:
SCHEDULE_FORCED_HOST_SVC_CHECKS;devstack01;1179271433
[1179271440] EXTERNAL COMMAND:
SCHEDULE_FORCED_HOST_SVC_CHECKS;devstack02;1179271440
[1179271445] HOST ALERT: devstack01;DOWN;SOFT;1;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271448] HOST ALERT: devstack01;DOWN;SOFT;2;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271451] HOST ALERT: devstack01;DOWN;SOFT;3;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271451] EXTERNAL COMMAND:
SCHEDULE_FORCED_HOST_SVC_CHECKS;ilom-cp1;1179271449
[1179271454] HOST ALERT: devstack01;DOWN;SOFT;4;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271457] HOST ALERT: devstack01;DOWN;SOFT;5;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271460] HOST ALERT: devstack01;DOWN;SOFT;6;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271460] EXTERNAL COMMAND:
SCHEDULE_FORCED_HOST_SVC_CHECKS;ilom-cp2;1179271459
[1179271463] HOST ALERT: devstack01;DOWN;SOFT;7;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271466] HOST ALERT: devstack01;DOWN;SOFT;8;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271469] HOST ALERT: devstack01;DOWN;SOFT;9;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271469] EXTERNAL COMMAND:
SCHEDULE_FORCED_HOST_SVC_CHECKS;ilom-cp3;1179271467
[1179271472] HOST ALERT: devstack01;DOWN;HARD;10;CRITICAL - Host Unreachable
(10.0.0.160)
[1179271472] HOST NOTIFICATION:
lbeavers-pager;devstack01;DOWN;host-notify-by-epager;CRITICAL - Host
Unreachable (10.0.0.160)
[1179271472] HOST NOTIFICATION:
lbeavers;devstack01;DOWN;host-notify-by-email;CRITICAL - Host Unreachable (
10.0.0.160)
[1179271472] HOST NOTIFICATION:
gpoly-pager;devstack01;DOWN;host-notify-by-epager;CRITICAL - Host
Unreachable (10.0.0.160)
[1179271472] HOST NOTIFICATION:
gpoly;devstack01;DOWN;host-notify-by-email;CRITICAL - Host Unreachable (
10.0.0.160)
[1179271472] SERVICE ALERT: devstack01;ping;CRITICAL;HARD;1;CRITICAL - Host
Unreachable (10.0.0.160)

-------------------------------------------
As you can see, host notifications are being sent out

Today's log:

---------------------------------------------------
[1179337965] EXTERNAL COMMAND:
SCHEDULE_FORCED_SVC_CHECK;contactpoint3;var_disk;1179337960
[1179337974] SERVICE ALERT: contactpoint3;var_disk;UNKNOWN;SOFT;1;SNMP
problem - No data received from host
[1179338034] SERVICE ALERT: contactpoint3;var_disk;UNKNOWN;SOFT;2;SNMP
problem - No data received from host
[1179338094] SERVICE ALERT: contactpoint3;var_disk;UNKNOWN;HARD;3;SNMP
problem - No data received from host
[1179338408] EXTERNAL COMMAND:
SCHEDULE_FORCED_SVC_CHECK;contactpoint3;sendmail_check;1179338407
[1179338414] SERVICE ALERT:
contactpoint3;sendmail_check;CRITICAL;SOFT;1;sendmail Processes CRITICAL -
*0*
[1179338474] SERVICE ALERT:
contactpoint3;sendmail_check;CRITICAL;SOFT;2;sendmail Processes CRITICAL -
*0*
[1179338484] EXTERNAL COMMAND:
SCHEDULE_FORCED_SVC_CHECK;contactpoint3;sendmail_check;1179338481
[1179338494] SERVICE ALERT:
contactpoint3;sendmail_check;CRITICAL;HARD;3;sendmail Processes CRITICAL -
*0*
[1179338604] Warning: The results of service 'ping' on host 'contactpoint4'
are stale by 45 seconds (threshold=615 seconds).  I'm forcing an immediate
check of the service.
[1179338604] Warning: The results of service 'sendmail_check' on host
'contactpoint4' are stale by 45 seconds (threshold=615 seconds).  I'm
forcing an immediate check of the service.
[1179338604] Warning: The results of service 'ping' on host 'contactpoint5'
are stale by 45 seconds (threshold=61

--------------------------

As can be seen, it went thru the three criticals, went to CRIT HARD, but no
NOTIFICATIONS were sent, it just continued looking at other services.



I've got
enable_notifications=1 set in nagios.cfg
In services.cfg, I've got:
notification_period             24x7
notifications_enabled           1       ; Service notifications are enabled
notification_interval           15      ; Default interval - change only if
needed in the service config

and the web frontend reports ALL notifications enabled.


 Monitoring Features Flap Detection Notifications Event Handlers Active
Checks Passive Checks   [image: Flap Detection
Enabled]<http://nagios.quepasa.com/nagios/cgi-bin/cmd.cgi?cmd_typ=62>
   All Services Enabled No Services Flapping All Hosts Enabled No Hosts
Flapping     [image: Notifications
Enabled]<http://nagios.quepasa.com/nagios/cgi-bin/cmd.cgi?cmd_typ=11>
   All Services Enabled All Hosts Enabled     [image: Event Handlers
Enabled] <http://nagios.quepasa.com/nagios/cgi-bin/cmd.cgi?cmd_typ=42>    All
Services Enabled All Hosts Enabled     [image: Active Checks
Enabled]<http://nagios.quepasa.com/nagios/cgi-bin/extinfo.cgi?type=0>
   All Services Enabled All Hosts Enabled     [image: Passive Checks
Enabled] <http://nagios.quepasa.com/nagios/cgi-bin/extinfo.cgi?type=0>    All
Services Enabled All Hosts Enabled
Any idea where else to check???? I've deleted my retention file and
restarted nagios as well
Pulling my hair out here

G.~


-- 
Gary Every
"Pay it Forward!"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-lists.org/archive/users/attachments/20070516/cb08f10b/attachment.html>
-------------- next part --------------
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
-------------- next part --------------
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null