<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40"> <head> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 11 (filtered medium)"> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"/>  <style>  </style> </head> <body lang=EN-US link=blue vlink=purple> <div class=Section1> Hi,<o:p></o:p> <o:p> </o:p> We have several nagios servers doing a total of about 1300 service checks and 300 host checks using Nagios 1.2 and Nagios plugins 1.3.1.<o:p></o:p> <o:p> </o:p> Unfortunately something a little annoying keeps happening, not to mention strange:<o:p></o:p> <o:p> </o:p> Nagios keeps sending HOST DOWN alerts when our hosts are not down. For example we do a ping check and HTTP-QA check for a website. Nagios will send a HOST DOWN alert, but at the same time, the ping check and HTTP check will both be just fine. Nagios will think the host is down for quite some time, but it keeps doing the ping and HTTP-QA checks anyway despite thinking the host is down. The only way I can make it think the host is back up is to totally restart Nagios, then it forgets that it thought the host was down (even with retain_state_information=1)<o:p></o:p> <o:p> </o:p> At first this happened to a couple load balanced websites and mail servers we had. Now this is happening to several other sites and mail servers that are not being load balanced. Every time it says a host is down like this, I can SSH into the Nagios server, and ping the exact hostname Nagios is using (either the FQDN or the IP depending on what Nagios is using in hosts.cfg for the given site), and the ping has no problems at all.<o:p></o:p> <o:p> </o:p> Just to give an example - we often get HOST DOWN warnings for "mail.ikea-usa.net" even though our SMTP and ping checks continue to be OK long after the "HOST DOWN" alert. We also have this problem with https://www.verepay.cc - but I think that's because we have ping turned off in our firewall for that site at the moment. Our load balanced anti-spam/virus mail servers located at scrubber.gsi-kc.com also suffer from this problem but I've never had any troubles pinging them. Just throwing out those examples incase anyone notices anything particularly wrong with them, since Nagios seems to like those sites the best for doing this odd "HOST DOWN" behavior.<o:p></o:p> <o:p> </o:p> Here's what I'll see in the nagios.log file:<o:p></o:p> <o:p> </o:p> [1087381552] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;1;Socket timeout after 10 seconds<o:p></o:p> [1087381562] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;2;Socket timeout after 10 seconds<o:p></o:p> [1087381572] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;3;Socket timeout after 10 seconds<o:p></o:p> [1087381582] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;4;Socket timeout after 10 seconds<o:p></o:p> [1087381592] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;5;Socket timeout after 10 seconds<o:p></o:p> [1087381602] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;6;Socket timeout after 10 seconds<o:p></o:p> [1087381612] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;7;Socket timeout after 10 seconds<o:p></o:p> [1087381622] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;8;Socket timeout after 10 seconds<o:p></o:p> [1087381632] HOST ALERT: scrubber.gsi-kc.com;DOWN;SOFT;9;Socket timeout after 10 seconds<o:p></o:p> [1087381642] HOST ALERT: scrubber.gsi-kc.com;DOWN;HARD;10;Socket timeout after 10 seconds<o:p></o:p> <o:p> </o:p> How can that be when I can do this at the same time?<o:p></o:p> <o:p> </o:p> [root@kgsinm05 var]# ping scrubber.gsi-kc.com<o:p></o:p> <st1:place w:st="on">PING</st1:place> scrubber.gsi-kc.com (205.247.222.244) 56(84) bytes of data.<o:p></o:p> 64 bytes from scrubber.gsi-kc.com (205.247.222.244): icmp_seq=1 ttl=240 time=28.3 ms<o:p></o:p> 64 bytes from scrubber.gsi-kc.com (205.247.222.244): icmp_seq=2 ttl=240 time=26.8 ms<o:p></o:p> <o:p> </o:p> Thanks!! <o:p></o:p> Jeremy<o:p></o:p> </div> </body> </html>