<div>Over the weekend I had a weird event happen in my monitoring set. I have a host that has an Infiniband interface, and we are running the IP protocol over Infiniband, while using the check_ping plugin to verify network connectivity. This weekend our Ethernet to Infiniband gateway went down and the check_ping plugin reported that state as CRITICAL, however the it didn't increment the check attempt value, and thus a notification wasn't sent out since the max check attempts threshold was never reached. I've included some of the output of /var/log/messages that indicates the problem. Has anyone else seen this issue? As a side note I've since changed my service check to use check_icmp instead of check_ping.</div> <div><br></div><div>Jan 28 17:12:12 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:13:12 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:14:12 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:15:12 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br><br></div><div> Jan 28 17:42:12 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:43:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:44:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:47:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:48:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:49:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 17:50:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br><br></div><div> Jan 28 18:31:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 18:32:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 18:33:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br><br></div><div> Jan 28 18:42:23 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 18:43:23 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 18:44:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br> Jan 28 18:45:22 monitor nagios: SERVICE ALERT: <a href="http://hamrock.chpc.utah.edu">hamrock.chpc.utah.edu</a>;IPv4 IPoIB Ping Check;CRITICAL;HARD;1;CRITICAL - Host Unreachable (172.21.10.170) <br><br></div><div><br></div>--<div><b><font face="arial, helvetica, sans-serif">Robert V. Bolton</font></b></div><div><font face="arial, helvetica, sans-serif">Email: <a href="mailto:robert@robertvbolton.com" target="_blank">robert@robertvbolton.com</a></font></div> <div>Web: <a href="http://robertvbolton.com" target="_blank">http://robertvbolton.com</a></div> <br>