<html><head></head><body bgcolor="#FFFFFF"><div>Hi,</div><div><br></div><div>You are right., open files IS a major concern I forgot to mention. A quick and dirty method to solve it is to raise the number of open files putting ulimit command folllowed by a high value in The Nagios startup script.</div><div><br></div><div>ulimit -a will tell The current system wirde ulimit value.</div><div><br></div><div>Lucky you, ssd disks are a good improvement!<br><br>Ciao,<div><br></div><div>Giorgio</div></div><div><br>Il giorno 12/giu/2012, alle ore 03:59, Ian Orszaczki <<a href="mailto:ian@griggle.net">ian@griggle.net</a>> ha scritto:<br><br></div><div></div><blockquote type="cite"><div><div><br></div>Great advice. Funny you should mention status.dat in ramdisk as we have hit a hiccup this morning which has meant we have lost comments and downtimes.<div><br></div><div>We had moved status.dat to a ramdisk as recommended for large installations (we monitoring 3390 hosts with 18748 services from one server, latencies below 2 secs and load under 2) but after running out of open files the status.dat was zero'd.</div> <div><br></div><div><br></div><div>As an extreme hack I ran a quick script across the output of -</div><div># grep EXTERNAL nagios.log | grep ACK | cut -c57- > /tmp/acks.txt</div><div><br></div><div>Script -</div><div> <div class="line number5 index4 alt2" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">#!/bin/sh</code></div> <div class="line number6 index5 alt1" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important"># This is a sample shell script showing how you can submit the ACKNOWLEDGE_HOST_PROBLEM command</code></div> <div class="line number7 index6 alt2" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important"># to Nagios. Adjust variables to fit your environment as necessary.</code></div> <div class="line number8 index7 alt1" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">now=`date +%s`</code></div> <div class="line number9 index8 alt2" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">commandfile=</code><code class="java string" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,51,102)!important">'/app/nagios/var/rw/nagios.cmd'</code></div> <div class="line number10 index9 alt1" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">cat /tmp/acks.txt | </code><code class="java keyword" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-weight:bold!important;font-size:1em!important;min-height:inherit!important;color:rgb(51,102,153)!important">while</code> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">read line</code></div> <div class="line number11 index10 alt2" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java keyword" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-weight:bold!important;font-size:1em!important;min-height:inherit!important;color:rgb(51,102,153)!important">do</code></div> <div class="line number12 index11 alt1" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java spaces" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important"> </code><code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">echo $line</code></div> <div class="line number13 index12 alt2" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java spaces" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important"> </code><code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">/usr/bin/printf </code><code class="java string" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,51,102)!important">"[%lu] $line\n"</code> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">$now > $commandfile</code></div> <div class="line number14 index13 alt1" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:rgb(255,255,255);border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:15px;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:1em!important;padding-bottom:0px!important;padding-left:0em!important;text-align:left;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;font-size:13px;min-height:inherit!important;white-space:pre-wrap;color:rgb(51,51,51)"> <code class="java plain" style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;overflow-x:visible!important;overflow-y:visible!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">done</code></div> </div><div><br>Therefore I am going to move status.dat back onto the localdisk (luckily SSD drives) so that we can at least restore from a recent backup. I will probably also create valid copy, along with retention.dat, every hour to enable quick recovery. And yes, I have increased the process and open files limits for the nagios user.</div> <div><br></div><div>Am I missing anything obvious ></div><div><br><br><div class="gmail_quote">On Tue, Jun 12, 2012 at 5:40 AM, Giorgio Zarrelli <span dir="ltr"><<a href="mailto:zarrelli@linux.it" target="_blank">zarrelli@linux.it</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br> <br> I suggest to review your installation. Try with the large installation<br> tweaks <a href="http://nagios.sourceforge.net/docs/3_0/largeinstalltweaks.html" target="_blank">http://nagios.sourceforge.net/docs/3_0/largeinstalltweaks.html</a>.<br> <br> Then, check whether you need all your checks at 5 mins or you can move<br> some of them to 10 mins pace.<br> <br> Then, review your check plugins: Perl plugins eat more memory and CPU<br> cycles then C compiled checks. If they support EPN<br> <a href="http://nagios.sourceforge.net/docs/3_0/embeddedperl.html" target="_blank">http://nagios.sourceforge.net/docs/3_0/embeddedperl.html</a>, use it, it makes<br> your plugin faster and lighter.<br> <br> Then, check your checks. Some checks return data slower then others. Let's<br> say, SNMP checks are not lightning fast.<br> <br> Then, check your graphs. Graphing perfdata takes CPU cycles and uses<br> memory. Do you need all your graphs?<br> <br> Then, get rid of NDOUtils. They are chocking all the way, not efficient,<br> clumsy, old and heavy. If you want to store your data in MySQL, use Merlin<br> instead.<br> <br> Anyway, did you tune your MySQL? Is it causing too much I/O? Is it<br> munching too much RAM or CPU cycles?<br> <br> Did you tune your Apache or http server? Does it cope with your needs? Is<br> it munching too much RAM or CPU cycles?<br> <br> If you want live infos about your hosts and services, let's say to use<br> with Navis, grab MKlive: it's blazing fast and gives you access to the<br> core Nagios process.<br> <br> Are you using a virtualized environment? If so, remember that I/O layer in<br> virtualized environments has a poor performance, use fast and real disks<br> and your I/O will drop dramatically.<br> <br> Try to move status.dat to /dev/shm. The latter is a ram disk ready to use<br> and writing in ram is always faster then writing on disk.<br> <br> Avoid logging too much, it increases I/O and takes CPU and RAM.<br> <br> What iotop and iostat are telling you?<br> <br> What do you see in top or htop?<br> <br> If you can or wish, compile all from sources, it will go faster on your<br> system.<br> <br> You can use passive checks with NSCA or NRDP to reduce load, even though I<br> do not like them a lot.<br> <br> These are just few ideas that came to my mind.<br> <br> <br> Let's talk about sharing load.<br> <br> You can use different methods:<br> <br> Merlin<br> (<a href="http://www.op5.org/community/plugin-inventory/op5-projects/merlin" target="_blank">http://www.op5.org/community/plugin-inventory/op5-projects/merlin</a>): gives<br> you loadbalancing and redundancy. I use it for Ninja, never used for load<br> balancing and redundancy.<br> <br> DNX (<a href="http://dnx.sourceforge.net/" target="_blank">http://dnx.sourceforge.net/</a>): Something new, it's gaining momentum,<br> good to offload the checks. Worth to give a try.<br> <br> Mod_gearman (<a href="http://labs.consol.de/lang/de/nagios/mod-gearman/" target="_blank">http://labs.consol.de/lang/de/nagios/mod-gearman/</a>): Love at<br> first site :-) Easy, powerful, load balancing and fault tolerant. Compile<br> gearmand with memcached support and all the result checks will go directly<br> to ram, avoiding I/O on disk. It's really simple to setup, if one of the<br> workers go down, the others will share its work. Be careful: security is a<br> problem, there is not a good auth system, but using a VPN will solve the<br> problem. Efficient, I use a virtual machine with 2 cores and 2 gb of ram<br> to make about 5K checks. And the load is not a concern. You need more<br> horse power? Add a worker. You have some checks timing out due to poor<br> connections to the targets? Put a worker close to the target, but be<br> careful, the timing, let's say the rta of a ping, will be from the worker<br> perspective.<br> <br> Well, hope it helps.<br> <div class="HOEnZb"><div class="h5"><br> <br> <br> <br> <br> <br> <br> <br> <br> <br> ------------------------------------------------------------------------------<br> Live Security Virtual Conference<br> Exclusive live event will cover all the ways today's security and<br> threat landscape has changed and how IT managers can respond. Discussions<br> will include endpoint security, mobile security and the latest in malware<br> threats. <a href="http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/" target="_blank">http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/</a><br> _______________________________________________<br> Nagios-users mailing list<br> <a href="mailto:Nagios-users@lists.sourceforge.net">Nagios-users@lists.sourceforge.net</a><br> <a href="https://lists.sourceforge.net/lists/listinfo/nagios-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/nagios-users</a><br> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.<br> ::: Messages without supporting info will risk being sent to /dev/null<br> </div></div></blockquote></div><br></div> </div></blockquote><blockquote type="cite"><div><span>------------------------------------------------------------------------------</span><br><span>Live Security Virtual Conference</span><br><span>Exclusive live event will cover all the ways today's security and </span><br><span>threat landscape has changed and how IT managers can respond. Discussions </span><br><span>will include endpoint security, mobile security and the latest in malware </span><br><span>threats. <a href="http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/">http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/</a></span></div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Nagios-users mailing list</span><br><span><a href="mailto:Nagios-users@lists.sourceforge.net">Nagios-users@lists.sourceforge.net</a></span><br><span><a href="https://lists.sourceforge.net/lists/listinfo/nagios-users">https://lists.sourceforge.net/lists/listinfo/nagios-users</a></span><br><span>::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. </span><br><span>::: Messages without supporting info will risk being sent to /dev/null</span></div></blockquote></body></html>