Thank you for the reply.<br><br>My server has Intel(R) Xeon(TM) CPU 2.80GHz (1 core) and 3Gb of RAM.<br>I have 1300 checks , 320 hosts and mk_livestatus . CPU load is about 55-60%.<br>Also, one client is always connected to load nagvis maps and specific checks state (refresh rate is 30 secs).<br>
<br>Checks are performed in various intervals (ranging from 1 minute to 10 minutes).<br>All plugins are in perl; I think a lot of effort is required to convert them to C (compiled).<br>use_large_installation_tweaks is already enabled (use_large_installation_tweaks=1)<br>
<br>I am not using mysql; whole configuration is text based.<br><br>Also this is the report from nagiostats: <br><br>Nagios Stats 3.4.1<br>Copyright (c) 2003-2008 Ethan Galstad (<a href="http://www.nagios.org">www.nagios.org</a>)<br>
Last Modified: 05-11-2012<br>License: GPL<br><br>CURRENT STATUS DATA<br>------------------------------------------------------<br>Status File: /var/log/nagios/status.dat<br>Status File Age: 0d 0h 0m 6s<br>
Status File Version: 3.4.1<br><br>Program Running Time: 0d 2h 24m 37s<br>Nagios PID: 11485<br>Used/High/Total Command Buffers: 0 / 0 / 4096<br><br>Total Services: 1342<br>
Services Checked: 1342<br>Services Scheduled: 1341<br>Services Actively Checked: 1341<br>Services Passively Checked: 1<br>Total Service State Change: 0.000 / 39.540 / 0.076 %<br>
Active Service Latency: 0.005 / 0.717 / 0.203 sec<br>Active Service Execution Time: 0.013 / 20.340 / 2.241 sec<br>Active Service State Change: 0.000 / 11.580 / 0.047 %<br>Active Services Last 1/5/15/60 min: 199 / 1002 / 1294 / 1328<br>
Passive Service Latency: 34.021 / 34.021 / 34.021 sec<br>Passive Service State Change: 39.540 / 39.540 / 39.540 %<br>Passive Services Last 1/5/15/60 min: 0 / 0 / 0 / 0<br>Services Ok/Warn/Unk/Crit: 1278 / 19 / 18 / 27<br>
Services Flapping: 1<br>Services In Downtime: 0<br><br>Total Hosts: 318<br>Hosts Checked: 318<br>Hosts Scheduled: 318<br>
Hosts Actively Checked: 318<br>Host Passively Checked: 0<br>Total Host State Change: 0.000 / 0.000 / 0.000 %<br>Active Host Latency: 0.017 / 0.446 / 0.195 sec<br>
Active Host Execution Time: 0.019 / 30.026 / 5.615 sec<br>Active Host State Change: 0.000 / 0.000 / 0.000 %<br>Active Hosts Last 1/5/15/60 min: 43 / 285 / 318 / 318<br>Passive Host Latency: 0.000 / 0.000 / 0.000 sec<br>
Passive Host State Change: 0.000 / 0.000 / 0.000 %<br>Passive Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0<br>Hosts Up/Down/Unreach: 298 / 18 / 2<br>Hosts Flapping: 0<br>
Hosts In Downtime: 0<br><br>Active Host Checks Last 1/5/15 min: 51 / 319 / 986<br> Scheduled: 46 / 294 / 915<br> On-demand: 5 / 25 / 71<br> Parallel: 46 / 295 / 921<br>
Serial: 0 / 0 / 0<br> Cached: 5 / 24 / 66<br>Passive Host Checks Last 1/5/15 min: 0 / 0 / 0<br>Active Service Checks Last 1/5/15 min: 238 / 1048 / 3138<br>
Scheduled: 238 / 1048 / 3138<br> On-demand: 0 / 0 / 0<br> Cached: 0 / 0 / 0<br>Passive Service Checks Last 1/5/15 min: 0 / 0 / 0<br><br>
External Commands Last 1/5/15 min: 0 / 0 / 0<br><br>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br><br>I think the above statistics are ok; I want to use a second server (and move active checks) to keep load under 60% (or even lower that 40%) while checks increase.<br>
<br>Thank you<br><br><br><br><br><br><br><div class="gmail_quote">2012/6/12 Giorgio Zarrelli <span dir="ltr"><<a href="mailto:zarrelli@linux.it" target="_blank">zarrelli@linux.it</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF"><div>Hi,</div><div><br></div><div>You are right., open files IS a major concern I forgot to mention. A quick and dirty method to solve it is to raise the number of open files putting ulimit command folllowed by a high value in The Nagios startup script.</div>
<div><br></div><div>ulimit -a will tell The current system wirde ulimit value.</div><div><br></div><div>Lucky you, ssd disks are a good improvement!<br><br>Ciao,<div><br></div><div>Giorgio</div></div><div><br>Il giorno 12/giu/2012, alle ore 03:59, Ian Orszaczki <<a href="mailto:ian@griggle.net" target="_blank">ian@griggle.net</a>> ha scritto:<br>
<br></div><div><div class="h5"><div></div><blockquote type="cite"><div><div><br></div>Great advice. Funny you should mention status.dat in ramdisk as we have hit a hiccup this morning which has meant we have lost comments and downtimes.<div>
<br></div><div>We had moved status.dat to a ramdisk as recommended for large installations (we monitoring 3390 hosts with 18748 services from one server, latencies below 2 secs and load under 2) but after running out of open files the status.dat was zero'd.</div>
<div><br></div><div><br></div><div>As an extreme hack I ran a quick script across the output of -</div><div># grep EXTERNAL nagios.log | grep ACK | cut -c57- > /tmp/acks.txt</div><div><br></div><div>Script -</div><div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">#!/bin/sh</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important"># This is a sample shell script showing how you can submit the ACKNOWLEDGE_HOST_PROBLEM command</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important"># to Nagios. Adjust variables to fit your environment as necessary.</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">now=`date +%s`</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">commandfile=</code><code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,51,102)!important">'/app/nagios/var/rw/nagios.cmd'</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">cat /tmp/acks.txt | </code><code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-weight:bold!important;font-size:1em!important;min-height:inherit!important;color:rgb(51,102,153)!important">while</code> <code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">read line</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-weight:bold!important;font-size:1em!important;min-height:inherit!important;color:rgb(51,102,153)!important">do</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important"> </code><code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">echo $line</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important"> </code><code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">/usr/bin/printf </code><code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,51,102)!important">"[%lu] $line\n"</code> <code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">$now > $commandfile</code></div>
<div style="border-bottom-width:0px!important;text-align:left;border-style:initial!important;float:none!important;line-height:15px;margin-right:0px!important;border-top-right-radius:0px!important;font-size:13px;white-space:pre-wrap;border-right-width:0px!important;border-left-width:0px!important;background-image:none!important;border-bottom-left-radius:0px!important;padding-left:0em!important;padding-right:1em!important;padding-top:0px!important;outline-width:0px!important;width:auto!important;border-top-left-radius:0px!important;border-color:initial!important;margin-bottom:0px!important;min-height:inherit!important;padding-bottom:0px!important;border-top-width:0px!important;vertical-align:baseline!important;outline-style:initial!important;color:rgb(51,51,51);outline-color:initial!important;border-bottom-right-radius:0px!important;margin-left:0px!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace;margin-top:0px!important">
<code style="border-top-left-radius:0px!important;border-top-right-radius:0px!important;border-bottom-right-radius:0px!important;border-bottom-left-radius:0px!important;background-image:none!important;background-color:initial!important;border-top-width:0px!important;border-right-width:0px!important;border-bottom-width:0px!important;border-left-width:0px!important;border-style:initial!important;border-color:initial!important;float:none!important;min-height:auto!important;line-height:1.2em!important;margin-top:0px!important;margin-right:0px!important;margin-bottom:0px!important;margin-left:0px!important;outline-width:0px!important;outline-style:initial!important;outline-color:initial!important;padding-top:0px!important;padding-right:0px!important;padding-bottom:0px!important;padding-left:0px!important;vertical-align:baseline!important;width:auto!important;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important;font-size:1em!important;min-height:inherit!important;color:rgb(0,0,0)!important">done</code></div>
</div><div><br>Therefore I am going to move status.dat back onto the localdisk (luckily SSD drives) so that we can at least restore from a recent backup. I will probably also create valid copy, along with retention.dat, every hour to enable quick recovery. And yes, I have increased the process and open files limits for the nagios user.</div>
<div><br></div><div>Am I missing anything obvious ></div><div><br><br><div class="gmail_quote">On Tue, Jun 12, 2012 at 5:40 AM, Giorgio Zarrelli <span dir="ltr"><<a href="mailto:zarrelli@linux.it" target="_blank">zarrelli@linux.it</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I suggest to review your installation. Try with the large installation<br>
tweaks <a href="http://nagios.sourceforge.net/docs/3_0/largeinstalltweaks.html" target="_blank">http://nagios.sourceforge.net/docs/3_0/largeinstalltweaks.html</a>.<br>
<br>
Then, check whether you need all your checks at 5 mins or you can move<br>
some of them to 10 mins pace.<br>
<br>
Then, review your check plugins: Perl plugins eat more memory and CPU<br>
cycles then C compiled checks. If they support EPN<br>
<a href="http://nagios.sourceforge.net/docs/3_0/embeddedperl.html" target="_blank">http://nagios.sourceforge.net/docs/3_0/embeddedperl.html</a>, use it, it makes<br>
your plugin faster and lighter.<br>
<br>
Then, check your checks. Some checks return data slower then others. Let's<br>
say, SNMP checks are not lightning fast.<br>
<br>
Then, check your graphs. Graphing perfdata takes CPU cycles and uses<br>
memory. Do you need all your graphs?<br>
<br>
Then, get rid of NDOUtils. They are chocking all the way, not efficient,<br>
clumsy, old and heavy. If you want to store your data in MySQL, use Merlin<br>
instead.<br>
<br>
Anyway, did you tune your MySQL? Is it causing too much I/O? Is it<br>
munching too much RAM or CPU cycles?<br>
<br>
Did you tune your Apache or http server? Does it cope with your needs? Is<br>
it munching too much RAM or CPU cycles?<br>
<br>
If you want live infos about your hosts and services, let's say to use<br>
with Navis, grab MKlive: it's blazing fast and gives you access to the<br>
core Nagios process.<br>
<br>
Are you using a virtualized environment? If so, remember that I/O layer in<br>
virtualized environments has a poor performance, use fast and real disks<br>
and your I/O will drop dramatically.<br>
<br>
Try to move status.dat to /dev/shm. The latter is a ram disk ready to use<br>
and writing in ram is always faster then writing on disk.<br>
<br>
Avoid logging too much, it increases I/O and takes CPU and RAM.<br>
<br>
What iotop and iostat are telling you?<br>
<br>
What do you see in top or htop?<br>
<br>
If you can or wish, compile all from sources, it will go faster on your<br>
system.<br>
<br>
You can use passive checks with NSCA or NRDP to reduce load, even though I<br>
do not like them a lot.<br>
<br>
These are just few ideas that came to my mind.<br>
<br>
<br>
Let's talk about sharing load.<br>
<br>
You can use different methods:<br>
<br>
Merlin<br>
(<a href="http://www.op5.org/community/plugin-inventory/op5-projects/merlin" target="_blank">http://www.op5.org/community/plugin-inventory/op5-projects/merlin</a>): gives<br>
you loadbalancing and redundancy. I use it for Ninja, never used for load<br>
balancing and redundancy.<br>
<br>
DNX (<a href="http://dnx.sourceforge.net/" target="_blank">http://dnx.sourceforge.net/</a>): Something new, it's gaining momentum,<br>
good to offload the checks. Worth to give a try.<br>
<br>
Mod_gearman (<a href="http://labs.consol.de/lang/de/nagios/mod-gearman/" target="_blank">http://labs.consol.de/lang/de/nagios/mod-gearman/</a>): Love at<br>
first site :-) Easy, powerful, load balancing and fault tolerant. Compile<br>
gearmand with memcached support and all the result checks will go directly<br>
to ram, avoiding I/O on disk. It's really simple to setup, if one of the<br>
workers go down, the others will share its work. Be careful: security is a<br>
problem, there is not a good auth system, but using a VPN will solve the<br>
problem. Efficient, I use a virtual machine with 2 cores and 2 gb of ram<br>
to make about 5K checks. And the load is not a concern. You need more<br>
horse power? Add a worker. You have some checks timing out due to poor<br>
connections to the targets? Put a worker close to the target, but be<br>
careful, the timing, let's say the rta of a ping, will be from the worker<br>
perspective.<br>
<br>
Well, hope it helps.<br>
<div><div><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
------------------------------------------------------------------------------<br>
Live Security Virtual Conference<br>
Exclusive live event will cover all the ways today's security and<br>
threat landscape has changed and how IT managers can respond. Discussions<br>
will include endpoint security, mobile security and the latest in malware<br>
threats. <a href="http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/" target="_blank">http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/</a><br>
_______________________________________________<br>
Nagios-users mailing list<br>
<a href="mailto:Nagios-users@lists.sourceforge.net" target="_blank">Nagios-users@lists.sourceforge.net</a><br>
<a href="https://lists.sourceforge.net/lists/listinfo/nagios-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/nagios-users</a><br>
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.<br>
::: Messages without supporting info will risk being sent to /dev/null<br>
</div></div></blockquote></div><br></div>
</div></blockquote><blockquote type="cite"><div><span>------------------------------------------------------------------------------</span><br><span>Live Security Virtual Conference</span><br><span>Exclusive live event will cover all the ways today's security and </span><br>
<span>threat landscape has changed and how IT managers can respond. Discussions </span><br><span>will include endpoint security, mobile security and the latest in malware </span><br><span>threats. <a href="http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/" target="_blank">http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/</a></span></div>
</blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Nagios-users mailing list</span><br><span><a href="mailto:Nagios-users@lists.sourceforge.net" target="_blank">Nagios-users@lists.sourceforge.net</a></span><br>
<span><a href="https://lists.sourceforge.net/lists/listinfo/nagios-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/nagios-users</a></span><br><span>::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. </span><br>
<span>::: Messages without supporting info will risk being sent to /dev/null</span></div></blockquote></div></div></div><br>------------------------------------------------------------------------------<br>
Live Security Virtual Conference<br>
Exclusive live event will cover all the ways today's security and<br>
threat landscape has changed and how IT managers can respond. Discussions<br>
will include endpoint security, mobile security and the latest in malware<br>
threats. <a href="http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/" target="_blank">http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/</a><br>_______________________________________________<br>
Nagios-users mailing list<br>
<a href="mailto:Nagios-users@lists.sourceforge.net">Nagios-users@lists.sourceforge.net</a><br>
<a href="https://lists.sourceforge.net/lists/listinfo/nagios-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/nagios-users</a><br>
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.<br>
::: Messages without supporting info will risk being sent to /dev/null<br></blockquote></div><br>