<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Dear Alessandro,<div><br></div><div>You are more than likely eating up the cpu and memory with the memcpy's executed by each fork of your check_nrpe and check_icmp services. You can prove this out to yourself by using top to observe the behaviour of the nagios processes. I would also suggest that there is nothing else eating up CPU and memory on your nagios server box and keep the box dedicated. Running top will show if there is resource contention on your monitoring server. Keep in mind that check_nrpe is amongst the slowest possible commands nagios can execute because it has to wait for whatever timeout period you entered in your client nrpe.cfg for the nrpe daemon to respond. This can take seconds in some cases. A much more scalable solution is to enable passive checks (using nsca/send_nsca) on some or all of your clients)</div><div><br></div><div>I would suggest the following things (from the nagios performance tuning guide):</div><div><br></div><div><div># <b>Check service latencies</b> to determine best value for maximum concurrent checks. Nagios can restrict the number of maximum concurrently executing service checks to the value you specify with the max_concurrent_checks option. This is good because it gives you some control over how much load Nagios will impose on your monitoring host, but it can also slow things down. If you are seeing high latency values (> 10 or 15 seconds) for the majority of your service checks (via the extinfo CGI), you are probably starving Nagios of the checks it needs. That's not Nagios's fault - its yours. Under ideal conditions, all service checks would have a latency of 0, meaning they were executed at the exact time that they were scheduled to be executed. However, it is normal for some checks to have small latency values. I would recommend taking the minimum number of maximum concurrent checks reported when running Nagios with the -s command line argument and doubling it. Keep increasing it until the average check latency for your services is fairly low. </div><div><br></div><div><div># <b>Optimize host check commands</b>. If you're checking host states using the check_ping plugin you'll find that host checks will be performed much faster if you break up the checks. Instead of specifying a max_attempts value of 1 in the host definition and having the check_ping plugin send 10 ICMP packets to the host, it would be much faster to set the max_attempts value to 10 and only send out 1 ICMP packet each time. This is due to the fact that Nagios can often determine the status of a host after executing the plugin once, so you want to make the first check as fast as possible. This method does have its pitfalls in some situations (i.e. hosts that are slow to respond may be assumed to be down), but you'll see faster host checks if you use it. Another option would be to use a faster plugin (i.e. check_fping) as the host_check_command instead of check_ping.</div><div><br></div><div># <b>Schedule regular host checks.</b> Scheduling regular checks of hosts can actually help performance in Nagios. This is due to the way the cached check logic works (see below). Prior to Nagios 3, regularly scheduled host checks used to result in a big performance hit. This is no longer the case, as host checks are run in parallel - just like service checks. To schedule regular checks of a host, set the check_interval directive in the host definition to something greater than 0.</div></div><div><br></div><div># <b>Enable cached host checks</b>. Beginning in Nagios 3, on-demand host checks can benefit from caching. On-demand host checks are performed whenever Nagios detects a service state change. These on-demand checks are executed because Nagios wants to know if the host associated with the service changed state. By enabling cached host checks, you can optimize performance. In some cases, Nagios may be able to used the old/cached state of the host, rather than actually executing a host check command. This can speed things up and reduce load on monitoring server. In order for cached checks to be effective, you need to schedule regular checks of your hosts (see above). More information on cached checks can be found here.</div></div><div><br></div><div>For more, see:</div><div><br></div><div><i><a href="http://nagios.sourceforge.net/docs/3_0/tuning.html">http://nagios.sourceforge.net/docs/3_0/tuning.html</a></i></div><div><br></div><div>If none of this works, you may have to use passive checks or multiple nagios instances to drop your latency.</div><div><br></div><div>Bon Chance!</div><div>Daniel.</div><div><br></div><div><div><div>On Feb 17, 2009, at 8:41 AM, Alessandro Ren wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On 2/17/2009 1:32 PM, D. Emmanuel Feinsmith wrote:<br><br> Answers bellow,<br><blockquote type="cite">Alessandro,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">1. what is the breakdown between passive and active checks? For<br></blockquote><blockquote type="cite">passive checks, there are many ways to increase the # of services<br></blockquote><blockquote type="cite">through bypassing the command pipe (which nsca writes to). With<br></blockquote><blockquote type="cite">passive checks done in this way I've gone to 50,000 services with<br></blockquote><blockquote type="cite">under 10 second latency.<br></blockquote><blockquote type="cite"><br></blockquote> All active checks, no passive.<br><br><blockquote type="cite">2. how many of those services are check_icmp or check_ping? If there<br></blockquote><blockquote type="cite">is a good number of those, you can use fping to reduce the # of fork/<br></blockquote><blockquote type="cite">exec's that nagios has to perform, which is a major area of resource<br></blockquote><blockquote type="cite">utilization within the nagios server.<br></blockquote><blockquote type="cite"><br></blockquote> Less than 5% are ping checks and we use check_icmp for all those. <br>Most checks are check_nrpe,.<br><br><blockquote type="cite">3. Are you using a performance data handler or OCSP? If so, you might<br></blockquote><blockquote type="cite">either find a way to get rid of these entirely, or be sure you are<br></blockquote><blockquote type="cite">using file based performance handling at the very minimum.<br></blockquote><blockquote type="cite"><br></blockquote> I am using perfparse to write to mysql. Disabling it has no effect <br>in the latency.<br><br><blockquote type="cite">The key to nagios scalability and latency reduction is to educe the #<br></blockquote><blockquote type="cite">of fork/exec's to the smallest amount possible and keep away from the<br></blockquote><blockquote type="cite">command pipe as much as you can if you are passive-check heavy. If you<br></blockquote><blockquote type="cite">are using all active checks, then you can balance the load between<br></blockquote><blockquote type="cite">active and passive checks and thereby gain some speed.<br></blockquote><blockquote type="cite"><br></blockquote><br> In my other nagios with just 2600 services, we see around 200 <br>nagios processes running in average, in the 11600 services system, the <br>average is 30 processes, it seems that the event loop in lagging, is is <br>not starting enough processes, thus raising the latency.<br><br> Thank you Daniel.<br><blockquote type="cite">Daniel.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Feb 17, 2009, at 8:17 AM, Alessandro Ren wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"> Hello,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I have a nagios system running with 427 hosts and 11160 services and<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">since I reached 8000 services, I am having problems with the latency<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">beeing around 100s and 200s.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> use_large_installation_tweaks is enabled, max_concurrent_checks<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">have<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">been tested with 0 and higher values and I have tested this setup in<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">two<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">different HWs, a dual core with 4GB RAM 32 bits a a Dual Xeon Dual<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">core<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">64bits with 8GB of RAM. We are using REdHat enterprise 5.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Also reaper is already at 2s, host checks with cache horizon are<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">enabled with a max retry of 3, all services check every 5min.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> I have no service dependency set up.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> I've noticed that nagios is not spawning too many processes as<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">another nagios I have running which has far less servicexs and it<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">seems<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">that the event loop if lagging behing, in my debugs.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Any ideas what could I do to fix that? Have I reached a limit in<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">nagios pooler code?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Tks.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-- <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Alessandro Ren<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="http://www.opservices.com.br">http://www.opservices.com.br</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:alessandro.ren@opservices.com.br">alessandro.ren@opservices.com.br</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">------------------------------------------------------------------------------<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Open Source Business Conference (OSBC), March 24-25, 2009, San<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Francisco, CA<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-OSBC tackles the biggest issue in open source: Open Sourcing the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Enterprise<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-Strategies to boost innovation and cut costs with open source<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">participation<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-Receive a $600 discount off the registration fee with the source<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">code: SFAD<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="http://p.sf.net/sfu/XcvMzF8H">http://p.sf.net/sfu/XcvMzF8H</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Nagios-devel mailing list<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:Nagios-devel@lists.sourceforge.net">Nagios-devel@lists.sourceforge.net</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="https://lists.sourceforge.net/lists/listinfo/nagios-devel">https://lists.sourceforge.net/lists/listinfo/nagios-devel</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">------------------------------------------------------------------------------<br></blockquote><blockquote type="cite">Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA<br></blockquote><blockquote type="cite">-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise<br></blockquote><blockquote type="cite">-Strategies to boost innovation and cut costs with open source participation<br></blockquote><blockquote type="cite">-Receive a $600 discount off the registration fee with the source code: SFAD<br></blockquote><blockquote type="cite"><a href="http://p.sf.net/sfu/XcvMzF8H">http://p.sf.net/sfu/XcvMzF8H</a><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Nagios-devel mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Nagios-devel@lists.sourceforge.net">Nagios-devel@lists.sourceforge.net</a><br></blockquote><blockquote type="cite"><a href="https://lists.sourceforge.net/lists/listinfo/nagios-devel">https://lists.sourceforge.net/lists/listinfo/nagios-devel</a><br></blockquote><blockquote type="cite"><br></blockquote><br>------------------------------------------------------------------------------<br>Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA<br>-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise<br>-Strategies to boost innovation and cut costs with open source participation<br>-Receive a $600 discount off the registration fee with the source code: SFAD<br><a href="http://p.sf.net/sfu/XcvMzF8H">http://p.sf.net/sfu/XcvMzF8H</a><br>_______________________________________________<br>Nagios-devel mailing list<br>Nagios-devel@lists.sourceforge.net<br>https://lists.sourceforge.net/lists/listinfo/nagios-devel<br></div></blockquote></div><br></div></body></html>