<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Have you tried running in debug mode?<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><br> Dan<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Frost, Mark {BIS} [mailto:mark.frost1@pepsico.com] <br> <b>Sent:</b> Saturday, February 18, 2012 11:48 AM<br> <b>To:</b> Nagios Users List<br> <b>Subject:</b> [Nagios-users] Have we reached some kind of Nagios limit?<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">A couple of days ago, I ran into a problem I’ve never seen before. We run a single large instance with mostly very heterogeneous checks and host types. One particular group of Windows hosts, however, are all quite similar and they, like most of our other checks rely on the use of templates. I needed to add 10 more hosts of this particular type and typically all I have to do is just define the hosts and the service checks happen automatically as the host templates include them in a group that includes all the relevant checks.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">I added maybe 5 of these new hosts, ran the pre-flight check and restarted. After the restart I started noticing that our failing service checks (for all services) went from around 260 to over 4K. All of those new failing checks were only on hosts of this same type (that particular application on Windows servers I mentioned above which is also what these new hosts were part of) and they were all reporting the same failure condition:<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">(Return code of 127 is out of bounds - plugin may be missing)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Now ordinarily this would indicate a client-side issue, but there isn’t one. I can validate that by running check_nrpe manually against any of these hosts. I could imagine a typo that would cause this, particular against other existing hosts that had not been touched, but I double-checked and did not find one (I was just adding host definitions to this group – nothing else).<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">I cloned this environment and went to play with it in a non-production instance that was identical to the production Nagios instance except for a slight newer version of Merlin in the backend (1.1.14 for the non-prod instance, 1.1.13 something for the production one), but both used the same Nagios 3.3.1 + downtime locking patches. I was able to reproduce the situation and after a couple of days of trial and error I’ve still not been able to completely isolate the issue, but I’ve determined that<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <ul type="disc"> <li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> <span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">it’s not got anything to do with the mk-livestatus module (turned it off, turned it back on), but it’s been very helpful in figuring out which of the 13K+ services and 1200+ hosts are impacted<o:p></o:p></span></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> <span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">it doesn’t seem to be about adding random hosts and services. I can add others and this doesn’t happen<o:p></o:p></span></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> <span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">the host definition uses a template that puts the host in a hostgroup. Those hostgroups are then used to in service definitions (12-15 services, depending on which group). I had thought that perhaps if the hostgroup_name line of the service definition expanded to too many hosts that could be the problem. I broke the service definitions down into 2 definitions, one for each production hostgroup rather than combining them and that didn’t matter.<o:p></o:p></span></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> <span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">the service templates that the service definitions use for these hosts all add them to a common servicegroup. My current line of thinking leads me to believe it’s got something to do with this. With a particular test scenario I created where I create a new host, but exclude it from the hostgroup definitions and instead manually create service definitions for this host (I know this “one more host” is right on the cusp of this problem), I find that when I add it so the 4,331</span><sup><span style="font-size:7.5pt;font-family:"Calibri","sans-serif"">st</span></sup><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> service gets added to the servicegroup, the problem starts. If I remove that from that host’s service definition all the other hosts’ services recover. However, based on this thinking, if I just comment out the servicegroup add from the service template these hosts use, the problem should stop – it doesn’t.<o:p></o:p></span></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> <span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">the only affect services are on all of the hostgroup I’m changing. Other unrelated hosts and services are unaffected. There are 3 hostgroups: Production Appname Hosts 1, Production Appname Hosts 2, and All Appname Hosts which is obviously a combination of the two. All Appname Hosts is around 324 hosts.<o:p></o:p></span></li></ul> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">I’m not really sure what to try at this point. It does seem like I’ve hit some kind of internal limitation with Nagios, but I don’t know how to determine anything else about it beyond this. If I were able to completely isolate this to say, not adding anything to a single servicegroup, I could avoid that and continue adding hosts as we need it, but I have so far not been able to find such a workaround. If there is a limitation like this, it would of course, be nice for the pre-flight check to tell me that I can’t have more than X members of a servicegroup or something.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Other info:<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Nagios version: Nagios 3.3.1 with locking patches<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Merlin backend: 1.1.13+ (production), 1.1.14 (test)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">MK-Livestatus module 1.1.12p6 installed (uninstalled doesn’t impact)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">OS: SLES 11.1 Linux, 64-bit<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Memory: 12GB<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">CPU: 2x 2.4Ghz quad-core Xeon<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">What can I do?<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Thanks<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Mark<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p> </div> </div> </body> </html>