<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; ">Hi Mahesh,<DIV><BR><DIV><DIV>On 19 Dec 2006, at 00:42, Mahesh Kunjal wrote:</DIV><BR class="Apple-interchange-newline"><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Here is what we did to resolve.</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">1. Edit the include/nagios.h.in</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">change</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">#define COMMAND_BUFFER_SLOTS 1024</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">to</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">#define COMMAND_BUFFER_SLOTS 60000</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">And change</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">#define SERVICE_BUFFER_SLOTS 1024</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">to</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">#define SERVICE_BUFFER_SLOTS 60000</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV></BLOCKQUOTE></DIV><DIV><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><DIV><BR class="khtml-block-placeholder"></DIV><DIV>I was intrigued by this as we have a performance issue, but not with the same symptoms. Our problem is that NSCA processes increase when the nagios server is under load. They appear to be blocking on writing to the command pipe. Switching NSCA to single daemon mitigates the problem (slaves will timeout their passive results), but we wanted to know where any slow downs could be.</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>From your findings, we've created a performance static patch, attached. This collects the maximum and current values for the command and service buffer slots and is then written to status.dat (by default every 10 seconds). What I found with a fake slave sending 128 results every 5 seconds was that the maximum values were fairly low (under 100), but when I put the server under load, the maximum_command_buffer_items shot up to 1969 and the maximum_service_buffer_items shot up to 2156 (had changed from defaults to your 60000).</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>This could show if the buffer is filled at various points or if there is not enough data ready for Nagios to process further down the chain.</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>I'd be interested in figures from other systems.</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Warning: the patch is not thread safe, so there is no guarantees that the statistic data will not be corrupted (but should not affect usual Nagios operation). Applies onto Nagios 2.5. Tested on Debian with 2.6 kernel.</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Ton</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV><A href="http://www.altinity.com">http://www.altinity.com</A></DIV><DIV>T: +44 (0)870 787 9243</DIV><DIV>F: +44 (0)845 280 1725</DIV><DIV>Skype: tonvoon</DIV><DIV><BR class="khtml-block-placeholder"></DIV></SPAN></SPAN></SPAN></SPAN></SPAN></SPAN></DIV></DIV></BODY></HTML>