I haven't tried the fork script but I did try starting nagios under
strace as described but the main process which strace is tracing
appears to exit after spawning children. Here is the
last snippet so you know what I mean:<br>
<br>
[pid 29599] clone(Process 29644 attached<br>
<unfinished ...><br>
[pid 29643] <... clone resumed> child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0xb75e50c8) = 29644<br>
[tcb table full]<br>
[pid 29599] <... clone resumed> child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0xb75e50c8) = 29645<br>
[tcb table full]<br>
Process 29641 detached<br>
Process 29599 detached<br>
Process 29632 detached<br>
Process 29640 detached<br>
Process 29635 detached<br>
Process 29644 detached<br>
Process 29643 detached<br>
<br>
<br>
Any other ideas?<br><br><div><span class="gmail_quote">On 9/2/05, <b class="gmail_sendername">Fred</b> <<a href="mailto:f1216@yahoo.com">f1216@yahoo.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div>Just for fun, you might try creating the problem and see how many forks</div>
<div>you *can* get, for example:</div>
<div> </div>
<div>#!/usr/bin/perl</div>
<div><br>my $c=0;</div>
<div>do {<br> my $pid = fork();<br> if ($pid)<br> {<br> $c++;<br> print "\rchildcount $c ";<br> }<br> else<br> {<br> sleep(1);<br> exit(0);<br> }<br> } while 1;
<br></div>
<div>to create as many procs as you can and test your limit. You would</div>
<div>want to do this under the same environment as the <span id="st" name="st" class="st0">nagios</span> process</div>
<div>runs.</div>
<div> </div>
<div>They will all be kept defunct until the process exits (when you</div>
<div>hit the max processes you can create)</div>
<div> </div>
<div>The other thing you might try is to start <span id="st" name="st" class="st0">nagios</span> under</div>
<div>strace -f and output the data to a log. You can specify</div>
<div>just forks for strace, i.e., strace -f -e trace=process >/tmp/,log 2>&1 <span id="st" name="st" class="st0">nagios</span> ....</div>
<div> </div>
<div>That would give you a good handle on what is going on when the failure</div>
<div>occurs. Might slow <span id="st" name="st" class="st0">nagios</span> down a bit, but probably nothing significant.</div><span class="q">
<div> </div>
<div>-FredC</div>
<div> </div>
<div> </div>
<div> </div>
<div><br><br><b><i>Terry <<a href="mailto:td3201@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">td3201@gmail.com</a>></i></b> wrote:</div></span>
<blockquote style="border-left: 2px solid rgb(16, 16, 255); padding-left: 5px; margin-left: 5px;"><div><span class="e" id="q_106187bad34a08d8_3">I have a program that checks the logs by the minute and pages when the<br>fork errors occur, so we are responding within minutes. I have looked
<br>at the resources every time it happens and we have plenty of<br>resources. Is there a single plugin I can put into debugging mode so<br>that when this happens I get more information as to why it is giving<br>these errors? Here are a few facts:
<br>- the system is fine with memory all the time, never runs out (resident/paging)<br>- there are not an unusual amount of processes running, maybe around<br>200 at a time, but no where near the ulimit setting<br>- ulimit for the 'nagios' user matches that of root (unlimited). here
<br>is the ulimit:<br>core file size (blocks, -c) 0<br>data seg size (kbytes, -d) unlimited<br>file size (blocks, -f) unlimited<br>max locked memory (kbytes, -l) 4<br>max memory size (kbytes, -m) unlimited<br>open files (-n)
1024<br>pipe size (512 bytes, -p) 8<br>stack size (kbytes, -s) 10240<br>cpu time (seconds, -t) unlimited<br>max user processes (-u) 7168<br>virtual memory (kbytes, -v) unlimited<br><br>Thanks,<br>Terry<br><br><br><br></span>
</div><span class="q">On 9/1/05, Fred wrote:<br>> My guess would be to look at your resource utilization on your system,<br>> most likely causes for fork() to fail are no more process slots, out of<br>> memory, or past some kind of per-user (non-root) limit. When this
<br>> occurs look at your system logs, ps output and see if you have *lots*<br>> of processes hanging around. It could be that <span id="st" name="st" class="st0">nagios</span> has stopped reaping<br>> its children (or another unrelated process has sucked up the resources)
<br>> and you have simply pushed your system to the edge. It might be that you<br>> get to that situation and it backs off before you even notice it and you<br>> are left with <span id="st" name="st" class="st0">
nagios</span> having problems dealing with the aftermath.<br>> <br>> -FredC<br>>
<br></span><div><span class="e" id="q_106187bad34a08d8_5">> --- Terry wrote:<br>> <br>> > Hello,<br>> ><br>> > I have been having this issue for quite some time. For some unknown<br>> > reason, nagios stops performing checks with these errors:
<br>> ><br>> > [1125536952] Warning: The check of service 'PING' on host 'hostname'<br>> > could not be performed due to a fork() error. The check will be<br>> > rescheduled.<br>> ><br>> > All checks fail like this until nagios is restarted. When this
<br>> > problem is occuring I can run the service checks manually both as the<br>> > nagios user and as the root user. There are no resource problems that<br>> > I can see at the time. We do not appear to be hitting a limit with
<br>> > open files or anything like that either. The nagios mirrors the root<br>> > user in that area.<br>> ><br>> > What could be wrong?<br>> ><br>> > Thanks!<br>> ><br>>
><br>> > -------------------------------------------------------<br>> > SF.Net email is Sponsored by the Better Software Conference & EXPO<br>> > September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
<br>> > Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA<br>> > Security * Process Improvement & Measurement * <a href="http://www.sqe.com/bsce5sf" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
http://www.sqe.com/bsce5sf</a><br>> > _______________________________________________<br>> > Nagios-users mailing list<br>> > <a href="mailto:Nagios-users@lists.sourceforge.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
Nagios-users@lists.sourceforge.net</a><br>> > <a href="https://lists.sourceforge.net/lists/listinfo/nagios-users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">https://lists.sourceforge.net/lists/listinfo/nagios-users
</a><br>> > ::: Please include Nagios version, plugin version (-v) and OS when reporting<br>> > any issue.<br>> > ::: Messages without supporting info will risk being sent to /dev/null<br>> ><br>>
<br>> <br>> <br>> <br>> <br>><br></span></div></blockquote></div><br><br><p><img style="width: 59px;" height="45" width="56"><img height="40" width="47"><img style="width: 51px;" height="60" width="67"><img style="width: 51px;" height="56" width="68">
</p>
</blockquote></div><br>