Kjournald is needed for journalling on ext3 filesystems. Be glad you didn't manage to kill them.<br><br>To find something that is running many many instances, try this: "ps -ax -o cmd | sort | uniq -c | sort -n"<br>
<br>The output will be like so:<br> 3 [kjournald]<br> 3 [sh] <defunct><br> 5 -bash<br> 7 crond<br><br>The column on the left is the number of processes with that command line. I occasionally have 10,000 instances of nsca that simply need to be killed. Do let us know what you find!<br>
<br>--Rick<br><br><div class="gmail_quote">On Tue, Dec 7, 2010 at 9:25 AM, Kaplan, Andrew H. <span dir="ltr"><<a href="mailto:AHKAPLAN@partners.org">AHKAPLAN@partners.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div vlink="purple" link="blue" lang="EN-US">
<div dir="ltr" align="left"><span><font size="2" color="#0000ff" face="Arial">Hi there --</font></span></div>
<div dir="ltr" align="left"><span><font size="2" color="#0000ff" face="Arial"></font></span> </div>
<div dir="ltr" align="left"><span><font size="2" color="#0000ff" face="Arial">The output shown below shows the top processes on the
server:</font></span></div>
<div dir="ltr" align="left"><span><font size="2" color="#0000ff" face="Arial"></font></span> </div>
<div dir="ltr" align="left"><span><font size="2" color="#0000ff" face="Arial">439 processes: 438 sleeping, 1 running, 0 zombie, 0
stopped<br>CPU0 states: 19.0% user, 9.4% system, 0.0% nice, 71.0%
idle<br>CPU1 states: 20.1% user, 13.0% system, 0.0% nice, 66.3%
idle<br>CPU2 states: 27.1% user, 17.3% system, 0.0% nice, 55.0%
idle<br>Mem: 2064324K av, 2013820K used, 50504K
free, 0K shrd, 487764K buff<br>Swap:
2096472K av, 12436K used, 2084036K
free
976244K cached</font></span></div>
<div> </div>
<div dir="ltr" align="left"><span><font size="2" color="#0000ff" face="Arial"> PID USER PRI NI
SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND<br> 2398
root 15 0 1280 1280
824 R 1.9 0.0 0:00 top<br> 5648
root 22 0 1196 1196 1104
S 1.3 0.0 0:00
ASMProServer<br> 1 root
15 0 488 484 448
S 0.0 0.0 2:28
init<br> 2 root 0K
0 0 0 0
SW 0.0 0.0 0:00
migration_CPU0<br> 3 root
0K 0 0
0 0 SW 0.0 0.0 0:00
migration_CPU1<br> 4 root
0K 0 0
0 0 SW 0.0 0.0 0:00
migration_CPU2<br> 5 root
15 0 0
0 0 SW 0.0 0.0 0:03
keventd<br> 6 root 34
19 0 0 0
SWN 0.0 0.0 17:52 ksoftirqd_CPU0<br> 7
root 34 19
0 0 0 SWN 0.0
0.0 16:39 ksoftirqd_CPU1<br> 8
root 34 19
0 0 0 SWN 0.0
0.0 17:33 ksoftirqd_CPU2<br> 9
root 15 0
0 0 0 SW 0.0
0.0 28:22 kswapd<br> 10 root
15 0 0
0 0 SW 0.0 0.0 42:39
bdflush<br> 11 root 15
0 0 0 0
SW 0.0 0.0 3:08 kupdated<br> 12
root 25 0
0 0 0 SW 0.0
0.0 0:00 mdrecoveryd<br> 18
root 16 0
0 0 0 SW 0.0
0.0 0:00 scsi_eh_0<br> 21
root 15 0
0 0 0 SW 0.0
0.0 4:38 kjournald<br> 101 root
15 0 0
0 0 SW 0.0 0.0 0:00
khubd<br> 265 root 15
0 0 0 0
SW 0.0 0.0 0:03 kjournald<br> 266
root 15 0
0 0 0 SW 0.0
0.0 3:43 kjournald<br> 267 root
15 0 0
0 0 SW 0.0 0.0 0:04
kjournald<br> 268 root 15
0 0 0 0
SW 0.0 0.0 0:01 kjournald<br> 269
root 15 0
0 0 0 SW 0.0
0.0 0:11 kjournald<br> 270 root
15 0 0
0 0 SW 0.0 0.0 4:34
kjournald<br> 271 root 15
0 0 0 0
SW 0.0 0.0 4:28 kjournald<br> 272
root 15 0
0 0 0 SW 0.0
0.0 0:08 kjournald<br> 273 root
15 0 0
0 0 SW 0.0 0.0 0:14
kjournald<br> 274 root 15
0 0 0 0
SW 0.0 0.0 0:07 kjournald<br> 275
root 15 0
0 0 0 SW 0.0
0.0 1:14 kjournald<br> 805 root
15 0 588 576 532
S 0.0 0.0 1:39 syslogd<br> 810
root 15 0 448
432 432 S 0.0 0.0 0:00
klogd<br> 830 rpc 15
0 596 572 508 S 0.0
0.0 0:04 portmap<br> 858 rpcuser 19
0 708 608 608 S 0.0
0.0 0:00 rpc.statd<br> 970 root
15 0 0
0 0 SW 0.0 0.0 0:21
rpciod<br> 971 root 15
0 0 0 0
SW 0.0 0.0 0:00 lockd<br> 999
ntp 15 0 1812 1812
1732 S 0.0 0.0 5:04 ntpd<br> 1022
root 15 0 772
720 632 S 0.0 0.0 0:00
ypbind<br> 1024 root 15
0 772 720 632 S 0.0
0.0 1:16 ypbind</font></span></div>
<div><span><font size="2" color="#0000ff" face="Arial"></font></span> </div>
<div><span><font size="2" color="#0000ff" face="Arial">What
caught my eye was the number of processes along with the number of sleeping
processes.</font></span></div>
<div><span><font size="2" color="#0000ff" face="Arial">I
tried running the kill command on the kjournald instances, but that did not
appear to stop them.</font></span></div>
<div><span><font size="2" color="#0000ff" face="Arial"></font></span> </div>
<div><span><font size="2" color="#0000ff" face="Arial">Aside
from rebooting the server, which can be done if necessary, what other approach
can I try?</font></span></div>
<div><span><font size="2" color="#0000ff" face="Arial"></font></span> </div>
<div><span><font size="2" color="#0000ff" face="Arial"> </font></span></div>
<div dir="ltr" align="left"><br></div><br>
<div dir="ltr" align="left" lang="en-us">
<hr>
<font size="2" face="Tahoma"><div class="im"><b>From:</b> Daniel Wittenberg
[mailto:<a href="mailto:daniel.wittenberg.r0ko@statefarm.com" target="_blank">daniel.wittenberg.r0ko@statefarm.com</a>] <br></div><b>Sent:</b> Tuesday, December
07, 2010 9:11 AM<div><div></div><div class="h5"><br><b>To:</b> Nagios Users List<br><b>Subject:</b> Re:
[Nagios-users] Determining what is causing a highloadreportedby check_load
plugin<br></div></div></font><br></div><div><div></div><div class="h5">
<div></div>
<div>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">So
what are the first few processes listed in top? That should be what is
causing your load then.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Dan</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<div>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0in 0in;">
<p class="MsoNormal"><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> Kaplan, Andrew H.
[mailto:<a href="mailto:AHKAPLAN@PARTNERS.ORG" target="_blank">AHKAPLAN@PARTNERS.ORG</a>] <br><b>Sent:</b> Tuesday, December 07, 2010 7:49
AM<br><b>To:</b> Nagios Users List<br><b>Subject:</b> Re: [Nagios-users]
Determining what is causing a high loadreportedby check_load
plugin</span></p></div></div>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span style="font-size: 10pt; color: blue;">Hi there
--</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span style="font-size: 10pt; color: blue;">The load
values that are displayed in top match those for the check_load plugin. This is
the case whether the plugin</span></p>
<p class="MsoNormal"><span style="font-size: 10pt; color: blue;">is run
either automatically or interactively. The output for the uptime command is
shown below:</span></p>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal"><span style="font-size: 10pt; color: blue;">8:48am
up 153 days, 23:21, 1 user, load average: 73.36, 73.29,
73.21</span></p></div>
<div>
<p class="MsoNormal"> </p></div>
<div>
<p class="MsoNormal"> </p></div>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<div class="MsoNormal" style="text-align: center;" align="center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal" style="margin-bottom: 12pt;"><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> Daniel Wittenberg
[mailto:<a href="mailto:daniel.wittenberg.r0ko@statefarm.com" target="_blank">daniel.wittenberg.r0ko@statefarm.com</a>] <br><b>Sent:</b> Monday, December
06, 2010 4:40 PM<br><b>To:</b> Nagios Users List<br><b>Subject:</b> Re:
[Nagios-users] Determining what is causing a high load reportedby check_load
plugin</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">In
top, does it show the same load values? The status of your memory
shouldn’t cause the nagios plugin to report high cpu. What does the uptime
command say? Try running the check_load script by hand on that host and
verify it returns the same results.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"><br>Dan</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0in 0in;">
<p class="MsoNormal"><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> Marc Powell
[mailto:<a href="mailto:lists@xodus.org" target="_blank">lists@xodus.org</a>] <br><b>Sent:</b> Monday, December 06, 2010 3:26
PM<br><b>To:</b> Nagios Users List<br><b>Subject:</b> Re: [Nagios-users]
Determining what is causing a high load reported by check_load
plugin</span></p></div>
<p class="MsoNormal"> </p>
<p class="MsoNormal" style="margin-bottom: 12pt;"> </p>
<div>
<p class="MsoNormal">On Mon, Dec 6, 2010 at 1:50 PM, Kaplan, Andrew H. <<a href="mailto:AHKAPLAN@partners.org" target="_blank">AHKAPLAN@partners.org</a>>
wrote:</p>
<div>
<p><span style="font-size: 10pt;">Hi there
--</span> </p>
<p><span style="font-size: 10pt;">We are
running Nagios 3.1.2 server, and the client that is the subject of this e-mail
is running version 2.6 of the nrpe client.</span></p>
<p><span style="font-size: 10pt;">The
check_load plugin, version 1.4, is indicating the past three readings are the
following:</span> </p>
<p><span style="font-size: 10pt;">load
average: 71.00, 71.00, 70.95 CRITICAL</span> </p>
<p><span style="font-size: 10pt;">The critical
threshold of the plugin has been set to the 30, 25, 20 settings.</span>
</p>
<p><span style="font-size: 10pt;">When I
checked the client in question, the first thing I did was to run the top
command. The results are shown below:</span> </p>
<p><span style="font-size: 10pt;">CPU0
states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle</span>
<br><span style="font-size: 10pt;">CPU1
states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle</span>
<br><span style="font-size: 10pt;">CPU2
states: 1.0% user, 4.0% system, 0.0% nice, 93.0% idle</span>
<br><span style="font-size: 10pt;">Mem:
2064324K av, 2032308K used, 32016K
free, 0K shrd, 509924K buff</span>
<br><span style="font-size: 10pt;">Swap:
2096472K av, 21432K used, 2075040K
free
1035592K cached</span> </p>
<p><span style="font-size: 10pt;">The one
thing that I noticed was the amount of free memory was at thirty-two megabytes.
I wanted to know if that was</span> <br><span style="font-size: 10pt;">what was causing the
critical status to occur, or if there is something(s) else that I should
investigate.</span></p></div>
<div>
<p class="MsoNormal" style="margin-bottom: 12pt;"><br>Memory is not a factor in the
load calculation, only the number of processes running or waiting to run. For at
least 15 minutes you had approximately 71 processes either running or ready to
run and waiting on CPU resources. Running top/ps was the right thing to do but
you really need to do it when the problem is occurring to see what's actually
using all the CPU resources. There are far too many reasons why load could be
high but it should be easy for someone familiar with your system to figure it
out (at least generally) while
in-the-act.<br><br>--<br>Marc</p></div></div>
<p class="MsoNormal"><span style="font-family: 'Courier New';"><br><br>The
information in this e-mail is intended only for the person to whom it
is<br>addressed. If you believe this e-mail was sent to you in error and the
e-mail<br>contains patient information, please contact the Partners Compliance
HelpLine at<br><a href="http://www.partners.org/complianceline" target="_blank">http://www.partners.org/complianceline</a> . If the e-mail was sent
to you in error<br>but does not contain patient information, please contact the
sender and properly<br>dispose of the
e-mail.</span></p></div></div></div></div>
<br>------------------------------------------------------------------------------<br>
What happens now with your Lotus Notes apps - do you make another costly<br>
upgrade, or settle for being marooned without product support? Time to move<br>
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,<br>
use, and manage than apps on traditional platforms. Sign up for the Lotus<br>
Notes Migration Kit to learn more. <a href="http://p.sf.net/sfu/salesforce-d2d" target="_blank">http://p.sf.net/sfu/salesforce-d2d</a><br>_______________________________________________<br>
Nagios-users mailing list<br>
<a href="mailto:Nagios-users@lists.sourceforge.net">Nagios-users@lists.sourceforge.net</a><br>
<a href="https://lists.sourceforge.net/lists/listinfo/nagios-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/nagios-users</a><br>
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.<br>
::: Messages without supporting info will risk being sent to /dev/null<br></blockquote></div><br>