<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Arial","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Arial","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>Greetings, listers,<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>We've got an on-going issue with i/o contention. There's
the obvious problem that we've got a whole lot of things all writing to the
same partition. In this case, there's just one big chunk of RAID 5 disk
on a single controller so I don't believe that making more partitions is going
to help.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>On this same partition we have:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>1) Nagios 3.2.1 running as the central/reporting server for
a couple of other Nagios nodes that are sending check results via NSCA. Approximately
6-7K checks.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>2) pnp4nagios 0.6.2 (with rrd 1.4.2) writing graph data.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>There's a 2nd server configured identically to the first
that's acting as a "hot spare" so it also receives check data from
the 2 distributed nodes and writes its own copy of the graph data locally as
well.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>At the moment I'm concerned about the graphdata, but because
I can only see i/o utilization as an aggregate, I can't tell what is the worst
component on that filesystem -- status.dat updates? graph data?
writes to the var/spool directory? We also look at continued growth so
this is only going to get worse.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>These systems are quite lightly loaded from a CPU (2
dual-core CPUs) and memory (4GB) perspective, but the i/o to the nagios
filesystem is queuing now.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>We're about to order new hardware for these servers and I
want to make a reasonable choice. I'd like to make some reasonable
changes without requiring too exotic of a setup. I believe these servers
are currently Dell 2950s and they're all running Suse Linux 10.3 SP2.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>My first thought was to potentially move the graphs to a NAS
share which would shift that i/o to the network. I don't know how that
would work though and it would ultimately be an experiment.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>What experiences do people out there have handling this kind
of i/o and what have you done to ease it?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thanks very much!<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Mark<span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>