Alleviating Nagios i/o contention problem

[Marc Powell]

On Sep 25, 2010, at 10:53 AM, Max wrote:

> I like the suggestions Matthias makes; those suggestions have worked
> well for us.
> 
> RRD updates are very expensive - I am pretty sure without knowing
> anything more about your system that the RRD writes are causing most
> of the I/O load.

I no longer have access to this system but my experience has been otherwise. We were running a nagios install with nearly 10,000 services received by external pollers every 5 minutes, and a cricket install on the same machine polling/updating 100,000+ rrd files during the same interval. This was on a Poweredge 6850, 5 disk RAID-5.
RRDtool itself writes very little data to disk. I think it's 8 Bytes per DS per RRA updated. Linux, though, wants to write 4KB chunks at a time so it performs a read-modify-write of 4KB just to update those 8 Bytes.

The OP can reduce his IO load particularly for RRD updates and help Linux better organize it's writes to disk by ensuring that he has enough RAM to keep key information for each RRD file in the filesystem cache. The OP will need at least 8K * number of rrd files available to be used as filesystem buffer cache. 

--
Marc
------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null