Extremely bad performance when enabling process_performance_data on Solaris 10?

[Andreas Ericsson]

Steffen Poulsen wrote:
> Hi,
> 
> We are trying to add graphs to our nagios installation using
> nagiosgrapher, and we would now like to enable the
> process_performance_data in nagios.cfg to allow for performance data to
> be sent to our graphing server, using udp.
> 
> We have tried various of doing this, invoking perl on udpsend.pl, using
> epn (the embedded perl compiler) and compiling udpecho.c.
> 
> udpecho.c is the fastest of the set, but still performance is alarming:
> only 4 (four!) passive checks were now processed per second, everything
> grinded to a halt and we had to disable process_performance_data again.
> We now have ~5000 services reporting in using NSCA, so we need another
> kind of performance.
> 
> We expect that others have probably ran into this issue as well and we
> are eager to hear how other people solved it?
> 

I haven't run into it, but I would solve it with a NEB-module that sends
the performance data to a graphing server. It's really quite trivial to
do, and a send(2) call generally finishes quickly enough.

> It seems it doesn't matter which command is run, it can be a no-op
> script - as soon as any command is set, processing "halts".
> 
> We are running Nagios 3b03 and the hardware platform is a Sun T1000
> (should be reasonably up to date,
> http://www.sun.com/servers/coolthreads/t1000/).
> 
> If there is no imminent patch for the above issue, perhaps it is a
> possibility to redirect performance data output to a file and have an
> external process read from it? (while readline, send udp). Any existing
> solutions to this end?
> 

Nagios can write performance data to a file that can be read in bulk once
every 30 seconds or so. Nagiosgraph can read that file.

> All inputs are appriciated - if we are on the wrong track using Sun hw
> for Nagios and we would be better of on i386, please let us know.
> 

Sun HW shouldn't be the problem, but I think Nagios is primarily developed
on and for Linux, where fork() is ridiculously cheap. It is (dramatically)
less so on other platforms.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/