High Latency with service checks

[Yueh-Hung Liu]

another possibility, do you use addon to do task after "every" checking?
for example, update RRD files for performance graphing, sometimes that
should increase the checking latency.


2011/2/16 Yu Watanabe <yu.watanabe at jp.fujitsu.com>:
> Thank you very much for the reply.
>
>
> Assaf Flatto さんは書きました:
>>Yu Watanabe wrote:
>>> Hello All.
>>>
>>> I would like to get an advice with nagios latency issue.
>>>
>>> In nagios 3.0.6 with RHEL 4, would there be a possibility
>>> that service check latency hikes even though the sar or iostat
>>> usage is not in relatively high load usage? (I am planning to upgrade to v 3.2.3 soon.)
>>>
>>> SAR average cpu usage were 40% and iowait was lying 0%. Swaping were not occuring.
>>>
>>> I have more than 1000 ping checks and average latency are 20 min. Check execution time for ping is
>>> all below 1 sec. Plugin I am using is check_icmp.
>>>
>>> CPU I am using is ,
>>> Intel(R) Xeon(TM) CPU 2.80GHz
>>> Memory is 8GB
>>> OS is RHEL 4.4
>>>
>>> Are there any possiblity that Nagios gets locked up with the service check scheduling?
>>>
>>> Thanks,
>>> Yu
>>>
>>>
>>>
>>have you tried throttling the amount of concurrent checks ?
>>could it be that the 1000 pings is flooding your network ?
>
>  I am not sure that I understand the meaning by throttling...
>  Would there be a parameter in Nagios to control this?
>
>  For flooding I will check the netstat -s periodically and see if there are too much.
>
>>
>>I've encountered a similar issue with a  a setup I had ( granted the
>>latency wasn't that extreme ) , and from what you describe the symptoms
>>sound the same .
>>
>>We tried several solutions (DNX , Mod_gearman) to reduce the latency ,
>>the solution that worked int he end was adding extra RAM to the machine
>>and that solved it - I know that is not the best method , but non of the
>>"regular" methods of tweaking nagios (large install , ramfs etc` )
>>worked , the boost in ram reduced the latency from 6+ minutes to 3 sec.
>
>  As long as I see the vmstat , there seems to be enough memories left
>  for buffer and cache since there aren't any swaping.
>
> Thanks ,
> Yu
>
>>
>>Assaf
>>
>>
>>------------------------------------------------------------------------------
>>The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
>>Pinpoint memory and threading errors before they happen.
>>Find and fix more than 250 security defects in the development cycle.
>>Locate bottlenecks in serial and parallel code that limit performance.
>>http://p.sf.net/sfu/intel-dev2devfeb
>>_______________________________________________
>>Nagios-users mailing list
>>Nagios-users at lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/nagios-users
>>::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
>>::: Messages without supporting info will risk being sent to /dev/null
>>
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> Nagios-users mailing list
> Nagios-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null