High latency on small installation
Assaf Flatto
nagios at flatto.net
Tue Jul 20 18:36:09 CEST 2010
Hello All
I am having a problem with very high latency on my main nagios server
(3.2.0 from source on SLES 10.3 x64).
I recompiled the core with the embedded perl and that helped for a
while to lower the latency but it keeps growing to times that are not
reasonable for this size of a nagios installation .
The setup of this server is as the main nagios with two other server
sending passive results to it , one server's time interval is 40 sec
(doing mainly http queries ) and the other has a 1 minute interval , and
monitors Switches ,firewalls and hosts.
The machine is a dual CPU Quad core , 6 GB ram and the load is about 3.3 .
I've attached the output of nagiostats below and the directives of
nagios.cfg .
moving the result/ stats to a ramfs is not an option at this time ( i
know about this , but sadly not a viable option).
Thanks
CURRENT STATUS DATA
------------------------------------------------------
Status File: /usr/local/nagios/var/status.dat
Status File Age: 0d 0h 0m 30s
Status File Version: 3.2.0
Program Running Time: 0d 1h 46m 11s
Nagios PID: 27584
Used/High/Total Command Buffers: 8 / 141 / 4096
Total Services: 2687
Services Checked: 2687
Services Scheduled: 2638
Services Actively Checked: 2638
Services Passively Checked: 49
Total Service State Change: 0.000 / 56.710 / 0.071 %
Active Service Latency: 376.745 / 438.212 / 413.404 sec
Active Service Execution Time: 0.040 / 26.399 / 0.439 sec
Active Service State Change: 0.000 / 56.710 / 0.072 %
Active Services Last 1/5/15/60 min: 0 / 1088 / 2627 / 2638
Passive Service Latency: 9.664 / 47.661 / 30.084 sec
Passive Service State Change: 0.000 / 0.000 / 0.000 %
Passive Services Last 1/5/15/60 min: 10 / 49 / 49 / 49
Services Ok/Warn/Unk/Crit: 2673 / 11 / 1 / 2
Services Flapping: 4
Services In Downtime: 0
Total Hosts: 244
Hosts Checked: 244
Hosts Scheduled: 219
Hosts Actively Checked: 232
Host Passively Checked: 12
Total Host State Change: 0.000 / 0.000 / 0.000 %
Active Host Latency: 0.018 / 435.890 / 400.684 sec
Active Host Execution Time: 0.041 / 4.136 / 3.944 sec
Active Host State Change: 0.000 / 0.000 / 0.000 %
Active Hosts Last 1/5/15/60 min: 0 / 96 / 219 / 219
Passive Host Latency: 0.184 / 67.224 / 18.642 sec
Passive Host State Change: 0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0
Hosts Up/Down/Unreach: 244 / 0 / 0
Hosts Flapping: 0
Hosts In Downtime: 0
Active Host Checks Last 1/5/15 min: 21 / 129 / 357
Scheduled: 20 / 124 / 334
On-demand: 1 / 5 / 23
Parallel: 20 / 124 / 335
Serial: 0 / 0 / 0
Cached: 1 / 5 / 22
Passive Host Checks Last 1/5/15 min: 0 / 0 / 0
Active Service Checks Last 1/5/15 min: 297 / 1481 / 4462
Scheduled: 297 / 1481 / 4462
On-demand: 0 / 0 / 0
Cached: 0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 21 / 37 / 37
External Commands Last 1/5/15 min: 77 / 350 / 1030
log_file=/usr/local/nagios/var/nagios.log
cfg_dir=/usr/local/nagios/etc/objects
cfg_dir=/usr/local/nagios/etc/LFI
cfg_dir=/usr/local/nagios/etc/GB
cfg_dir=/usr/local/nagios/etc/DE
cfg_dir=/usr/local/nagios/etc/SE
cfg_dir=/usr/local/nagios/etc/NOR-CLOUD
cfg_dir=/usr/local/nagios/etc/network
cfg_dir=/usr/local/nagios/etc/Leaf
cfg_dir=/usr/local/nagios/etc/sites
object_cache_file=/usr/local/nagios/var/objects.cache
precached_object_file=/usr/local/nagios/var/objects.precache
resource_file=/usr/local/nagios/etc/resource.cfg
status_file=/usr/local/nagios/var/status.dat
status_update_interval=10
nagios_user=nagios
nagios_group=nagios
check_external_commands=1
command_check_interval=-1
command_file=/var/spool/nagios/nagios.cmd
external_command_buffer_slots=4096
lock_file=/var/run/nagios/nagios.lock
temp_file=/usr/local/nagios/nagios.tmp
temp_path=/tmp
event_broker_options=-1
broker_module=/usr/local/nagios/bin/ndomod-3x.o
config_file=/usr/local/nagios/etc/ndomod.cfg
log_rotation_method=d
log_archive_path=/usr/local/nagios/var/archives
use_syslog=0
log_notifications=1
log_service_retries=1
log_host_retries=1
log_event_handlers=1
log_initial_states=0
log_passive_checks=1
service_inter_check_delay_method=s
max_service_check_spread=30
service_interleave_factor=s
host_inter_check_delay_method=s
max_host_check_spread=30
max_concurrent_checks=0
check_result_reaper_frequency=5
max_check_result_reaper_time=30
check_result_path=/var/spool/nagios
max_check_result_file_age=3600
cached_host_check_horizon=15
cached_service_check_horizon=15
enable_predictive_host_dependency_checks=1
enable_predictive_service_dependency_checks=1
soft_state_dependencies=0
sleep_time=0.25
service_check_timeout=60
host_check_timeout=30
event_handler_timeout=60
notification_timeout=30
ocsp_timeout=5
perfdata_timeout=5
retain_state_information=1
state_retention_file=/usr/local/nagios/lib/retention.dat
retention_update_interval=60
use_retained_program_state=1
use_retained_scheduling_info=1
retained_host_attribute_mask=0
retained_service_attribute_mask=0
retained_process_host_attribute_mask=0
retained_process_service_attribute_mask=0
retained_contact_host_attribute_mask=0
retained_contact_service_attribute_mask=0
interval_length=60
use_aggressive_host_checking=0
execute_service_checks=1
accept_passive_service_checks=1
execute_host_checks=1
accept_passive_host_checks=1
enable_notifications=1
enable_event_handlers=1
process_performance_data=1
host_perfdata_command=process-host-perfdata
service_perfdata_command=process-service-perfdata
obsess_over_services=0
obsess_over_hosts=0
translate_passive_host_checks=1
passive_host_checks_are_soft=0
check_for_orphaned_services=1
check_for_orphaned_hosts=1
check_service_freshness=1
service_freshness_check_interval=60
check_host_freshness=1
host_freshness_check_interval=120
additional_freshness_latency=10
enable_flap_detection=1
low_service_flap_threshold=5.0
high_service_flap_threshold=20.0
low_host_flap_threshold=5.0
high_host_flap_threshold=20.0
date_format=euro
p1_file=/usr/lib/nagios/p1.pl
enable_embedded_perl=1
use_embedded_perl_implicitly=1
illegal_object_name_chars=`~!$%^&*|'"<>?,()=
illegal_macro_output_chars=`~$&|'"<>
use_regexp_matching=0
use_true_regexp_matching=0
admin_email=nagios at localhost
admin_pager=pagenagios at localhost
daemon_dumps_core=0
use_large_installation_tweaks=1
enable_environment_macros=1
free_child_process_memory=1
debug_level=0
debug_verbosity=0
debug_file=/usr/local/nagios/var/nagios.debug
max_debug_file_size=1000000
--
Never,Ever Cut A Deal With a Dragon
------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
::: Messages without supporting info will risk being sent to /dev/null
More information about the Users
mailing list