Nagios check latency

Justin Pryzby justinp at norchemlab.com
Wed Aug 21 05:30:02 CEST 2013


I would have written: ps -fC nagios3 (but I guess your daemon is
called "nagios").  Note that all the "2ndary" lines are commands being
run (checks, notifications).  This will show only the parent
"scheduler" processes: ps -O ppid -C nagios3 |awk '$2==1'

Justin

On Wed, Aug 21, 2013 at 02:49:27AM +0000, Muhamad Faiz wrote:
> Hi Justin,
> 
> Is this the right way to check this? I have stop and start back only one line exist now. I'm watching the latency now.
> 
> [root at nagios ~]# ps faux | grep nagios.cfg
> root     21594  0.0  0.0 103244   832 pts/0    S+   10:42   0:00                          \_ grep nagios.cfg
> nagios   29669 52.4 13.6 1677324 1095904 ?     Rsl  Aug13 6014:30 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21086  0.3 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21291  0.5 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21293  0.5 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21295  0.5 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21296  0.5 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21338  0.6 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21346  0.4 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21453  1.0 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21519  1.5 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21521  2.0 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21530  2.5 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21538  2.5 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21547  2.0 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21555  1.0 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21579  4.0 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21586  4.0 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21589  4.0 13.6 1677328 1094340 ?     S    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> nagios   21591  2.0 13.6 1677324 1094328 ?     R    10:42   0:00  \_ /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
> 
> Regards,
> 
> Faiz
> Monitoring
> 
> 
> -----Original Message-----
> From: Justin Pryzby [mailto:justinp at norchemlab.com] 
> Sent: Tuesday, August 13, 2013 9:51 PM
> To: nagios-users at lists.sourceforge.net
> Subject: Re: [Nagios-users] Nagios check latency
> 
> Could you check that you're not accidentally running 2 nagios daemons?
> 
> On Tue, Aug 13, 2013 at 03:49:19AM +0000, Muhamad Faiz wrote:
> > Hi guys,
> > 
> > My Nagios setup are having high service check latency.
> > 
> > Check nagios latency result is 240 seconds!
> > 
> > ./check_nagios_latency.sh -w 30 -c 60
> > CRITICAL: Latency at 240.183 seconds!
> > 
> > Modification in my Nagios.cfg file is only as below. Other than this, is default.
> > 
> > # MODIFIED
> > admin_email=muhamad.faiz at asia.xchanging.com
> > admin_pager=muhamad.faiz at asia.xchanging.com
> > translate_passive_host_checks=1
> > log_event_handlers=0
> > use_large_installation_tweaks=1
> > enable_environment_macros=0
> > enable_embedded_perl=0
> > use_embedded_perl_implicitly=0
> > max_concurrent_checks=1600
> > service_perfdata_file_processing_command=process-service-perfdata-file
> > -pnp-bulk 
> > host_perfdata_file_processing_command=process-host-perfdata-file-pnp-b
> > ulk
> > service_perfdata_file_processing_interval=60
> > host_perfdata_file_processing_interval=60
> > status_update_interval=40
> > 
> > 
> > # /usr/local/nagios/bin/nagios -s /usr/local/nagios/etc/nagios.cfg
> > 
> > Nagios Core 3.5.0
> > Copyright (c) 2009-2011 Nagios Core Development Team and Community 
> > Contributors Copyright (c) 1999-2009 Ethan Galstad Last Modified: 
> > 03-15-2013
> > License: GPL
> > 
> > Website: http://www.nagios.org
> > Timing information on object configuration processing is listed below.  
> > You can use this information to see if precaching your object 
> > configuration would be useful.
> > 
> > Object Config Source: Config files (uncached)
> > 
> > OBJECT CONFIG PROCESSING TIMES      (* = Potential for precache savings with -u option)
> > ----------------------------------
> > Read:                 0.415803 sec
> > Resolve:              0.006900 sec  *
> > Recomb Contactgroups: 0.001862 sec  *
> > Recomb Hostgroups:    3.819786 sec  *
> > Dup Services:         0.637380 sec  *
> > Recomb Servicegroups: 3.419804 sec  *
> > Duplicate:            0.000417 sec  *
> > Inherit:              0.013748 sec  *
> > Recomb Contacts:      0.000000 sec  *
> > Sort:                 0.000000 sec  *
> > Register:             0.616385 sec
> > Free:                 0.029930 sec
> >                       ============
> > TOTAL:                8.962015 sec  * = 7.899897 sec (88.15%) estimated savings
> > 
> > 
> > RETENTION DATA TIMES
> > ----------------------------------
> > Read and Process:     0.863815 sec
> >                       ============
> > TOTAL:                0.863815 sec
> > 
> > 
> > Timing information on configuration verification is listed below.
> > 
> > CONFIG VERIFICATION TIMES          (* = Potential for speedup with -x option)
> > ----------------------------------
> > Object Relationships: 0.080360 sec
> > Circular Paths:       0.001152 sec  *
> > Misc:                 0.011244 sec
> >                       ============
> > TOTAL:                0.092756 sec  * = 0.001152 sec (1.2%) estimated savings
> > 
> > 
> > EVENT SCHEDULING TIMES
> > -------------------------------------
> > Get service info:        0.051265 sec
> > Get host info info:      0.032301 sec
> > Get service params:      0.000010 sec
> > Schedule service times:  0.002476 sec
> > Schedule service events: 0.002514 sec
> > Get host params:         0.000000 sec
> > Schedule host times:     0.000937 sec
> > Schedule host events:    0.001287 sec
> >                          ============
> > TOTAL:                   0.090790 sec
> > 
> > 
> > Projected scheduling information for host and service checks is listed 
> > below.  This information assumes that you are going to start running 
> > Nagios with your current config files.
> > 
> > HOST SCHEDULING INFORMATION
> > ---------------------------
> > Total hosts:                     11753
> > Total scheduled hosts:           7
> > Host inter-check delay method:   SMART
> > Average host check interval:     300.00 sec
> > Host inter-check delay:          42.86 sec
> > Max host check spread:           30 min
> > First scheduled check:           Tue Aug 13 11:43:54 2013
> > Last scheduled check:            Tue Aug 13 11:43:54 2013
> > 
> > 
> > SERVICE SCHEDULING INFORMATION
> > -------------------------------
> > Total services:                     18662
> > Total scheduled services:           170
> > Service inter-check delay method:   SMART
> > Average service check interval:     535.76 sec
> > Inter-check delay:                  3.15 sec
> > Interleave factor method:           SMART
> > Average services per host:          1.59
> > Service interleave factor:          1
> > Max service check spread:           30 min
> > First scheduled check:              Tue Aug 13 11:52:49 2013
> > Last scheduled check:               Tue Aug 13 11:58:39 2013
> > 
> > 
> > CHECK PROCESSING INFORMATION
> > ----------------------------
> > Check result reaper interval:       10 sec
> > Max concurrent service checks:      1600
> > 
> > 
> > PERFORMANCE SUGGESTIONS
> > -----------------------
> > I have no suggestions - things look okay.
> > 
> > Any advice will be much appreciated.
> > 
> > Regards,
> > Faiz
> > "CONFIDENTIALITY NOTICE: This message and any attachment are confidential and may also be privileged. If you are not the intended recipient of this e-mail you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you are not the intended recipient please telephone or e-mail the sender and delete this message and any attachment from your system."
> 
> > BEGIN:VCARD
> > VERSION:2.1
> > X-MS-SIGNATURE:YES
> > N;LANGUAGE=en-us:Faiz
> > FN:Faiz
> > ORG:YTLC Network Operations Monitoring Team
> > TEL;WORK;VOICE:+60 192260176
> > ADR;WORK;PREF:;;;;;;United States of America
> > X-MS-OL-DEFAULT-POSTAL-ADDRESS:2
> > EMAIL;PREF;INTERNET:Muhamad.Faiz at Asia.Xchanging.com
> > X-MS-IMADDRESS:Skype: me.faiz
> > X-MS-CARDPICTURE;TYPE=JPEG;ENCODING=BASE64:
> >  
> > /9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAcFBQYFBAcGBQYIBwcIChELCgkJChUPEAwRGB
> > Ua  
> > GRgVGBcbHichGx0lHRcYIi4iJSgpKywrGiAvMy8qMicqKyr/2wBDAQcICAoJChQLCxQqHB
> > gc  
> > KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKir/wA
> > AR  
> > CAAVADYDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtR
> > AA  
> > AgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2Jygg
> > kK  
> > FhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhI
> > WG  
> > h4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+
> > Tl  
> > 5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtR
> > EA  
> > AgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRCh
> > Yk  
> > NOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6go
> > OE  
> > hYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4u
> > Pk  
> > 5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD2/wAWnWYNI+1aDciKSAlpUKBg6dyPkY
> > 5H
> >  
> > oBzzXK6d8SW07SpB4hVp7zf+5EcboZV753RoBj8c5rf+IXjOHwV4Ye7ba95NmO0hP8b46k
> > f3  
> > R1P4DvXinhexuPFE51rU7YLbo5QFIwkckmMkDBXGMg8GvUwdCFWF6i0/F+R5OPxE6WlJ6/
> > gv  
> > M9w0HxhDrWmz6hPbNYWcLBfOncbWJOPyGRz71uw3ttPGksE8ciO7RqysMMwJBA9SNp/I1h
> > xx  
> > 6VP4HazgdrOxeL7PIIwGdAxww78nJ55POa5+HQfCyqInvnuYY12KkkCuUULsVd2zdtUYwP
> > UZ
> >  
> > OSa5XTjKUuVNK/qdFKrKNOPPJNtb7Ho2ajEqGQxhxvABK55APfH4GvP4/DfhoTvdSajNKZ
> > B8  
> > knlDcGDlt+4JkuCSNx5xiqmn+HtAtoWa/wBSWSZzmMQwhVj2qFDj5Bhh13ccn8SvYLu/uZ
> > bx  
> > CXb7z0/rRXL+B9Ni03SrlIbmO5SW437o49mMRogzwMkhQScdSaKwlCzsbxqXVzo5LeKfBm
> > iS  
> > THTeoOKq32lWl/YNazR7Ys5AjO0qfUEdKKKUZNPQc4xcWmiHSdCtdIaQ2jTESYDLJKWGfX
> > nv  
> > WoEVSSABk5PHWiinKTlK7JpwjGNooasSRghFVQTkgDGSe9SYHoKKKk0sgooooGf/2Q==
> > 
> > X-MS-OL-DESIGN;CHARSET=utf-8:<card 
> > xmlns="http://schemas.microsoft.com/office/outlook/12/electronicbusine
> > sscards" ver="1.0" layout="right" bgcolor="ffffff"><img xmlns="" 
> > align="midright" area="50" use="cardpicture"/><fld xmlns="" 
> > prop="name" align="left" dir="ltr" style="b" color="000000" 
> > size="10"/><fld xmlns="" prop="org" align="left" dir="ltr" 
> > color="000000" size="8"/><fld xmlns="" prop="title" align="left" 
> > dir="ltr" color="000000" size="8"/><fld xmlns="" prop="dept" 
> > align="left" dir="ltr" color="000000" size="8"/><fld xmlns="" 
> > prop="blank" size="8"/><fld xmlns="" prop="telwork" align="left" 
> > dir="ltr" color="000000" size="8"/><fld xmlns="" prop="email" 
> > align="left" dir="ltr" color="000000" size="8"/><fld xmlns="" 
> > prop="addrwork" align="left" dir="ltr" color="000000" size="8"/><fld 
> > xmlns="" prop="blank" size="8"/><fld xmlns="" prop="im" align="left" 
> > dir="ltr" color="000000" size="8"/><fld xmlns="" prop="blank" 
> > size="8"/><fld xmlns="" prop="blank" size="8"/><fld xmlns="" 
> > prop="blank" size="8"/><fld xmlns="" prop="blank" size="8"/><fld 
> > xmlns="" prop="blank" size="8"/><fld xmlns="" prop="blank" 
> > size="8"/></card> REV:20121031T020436Z END:VCARD
> 
> > ----------------------------------------------------------------------
> > -------- Get 100% visibility into Java/.NET code with AppDynamics 
> > Lite!
> > It's a free troubleshooting tool designed for production.
> > Get down to code-level detail for bottlenecks, with <2% overhead. 
> > Download for free and get started troubleshooting in minutes. 
> > http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.c
> > lktrk
> 
> > _______________________________________________
> > Nagios-users mailing list
> > Nagios-users at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nagios-users
> > ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
> > ::: Messages without supporting info will risk being sent to /dev/null
> 
> 
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead. 
> Download for free and get started troubleshooting in minutes. 
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Nagios-users mailing list
> Nagios-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
> ::: Messages without supporting info will risk being sent to /dev/null
> 
> ------------------------------------------------------------------------------
> Introducing Performance Central, a new site from SourceForge and 
> AppDynamics. Performance Central is your source for news, insights, 
> analysis and resources for efficient Application Performance Management. 
> Visit us today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
> _______________________________________________
> Nagios-users mailing list
> Nagios-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
> ::: Messages without supporting info will risk being sent to /dev/null
> 

------------------------------------------------------------------------------
Introducing Performance Central, a new site from SourceForge and 
AppDynamics. Performance Central is your source for news, insights, 
analysis and resources for efficient Application Performance Management. 
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null





More information about the Users mailing list