Issues with NEB modules breaking after restart

Andrew Widdersheim awiddersheim at hotmail.com
Thu Jun 13 16:55:53 CEST 2013


I recently just upgraded to the latest 3.5.0 release of nagios-core and just added livestatus into my environment. We are trying to replace NDO but currently have the two running at the same time along with NCPD for perfdata which as far as I know there shouldn't be an issues .

The first issue I had was where Nagios would segfault every night during it's routine log rotation so I applied the 0007-fix_downtime_struct.dif patch which seems to have fixed that issue. I experienced a new issue this morning where when restarting Nagios none of the NEB modules uninitialized properly. Nagios was able to start and initialized all of the NEB modules but a few seconds later Nagios uninitialized them again. This isn't like anything I've seen before and none of the NEB modules worked after this occurred. Here is what the logs looked like.


[Thu Jun 13 09:30:29 2013] Caught SIGTERM, shutting down...
[Thu Jun 13 09:30:30 2013] Successfully shutdown... (PID=14098)
[Thu Jun 13 09:30:31 2013] livestatus: Socket thread has terminated
[Thu Jun 13 09:30:41 2013] Nagios 3.5.0 starting... (PID=481)
[Thu Jun 13 09:30:41 2013] Local time is Thu Jun 13 09:30:41 EDT 2013
[Thu Jun 13 09:30:41 2013] LOG VERSION: 2.0
[Thu Jun 13 09:30:41 2013] livestatus: Livestatus 1.2.2p2 by Mathias Kettner. Socket: '/usr/local/nagios/var/rw/livestatus.sock'
[Thu Jun 13 09:30:41 2013] livestatus: Please visit us at http://mathias-kettner.de/
[Thu Jun 13 09:30:41 2013] livestatus: Hint: please try out OMD - the Open Monitoring Distribution
[Thu Jun 13 09:30:41 2013] livestatus: Please visit OMD at http://omdistro.org
[Thu Jun 13 09:30:41 2013] livestatus: Removed old left over socket file /usr/local/nagios/var/rw/livestatus.sock
[Thu Jun 13 09:30:41 2013] livestatus: archive path /drbd/r1/nagios/archives
[Thu Jun 13 09:30:41 2013] livestatus: Finished initialization. Further log messages go to /drbd/r1/nagios/livestatus.log
[Thu Jun 13 09:30:41 2013] Event broker module '/usr/local/mk-livestatus/livestatus.o' initialized successfully.
[Thu Jun 13 09:30:41 2013] npcdmod: Copyright (c) 2008-2009 Hendrik Baecker (andurin at process-zero.de) - http://www.pnp4nagios.org
[Thu Jun 13 09:30:41 2013] npcdmod: /usr/local/pnp4nagios/etc/npcd.cfg initialized
[Thu Jun 13 09:30:41 2013] npcdmod: spool_dir = '/dev/shm/pnp4nagios/var/spool/'.
[Thu Jun 13 09:30:41 2013] npcdmod: perfdata file '/dev/shm/pnp4nagios/var/perfdata.dump'.
[Thu Jun 13 09:30:41 2013] npcdmod: Ready to run to have some fun!
[Thu Jun 13 09:30:41 2013] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[Thu Jun 13 09:30:41 2013] Event broker module '/usr/local/pnp4nagios/lib64/npcdmod.o' initialized successfully.
[Thu Jun 13 09:30:41 2013] ndomod: NDOMOD 1.5.2 (06-08-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[Thu Jun 13 09:30:41 2013] ndomod: Successfully connected to data sink.  0 queued items to flush.
[Thu Jun 13 09:30:41 2013] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[Thu Jun 13 09:30:43 2013] Finished daemonizing... (New PID=482)
[Thu Jun 13 09:30:44 2013] TIMEPERIOD TRANSITION: 24x7;-1;1
[Thu Jun 13 09:30:47 2013] Event broker module '/usr/local/mk-livestatus/livestatus.o' deinitialized successfully.
[Thu Jun 13 09:30:47 2013] npcdmod: If you don't like me, I will go out! Bye.
[Thu Jun 13 09:30:47 2013] Event broker module '/usr/local/pnp4nagios/lib64/npcdmod.o' deinitialized successfully.
[Thu Jun 13 09:30:47 2013] ndomod: Shutdown complete.
[Thu Jun 13 09:30:47 2013] Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.

Here is the next restart after this where things happened as I would expect:

[Thu Jun 13 09:52:25 2013] Successfully shutdown... (PID=482)
[Thu Jun 13 09:52:26 2013] livestatus: Socket thread has terminated
[Thu Jun 13 09:52:26 2013] Event broker module '/usr/local/mk-livestatus/livestatus.o' deinitialized successfully.
[Thu Jun 13 09:52:26 2013] npcdmod: If you don't like me, I will go out! Bye.
[Thu Jun 13 09:52:26 2013] Event broker module '/usr/local/pnp4nagios/lib64/npcdmod.o' deinitialized successfully.
[Thu Jun 13 09:52:26 2013] ndomod: Shutdown complete.
[Thu Jun 13 09:52:26 2013] Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
[Thu Jun 13 09:52:29 2013] Nagios 3.5.0 starting... (PID=20081)
[Thu Jun 13 09:52:29 2013] Local time is Thu Jun 13 09:52:29 EDT 2013
[Thu Jun 13 09:52:29 2013] LOG VERSION: 2.0
[Thu Jun 13 09:52:29 2013] livestatus: Livestatus 1.2.2p2 by Mathias Kettner. Socket: '/usr/local/nagios/var/rw/livestatus.sock'
[Thu Jun 13 09:52:29 2013] livestatus: Please visit us at http://mathias-kettner.de/
[Thu Jun 13 09:52:29 2013] livestatus: Hint: please try out OMD - the Open Monitoring Distribution
[Thu Jun 13 09:52:29 2013] livestatus: Please visit OMD at http://omdistro.org
[Thu Jun 13 09:52:29 2013] livestatus: archive path /drbd/r1/nagios/archives
[Thu Jun 13 09:52:29 2013] livestatus: Finished initialization. Further log messages go to /drbd/r1/nagios/livestatus.log
[Thu Jun 13 09:52:29 2013] Event broker module '/usr/local/mk-livestatus/livestatus.o' initialized successfully.
[Thu Jun 13 09:52:29 2013] npcdmod: Copyright (c) 2008-2009 Hendrik Baecker (andurin at process-zero.de) - http://www.pnp4nagios.org
[Thu Jun 13 09:52:29 2013] npcdmod: /usr/local/pnp4nagios/etc/npcd.cfg initialized
[Thu Jun 13 09:52:29 2013] npcdmod: spool_dir = '/dev/shm/pnp4nagios/var/spool/'.
[Thu Jun 13 09:52:29 2013] npcdmod: perfdata file '/dev/shm/pnp4nagios/var/perfdata.dump'.
[Thu Jun 13 09:52:29 2013] npcdmod: Ready to run to have some fun!
[Thu Jun 13 09:52:29 2013] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[Thu Jun 13 09:52:29 2013] Event broker module '/usr/local/pnp4nagios/lib64/npcdmod.o' initialized successfully.
[Thu Jun 13 09:52:29 2013] ndomod: NDOMOD 1.5.2 (06-08-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[Thu Jun 13 09:52:29 2013] ndomod: Successfully connected to data sink.  0 queued items to flush.
[Thu Jun 13 09:52:29 2013] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[Thu Jun 13 09:52:30 2013] Finished daemonizing... (New PID=20136)

You'll notice in the first snippet of the logs things clearly did not clean up because when livestatus started it had:

Removed old left over socket file /usr/local/nagios/var/rw/livestatus.sock

Anyone experience the same thing? Any idea what I can do to fix? 		 	   		  
------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null





More information about the Users mailing list