<tt>Hi Bernd,</tt> <tt>hi Andreas, </tt> <tt>> To alleviate your issue, you should be running an ntp daemon > on the Nagios server which slews the clock into its right > time rather than sets it (slew = make it go slightly faster > or slower until it matches the correct time). Are you running > ntpdate via a cronjob or something? ></tt> <tt>> I'm not sure how one would go about debugging this, as the > time required to run a single test is prohibitive for rapid > repeated testing. </tt> <tt>I already encountered that problem before and started debugging it,</tt> <tt>so I'll just share my knowledge so far. Sadly I didn't get the time</tt> <tt>yet to really pinpoint a solution to it and produce a patch.</tt> <tt>I'm not that big fan of C ;)</tt> <tt>How to produce it:</tt> <tt>- define a check "freaky_check" with limited check_period, let's</tt> <tt> call it 7to11 and a check_interval of 3</tt> <tt>- produce steady time-shifts backwards (nagios running in a VM someone?)</tt> <tt>What happens:</tt> <tt>1. it's 11pm, nagios schedules freaky_check for 7am according to its check_period</tt> <tt>2. Every X minutes timeshift -1 sec (jittering timesource)</tt> <tt>3. nagios tries to compensate it and adjusts _all_ checks to the timeshift (next_check = next_check - timeshift)</tt> <tt>4. time goes by from 11pm to 6am, shifting time for - let's say - 8 minutes back</tt> <tt>5. freaky_check is now scheduled for 6:52am because of the timeshifts</tt> <tt>6. it's 6:52am and nagios tries to run the freaky_check according to the schedule</tt> <tt>7. sanity check says: ERROR: check outside check_period</tt> <tt>8. nagios tries to compensate with a strange logic: next_check = next_check + check_interval and just hopes it will fit</tt> <tt>9. nagios reruns the sanity check: FATAL ERROR: check still outside check_period - I have no clue what to do: rescheduling freaky_check: next_check = next_check + 1year</tt> <tt>10. user puzzled and nagios thinks it's all cool</tt> <tt>Conclusion:</tt> <tt>This behaviour turns up when the following criterias are met:</tt> <tt>- check has a reduced check_period</tt> <tt>- time is shifting back</tt> <tt>- the timeshift outside the check_period is greater then 2 times the</tt> <tt> check_interval</tt> <tt>You can look it up in base/checks.c within the</tt> <tt>run_scheduled_service_check(service *svc, int check_options, double latency)</tt> <tt>function for example. </tt> <tt>After some basic checks this will be run:</tt> <tt>/* attempt to run the check */</tt> <tt>result=run_async_service_check(svc,check_options,latency,TRUE,TRUE,&time_is_valid,&preferred_time);</tt> <tt>which in turn ends up with:</tt> <tt>/* is the service check viable at this time? */</tt> <tt>if(check_service_check_viability(svc,check_options,time_is_valid,preferred_time)==ERROR)</tt> <tt> return ERROR;</tt> <tt>No, since nagios shifted it outside its check_period, the time is NOT valid.</tt> <tt>Back in run_scheduled_service_check we now enter the (if result==ERROR) tree:</tt> <tt>/* get current time */</tt> <tt>time(&current_time);</tt> <tt>/* determine next time we should check the service if needed */</tt> <tt>/* if service has no check interval, schedule it again for 5 minutes from now */</tt> <tt>if(current_time>=preferred_time)</tt> <tt> preferred_time=current_time+((svc->check_interval<=0)?300:(svc->check_interval*interval_length));</tt> <tt>COMMENT: nagios added the check_interval to preferred_time</tt> <tt>/* make sure we rescheduled the next service check at a valid time */</tt> <tt>get_next_valid_time(preferred_time,&next_valid_time,svc->check_period_ptr);</tt> <tt>COMMENT: No, it didn't do as adding check_interval was not enough to compensate the backshift in time</tt> <tt>/* the service could not be rescheduled properly - set the next check time for next year, but don't</tt> <tt> actually reschedule it */</tt> <tt>if(time_is_valid==FALSE && next_valid_time==preferred_time){</tt> <tt>COMMENT: nagios it bailing out here and just adding 1 year to preferred_time to get the scheduler running again</tt> <tt>svc->next_check=(time_t)(next_valid_time+(60*60*24*365));</tt> <tt>svc->should_be_scheduled=FALSE;</tt> <tt>log_debug_info(DEBUGL_CHECKS,1,"Unable to find any valid times to reschedule the next service check!\n");</tt> <tt> }</tt> <tt>/* this service could be rescheduled... */</tt> <tt> else{</tt> <tt> svc->next_check=next_valid_time;</tt> <tt> svc->should_be_scheduled=TRUE;</tt> <tt> log_debug_info(DEBUGL_CHECKS,1,"Rescheduled next service check for %s",ctime(&next_valid_time));</tt> <tt> }</tt> <tt>}</tt> <tt>COMMENT: BÄNG - our check just got shoved to mars - landing in 1 year and we don't even get</tt> <tt>a notification for it and it does not orphan or whatever...</tt> <tt>The question is now - what's the smartest way to handle this?</tt> <tt>Basically I see 2 different approaches:</tt> <tt>1. When compensating timeshifts - doublecheck that you do not move a check outside its valid check_period</tt> <tt>2. When trying to schedule checks, that somehow ran outside its check_period - try to be smart and look for</tt> <tt>the next valid time inside the check_period of that check instead of just adding check_interval and naivly</tt> <tt>hoping for it to be allright</tt> <tt>Ok, so far from me - /discuss :-)</tt> <tt>S</tt> -- Sascha Runschke IT-Infrastruktur GFKL Financial Services AG Limbecker Platz 1 45127 Essen Telefon : +49 (201) 102-1879 Mobil : +49 (173) 5419665 Fax : +49 (201) 102-1102105 GFKL Financial Services AG Vorstand: Dr. Peter Jänsch (Vors.), Jürgen Baltes, Dr. Till Ergenzinger, Dr. Tom Haverkamp Vorsitzender des Aufsichtsrats: Dr. Georg F. Thoma Sitz: Limbecker Platz 1, 45127 Essen, Amtsgericht Essen, HRB 13522