<div dir="ltr">Hi,<br><br>Been running a naemon/thruk instance since a few years now.<div><br></div><div>As of the last 2 or 3 days my instance is now creating notifications</div><div>for services and hosts that are in scheduled downtime.</div><div><br></div><div>Not obvious that I have changed something.</div><div><br></div><div>Logs are confusing however since they are perfect and do not demonstrate the problem.<br><br><pre class="gmail-code gmail-highlight" lang="plaintext"><span lang="plaintext" class="gmail-line" id="gmail-LC4"># A downtime was entered from 1 second ago to 1 year later.</span> <span lang="plaintext" class="gmail-line" id="gmail-LC5">[1662374724] EXTERNAL COMMAND: SCHEDULE_SVC_DOWNTIME;<a href="http://my.example.org">my.example.org</a>;lbd;1662374723;1693910723;1;0;0;User lxman;roger appstatus destroy : Machine had been created more than 5184000 seconds ago</span> <span lang="plaintext" class="gmail-line" id="gmail-LC6">[1662374724] SERVICE DOWNTIME ALERT: <a href="http://my.example.org">my.example.org</a>;lbd;STARTED; Service has entered a period of scheduled downtime</span> <span lang="plaintext" class="gmail-line" id="gmail-LC7"></span> <span lang="plaintext" class="gmail-line" id="gmail-LC8"># Indeed the downtime reports that the service is now suppressed for notification</span> <span lang="plaintext" class="gmail-line" id="gmail-LC9">[1662374724] SERVICE NOTIFICATION SUPPRESSED: <a href="http://my.example.org">my.example.org</a>;lbd;Notifications about SCHEDULED DOWNTIME events blocked for this object.</span> <span lang="plaintext" class="gmail-line" id="gmail-LC10"></span> <span lang="plaintext" class="gmail-line" id="gmail-LC11"># Service gos bad.</span> <span lang="plaintext" class="gmail-line" id="gmail-LC12">[1662375882] SERVICE ALERT: <a href="http://my.example.org">my.example.org</a>;lbd;CRITICAL;SOFT;1;SNMP CRITICAL - *-1*</span> <span lang="plaintext" class="gmail-line" id="gmail-LC13">[1662376182] SERVICE ALERT: <a href="http://my.example.org">my.example.org</a>;lbd;CRITICAL;SOFT;2;SNMP CRITICAL - *-1*</span> <span lang="plaintext" class="gmail-line" id="gmail-LC14">[1662376482] SERVICE ALERT: <a href="http://my.example.org">my.example.org</a>;lbd;CRITICAL;HARD;3;SNMP CRITICAL - *-1*</span> <span lang="plaintext" class="gmail-line" id="gmail-LC15"></span> <span lang="plaintext" class="gmail-line" id="gmail-LC16"># Indeed now at hard state and service is logged as notification suppressed as you would expect.</span> <span lang="plaintext" class="gmail-line" id="gmail-LC17">[1662376482] SERVICE NOTIFICATION SUPPRESSED: <a href="http://my.example.org">my.example.org</a>;lbd;Notification blocked for object currently in a scheduled downtime.</span> <span lang="plaintext" class="gmail-line" id="gmail-LC18"></span> <span lang="plaintext" class="gmail-line" id="gmail-LC19">1662376482 = Monday, 5 September 2022 13:14:42</span> <span lang="plaintext" class="gmail-line" id="gmail-LC20"></span> <span lang="plaintext" class="gmail-line" id="gmail-LC21"></span> <span lang="plaintext" class="gmail-line" id="gmail-LC22">However a notification was totally sent out at this time.</span> <span lang="plaintext" class="gmail-line" id="gmail-LC23"></span> <span lang="plaintext" class="gmail-line" id="gmail-LC24">```</span> <span lang="plaintext" class="gmail-line" id="gmail-LC25">:fire: __PROBLEM__ <a href="http://my.example.org/lbd">my.example.org/lbd</a> is CRITICAL for 0d 0h 10m 1s. , lnodes/login, production, S513-C-VM960.</span> <span lang="plaintext" class="gmail-line" id="gmail-LC26"></span> <span lang="plaintext" class="gmail-line" id="gmail-LC27">SNMP CRITICAL - *-1* [Nag :link:](<a href="https://cernnag.example.ch/thruk/cgi-bin/extinfo.cgi?type=2&host=my.example.org&service=lbd">https://cernnag.example.ch/thruk/cgi-bin/extinfo.cgi?type=2&host=my.example.org&service=lbd</a>), [Monit :link:](<a href="https://monit-grafana.example.ch/d/RwtmMDXmz/single-host-metrics?orgId=1&var-hostname=my.example.org">https://monit-grafana.example.ch/d/RwtmMDXmz/single-host-metrics?orgId=1&var-hostname=my.example.org</a>), [Fore :link:](<a href="https://judy.example.ch/hosts/my.example.org">https://judy.example.ch/hosts/my.example.org</a>), [SSH :link:](ssh://<a href="mailto:root@my.example.org">root@my.example.org</a>)</span> <span lang="plaintext" class="gmail-line" id="gmail-LC28">```</span> </pre>Service appears as in downtime on thruk interface. There is no naemon.log entry for that notification that went out.<br><br>Only recent action was during some network instability of the naemon server itself I hit 'Disable all notifications' and then 'Enable all notifications'<br>That said I have tried to remove all histroy in the service by stopping nameon and cleaning up all these files.<br>* /var/lib/naemon/ objects.cache, retention.cache, status.dat.<br>* /var/log/naemon naemon.log archives/*<br><br>Any idea why notifications might still be being sent.<br><br></div><div>Versions on CentOS 7:</div><div>rpm -q naemon thruk naemon-livestatus<br>naemon-1.3.1-0.noarch<br>thruk-2.48.3-11458.1.x86_64<br>naemon-livestatus-1.3.1-0.x86_64<br></div><div><br></div><div>Many Thanks</div><div><br></div><div>Steve.<br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Steve Traylen<br></div></div></div>