<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-15"> </head> <body text="#000000" bgcolor="#FFFFFF"> Great to here that you solved it. I was pretty sure that redis was not your problem, and glad that it was not bischeck. :) /Anders <div class="moz-cite-prefix">On 12/04/2017 12:26 PM, Francesco Giuseppe Toffoli wrote: </div> <blockquote type="cite" cite="mid:c54b35f8-55bf-4250-c483-48e4bfbea28c@skylogic.it"> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-15"> Hi Handers, thanks for your reply, we found out the problem. We had a list on which we did an update on a value to reset a spike in the value itself. Due to a syntax mistake the LSET command set the field in a non standard format, (in the bischeck log we have found an error stating the impossibility to perform the purge operation). For several lists (on which the max size is 1000 points) we had a much higher number of points. We solved purging manually the problematic list with the LTRIM command. Thanks for your ime, Regards, Francesco <div class="moz-cite-prefix">Il 04/12/2017 11:24, Anders Håål ha scritto: </div> <blockquote type="cite" cite="mid:20b32b87-4a72-cd11-50a9-e5129362b213@ingby.com"> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-15"> It was not clear in your answer, but have you restarted bischeck? You say you have restarted redis, but what about bischeck? The redis config you restarted with, what did you configure differently? Anders <div class="moz-cite-prefix">On 12/04/2017 11:12 AM, Francesco Giuseppe Toffoli wrote: </div> <blockquote type="cite" cite="mid:c313d2e4-55fd-c1b8-2776-c011be3d4170@skylogic.it"> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-15"> Il 04/12/2017 10:43, Anders Håål ha scritto: <blockquote type="cite" cite="mid:b7862bea-57da-ee27-c25b-9f0fd635c8e2@ingby.com"> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-15"> Hi Francesco, Redis memory will increase with the number of data points you collect and how long time you configured to keep them. So what was the memory before it started to grow? Was it steady on this level? How much is it growing? And not using the redis instance for anything else? (redis-cli info) </blockquote> Hi Anders, thanks for your fast and really appreciated reply. The memory has also been stable, around 8/9% of total server available RAM. Now is around 61% and it grows more or less of +4% a day. So to be optimist we have a week before everything will crash :-( Redis is used only by bischeck, except for a couple of script reading datas from it (but are very old and light scripts, so I exclude they can cause the issue). Here below a screenshot of memory growth: <img moz-do-not-send="false" src="cid:part1.B6253CE3.7F5301B0@ingby.com" alt="asdasd" class="" width="844" height="260"> <blockquote type="cite" cite="mid:b7862bea-57da-ee27-c25b-9f0fd635c8e2@ingby.com"> It's a good sign that your memory is the same between restarts, since that means that the redis persistence works. Have you restarted bischeck after the 20 Nov? </blockquote> Yes we have done some restarts, the last this morning to load a new redis configuration ( I set logs to debug level). <blockquote type="cite" cite="mid:b7862bea-57da-ee27-c25b-9f0fd635c8e2@ingby.com"> When it comes to the memory config in redis there are a number of strategies to manage memory, but in the bischeck setup redis is used as a data store and not as a cache, so you would not like to have any eviction polices. Setting max-memory will just make any write fail when redis is at its max-memory. I think its safe to upgrade redis to a later version, but please try it first. Currently I run bischeck on 4.0.2 without problem. At the same time I do not think this is a redis bug. To track what goes on in the your redis instance you can check with redis-cli. Every servicedefinition becomes its own list in redis. You can check the size (llen) of the list and see if its relates to your purge configuration. You can also read the first and last entities in the list to check the time stamp (lrange). Depending on the bischeck config you may also have additional lists for the servicedefinition for aggregrations. /Anders </blockquote> Ok, i'm writing a script to get, from Redis server, the lenght and the timestamp of every list writtend by Bischeck. I hope to find out if there is something growing too much. I'll keep you updated. Thanks for your support. Francesco <blockquote type="cite" cite="mid:b7862bea-57da-ee27-c25b-9f0fd635c8e2@ingby.com"> <div class="moz-cite-prefix">On 12/04/2017 10:11 AM, Francesco Giuseppe Toffoli wrote: </div> <blockquote type="cite" cite="mid:ced8dbf3-98fa-41a6-e92e-055739a62bed@skylogic.it"> <meta http-equiv="content-type" content="text/html; charset=iso-8859-15"> Hi everybody, since a coupe of weeks the memory used by Redis in increasing linearly, continuosly. We are a bit worried as also after a Redis server restart the situation is the same (the RAM taken is equal before and after the restart). I would exclude the issued has been caused by a configuration change, the issue started on 20 November at 11:44PM when nobody was working on bischeck. I'm enabling logs in order to see if there is something wrong. In the meanwhile does anyone have a suggestion on how to proceed? Upgrading to Redis 4.0 (stable) can be done maintaining the retro-compatibility with Bischeck? (we have Redis 2.8.23) As a workaround I was thinking to enable the Redis option 'maxmemory' with the 'maxmemory policy': about this last option, as we need all keys to be maintained on database, does anyone know as can be set? (I've read something about key eviction but i still don't have any clear idea). Regards, Francesco <div class="moz-signature">-- <meta http-equiv="content-type" content="text/html; charset=iso-8859-15"> <title></title> Francesco Giuseppe Toffoli Monitoring Engineer GSE Department Tel: +39 01127387488<o:p></o:p> Mobile: +39 349.800.60.35 Email: <a href="mailto:ftoffoli@skylogic.it" moz-do-not-send="true">ftoffoli@skylogic.it</a> <img shrinktofit="true" id="_x0000_i1025" src="imap://aha%40ingby%2Ecom@mail.messagingengine.com:993/fetch%3EUID%3E.INBOX.Drafts%3E5796?part=&filename=/home/franz/skylogic_logo.jpeg" moz-do-not-send="true" width="190" border="0" height="44"> Skylogic S. p. A. Strada Pianezza, 289 10151 Torino, Italy </div> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. No employee or agent is authorized to conclude any binding agreement on behalf of this Company nor, through thi s latter, any of the Eutelsat Communication group with another party by email without express written confirmation by a duly authorized officer of the Company. The list of duly authorized officers and the scope of their powers is published on the Trade Register according to the national law of each affiliate. </blockquote> <pre class="moz-signature" cols="72">-- Ingby <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com" moz-do-not-send="true"><http://www.ingby.com></a> bischeck - dynamic and adaptive monitoring for Nagios <a class="moz-txt-link-rfc2396E" href="http://www.bischeck.org" moz-do-not-send="true"><http://www.bischeck.org></a> <a class="moz-txt-link-abbreviated" href="mailto:anders.haal@ingby.com" moz-do-not-send="true">anders.haal@ingby.com</a><a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com" moz-do-not-send="true"><mailto:anders.haal@ingby.com></a> Mjukvara genom ingenjörsmässig kreativitet och kompetens Ingenjörsbyn Box 531 101 30 Stockholm Sweden <a class="moz-txt-link-abbreviated" href="http://www.ingby.com" moz-do-not-send="true">www.ingby.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com/" moz-do-not-send="true"><http://www.ingby.com/></a> Mobil: +46 70 575 35 46 Tele: +46 75 75 75 090 Fax: +46 75 75 75 091 </pre> </blockquote> <div class="moz-signature">-- <meta http-equiv="content-type" content="text/html; charset=iso-8859-15"> <title></title> Francesco Giuseppe Toffoli Monitoring Engineer GSE Department Tel: +39 01127387488<o:p></o:p> Mobile: +39 349.800.60.35 Email: <a href="mailto:ftoffoli@skylogic.it" moz-do-not-send="true">ftoffoli@skylogic.it</a> <img shrinktofit="true" id="_x0000_i1025" src="imap://ftoffoli@imap.skylogic.it:143/fetch%3EUID%3E/INBOX%3E233968?part=1.2&filename=/home/franz/skylogic_logo.jpeg" moz-do-not-send="true" width="190" border="0" height="44"> Skylogic S. p. A. Strada Pianezza, 289 10151 Torino, Italy </div> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. No employee or agent is authorized to conclude any binding agreement on behalf of this Company nor, through thi s latter, any of the Eutelsat Communication group with another party by email without express written confirmation by a duly authorized officer of the Company. The list of duly authorized officers and the scope of their powers is published on the Trade Register according to the national law of each affiliate. </blockquote> <pre class="moz-signature" cols="72">-- Ingby <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com" moz-do-not-send="true"><http://www.ingby.com></a> bischeck - dynamic and adaptive monitoring for Nagios <a class="moz-txt-link-rfc2396E" href="http://www.bischeck.org" moz-do-not-send="true"><http://www.bischeck.org></a> <a class="moz-txt-link-abbreviated" href="mailto:anders.haal@ingby.com" moz-do-not-send="true">anders.haal@ingby.com</a><a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com" moz-do-not-send="true"><mailto:anders.haal@ingby.com></a> Mjukvara genom ingenjörsmässig kreativitet och kompetens Ingenjörsbyn Box 531 101 30 Stockholm Sweden <a class="moz-txt-link-abbreviated" href="http://www.ingby.com" moz-do-not-send="true">www.ingby.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com/" moz-do-not-send="true"><http://www.ingby.com/></a> Mobil: +46 70 575 35 46 Tele: +46 75 75 75 090 Fax: +46 75 75 75 091 </pre> </blockquote> <div class="moz-signature">-- <meta http-equiv="content-type" content="text/html; charset=iso-8859-15"> <title></title> Francesco Giuseppe Toffoli Monitoring Engineer GSE Department Tel: +39 01127387488<o:p></o:p> Mobile: +39 349.800.60.35 Email: <a href="mailto:ftoffoli@skylogic.it" moz-do-not-send="true">ftoffoli@skylogic.it</a> <img shrinktofit="true" id="_x0000_i1025" src="imap://ftoffoli@imap.skylogic.it:143/fetch%3EUID%3E/INBOX%3E233968?part=1.2&filename=/home/franz/skylogic_logo.jpeg" moz-do-not-send="true" width="190" border="0" height="44"> Skylogic S. p. A. Strada Pianezza, 289 10151 Torino, Italy </div> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. No employee or agent is authorized to conclude any binding agreement on behalf of this Company nor, through thi s latter, any of the Eutelsat Communication group with another party by email without express written confirmation by a duly authorized officer of the Company. The list of duly authorized officers and the scope of their powers is published on the Trade Register according to the national law of each affiliate. </blockquote> <pre class="moz-signature" cols="72">-- Ingby <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com"><http://www.ingby.com></a> bischeck - dynamic and adaptive monitoring for Nagios <a class="moz-txt-link-rfc2396E" href="http://www.bischeck.org"><http://www.bischeck.org></a> <a class="moz-txt-link-abbreviated" href="mailto:anders.haal@ingby.com">anders.haal@ingby.com</a><a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com"><mailto:anders.haal@ingby.com></a> Mjukvara genom ingenjörsmässig kreativitet och kompetens Ingenjörsbyn Box 531 101 30 Stockholm Sweden <a class="moz-txt-link-abbreviated" href="http://www.ingby.com">www.ingby.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com/"><http://www.ingby.com/></a> Mobil: +46 70 575 35 46 Tele: +46 75 75 75 090 Fax: +46 75 75 75 091 </pre> </body> </html>