redis memory increasing linearly

[Francesco Giuseppe Toffoli]

Hi Handers,
thanks for your reply, we found out the problem.
We had a list on which we did an update on a value to reset a spike in 
the value itself. Due to a syntax mistake the LSET command set the field 
in a non standard format, (in the bischeck log we have found an error 
stating the impossibility to perform the purge operation).
For several lists (on which the max size is 1000 points) we had a much 
higher number of points.

We solved purging manually the problematic list with the LTRIM command.
Thanks for your ime,

Regards,
Francesco


Il 04/12/2017 11:24, Anders Håål ha scritto:
>
> It was not clear in your answer, but have you restarted bischeck? You 
> say you have restarted redis, but what about bischeck?
>
> The redis config you restarted with, what did you configure differently?
>
> Anders
>
>
> On 12/04/2017 11:12 AM, Francesco Giuseppe Toffoli wrote:
>> Il 04/12/2017 10:43, Anders Håål ha scritto:
>>>
>>> Hi Francesco,
>>> Redis memory will increase with the number of data points you 
>>> collect and how long time you configured to keep them. So what was 
>>> the memory before it started to grow? Was it steady on this level? 
>>> How much is it growing? And not using the redis instance for 
>>> anything else? (redis-cli info)
>>>
>>
>> Hi Anders, thanks for your fast and really appreciated reply. The 
>> memory has also been stable, around 8/9% of total server available 
>> RAM. Now is around 61% and it grows more or less of +4% a day. So to 
>> be optimist we have a week before everything will crash :-(
>> Redis is used only by bischeck, except for a couple of script reading 
>> datas from it (but are very old and light scripts, so I exclude they 
>> can cause the issue).
>> Here below a screenshot of memory growth:
>> asdasd
>>
>>> It's a good sign that your memory is the same between restarts, 
>>> since that means that the redis persistence works.
>>>
>>> Have you restarted bischeck after the 20 Nov?
>>>
>> Yes we have done some restarts, the last this morning to load a new 
>> redis configuration ( I set logs to debug level).
>>
>>> When it comes to the memory config in redis there are a number of 
>>> strategies to manage memory, but in the bischeck setup redis is used 
>>> as a data store and not as a cache, so you would not like to have 
>>> any eviction polices. Setting max-memory will just make any write 
>>> fail when redis is at its max-memory.
>>> I think its safe to upgrade redis to a later version, but please try 
>>> it first. Currently I run bischeck on 4.0.2 without problem. At the 
>>> same time I do not think this is a redis bug.
>>> To track what goes on in the your redis instance you can check with 
>>> redis-cli. Every servicedefinition becomes its own list in redis. 
>>> You can check the size (llen) of the list and see if its relates to 
>>> your purge configuration. You can also read the first and last 
>>> entities in the list to check the time stamp (lrange). Depending on 
>>> the bischeck config you may also have additional lists for the 
>>> servicedefinition for aggregrations.
>>> /Anders
>>>
>>
>> Ok, i'm writing a script to get, from Redis server, the lenght and 
>> the timestamp of every list writtend by Bischeck. I hope to find out 
>> if there is something growing too much.
>> I'll keep you updated.
>> Thanks for your support.
>>
>> Francesco
>>
>>
>>
>>>
>>> On 12/04/2017 10:11 AM, Francesco Giuseppe Toffoli wrote:
>>>>
>>>> Hi everybody,
>>>> since a coupe of weeks the memory used by Redis in increasing 
>>>> linearly, continuosly.
>>>> We are a bit worried as also after a Redis server restart the 
>>>> situation is the same (the RAM taken is equal before and after the 
>>>> restart).
>>>> I would exclude the issued has been caused by a configuration 
>>>> change, the issue started on 20 November at 11:44PM when nobody was 
>>>> working on bischeck.
>>>> I'm enabling logs in order to see if there is something wrong.
>>>>
>>>> In the meanwhile does anyone have a suggestion on how to proceed?
>>>> Upgrading to Redis 4.0 (stable) can be done maintaining the 
>>>> retro-compatibility with Bischeck? (we have Redis 2.8.23)
>>>>
>>>>
>>>> As a workaround I was thinking to enable the Redis option 
>>>> 'maxmemory' with the 'maxmemory policy': about this last option, as 
>>>> we need all keys to be maintained on database, does anyone know as 
>>>> can be set? (I've read something about key eviction but i still 
>>>> don't have any clear idea).
>>>>
>>>>
>>>> Regards,
>>>> Francesco
>>>>
>>>>
>>>> -- 
>>>>
>>>> Francesco Giuseppe Toffoli
>>>> Monitoring Engineer
>>>>
>>>> GSE Department
>>>>
>>>> Tel: +39 01127387488
>>>>
>>>> Mobile: +39 349.800.60.35
>>>> Email: _ftoffoli at skylogic.it <mailto:ftoffoli at skylogic.it>_
>>>> *
>>>> **Skylogic S. p. A.*
>>>> Strada Pianezza, 289
>>>> 10151 Torino, Italy
>>>>
>>>> This message contains confidential information and is intended only 
>>>> for the individual named. If you are not the named addressee you 
>>>> should not disseminate, distribute or copy this e-mail. Please 
>>>> notify the sender immediately by e-mail if you have received this 
>>>> e-mail by mistake and delete this e-mail from your system. E-mail 
>>>> transmission cannot be guaranteed to be secure or error-free as 
>>>> information could be intercepted, corrupted, lost, destroyed, 
>>>> arrive late or incomplete, or contain viruses. The sender therefore 
>>>> does not accept liability for any errors or omissions in the 
>>>> contents of this message, which arise as a result of e-mail 
>>>> transmission. If verification is required please request a 
>>>> hard-copy version. Please note that any views or opinions presented 
>>>> in this email are solely those of the author and do not necessarily 
>>>> represent those of the Company.
>>>> No employee or agent is authorized to conclude any binding 
>>>> agreement on behalf of this Company nor, through thi s latter, any 
>>>> of the Eutelsat Communication group with another party by email 
>>>> without express written confirmation by a duly authorized officer 
>>>> of the Company. The list of duly authorized officers and the scope 
>>>> of their powers is published on the Trade Register according to the 
>>>> national law of each affiliate.
>>>
>>> -- 
>>>
>>>
>>> Ingby<http://www.ingby.com>
>>>
>>> bischeck - dynamic and adaptive monitoring for Nagios<http://www.bischeck.org>
>>>
>>> anders.haal at ingby.com<mailto:anders.haal at ingby.com>
>>>
>>> Mjukvara genom ingenjörsmässig kreativitet och kompetens
>>>
>>> Ingenjörsbyn
>>> Box 531
>>> 101 30 Stockholm
>>> Sweden
>>> www.ingby.com  <http://www.ingby.com/>
>>> Mobil: +46 70 575 35 46
>>> Tele: +46 75 75 75 090
>>> Fax:  +46 75 75 75 091
>>
>> -- 
>>
>> Francesco Giuseppe Toffoli
>> Monitoring Engineer
>>
>> GSE Department
>>
>> Tel: +39 01127387488
>>
>> Mobile: +39 349.800.60.35
>> Email: _ftoffoli at skylogic.it <mailto:ftoffoli at skylogic.it>_
>> *
>> **Skylogic S. p. A.*
>> Strada Pianezza, 289
>> 10151 Torino, Italy
>>
>> This message contains confidential information and is intended only 
>> for the individual named. If you are not the named addressee you 
>> should not disseminate, distribute or copy this e-mail. Please notify 
>> the sender immediately by e-mail if you have received this e-mail by 
>> mistake and delete this e-mail from your system. E-mail transmission 
>> cannot be guaranteed to be secure or error-free as information could 
>> be intercepted, corrupted, lost, destroyed, arrive late or 
>> incomplete, or contain viruses. The sender therefore does not accept 
>> liability for any errors or omissions in the contents of this 
>> message, which arise as a result of e-mail transmission. If 
>> verification is required please request a hard-copy version. Please 
>> note that any views or opinions presented in this email are solely 
>> those of the author and do not necessarily represent those of the 
>> Company.
>> No employee or agent is authorized to conclude any binding agreement 
>> on behalf of this Company nor, through thi s latter, any of the 
>> Eutelsat Communication group with another party by email without 
>> express written confirmation by a duly authorized officer of the 
>> Company. The list of duly authorized officers and the scope of their 
>> powers is published on the Trade Register according to the national 
>> law of each affiliate.
>
> -- 
>
>
> Ingby<http://www.ingby.com>
>
> bischeck - dynamic and adaptive monitoring for Nagios<http://www.bischeck.org>
>
> anders.haal at ingby.com<mailto:anders.haal at ingby.com>
>
> Mjukvara genom ingenjörsmässig kreativitet och kompetens
>
> Ingenjörsbyn
> Box 531
> 101 30 Stockholm
> Sweden
> www.ingby.com  <http://www.ingby.com/>
> Mobil: +46 70 575 35 46
> Tele: +46 75 75 75 090
> Fax:  +46 75 75 75 091

-- 

Francesco Giuseppe Toffoli
Monitoring Engineer

GSE Department

Tel: +39 01127387488

Mobile: +39 349.800.60.35
Email: _ftoffoli at skylogic.it <mailto:ftoffoli at skylogic.it>_
*
**Skylogic S. p. A.*
Strada Pianezza, 289
10151 Torino, Italy

This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. No employee or agent is authorized to conclude any binding agreement on behalf of this Company nor, through this latter, any of 
 the Eutelsat Communication group with another party by email without express written confirmation by a duly authorized officer of the Company. The list of duly authorized officers and the scope of their powers is published on the Trade Register according to the national law of each affiliate.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-lists.org/archive/bischeck-users/attachments/20171204/d1653318/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bischeck_memory.png
Type: image/png
Size: 14993 bytes
Desc: not available
URL: <https://www.monitoring-lists.org/archive/bischeck-users/attachments/20171204/d1653318/attachment-0001.png>