Large scale installation

[Andreas Ericsson]

On 06/12/2012 03:59 AM, Ian Orszaczki wrote:
> Great advice.  Funny you should mention status.dat in ramdisk as we have
> hit a hiccup this morning which has meant we have lost comments and
> downtimes.
> 
> We had moved status.dat to a ramdisk as recommended for large installations
> (we monitoring 3390 hosts with 18748 services from one server, latencies
> below 2 secs and load under 2) but after running out of open files the
> status.dat was zero'd.
> 
> 
> As an extreme hack I ran a quick script across the output of -
> # grep EXTERNAL nagios.log | grep ACK | cut -c57->  /tmp/acks.txt
> 
> Script -
>   #!/bin/sh
>   # This is a sample shell script showing how you can submit the
> ACKNOWLEDGE_HOST_PROBLEM command
>   # to Nagios.  Adjust variables to fit your environment as necessary.
>   now=`date +%s`
>   commandfile='/app/nagios/var/rw/nagios.cmd'
>   cat /tmp/acks.txt | while read line
>   do
>           echo $line
>           /usr/bin/printf "[%lu] $line\n" $now>  $commandfile
>   done
> 
> Therefore I am going to move status.dat back onto the localdisk (luckily
> SSD drives) so that we can at least restore from a recent backup. I will
> probably also create valid copy, along with retention.dat, every hour to
> enable quick recovery. And yes, I have increased the process and open files
> limits for the nagios user.
> 
> Am I missing anything obvious>
> 

You're using embedded perl. Nagios doesn't leak at all without that.
If you can do some valgrind running of it I could take a look at the
output and see if there's anything problematic that can actually be
fixed, as opposed to something in the perl library.

> 
> On Tue, Jun 12, 2012 at 5:40 AM, Giorgio Zarrelli<zarrelli at linux.it>  wrote:
> 
>> Hi,
>>
>> I suggest to review your installation. Try with the large installation
>> tweaks http://nagios.sourceforge.net/docs/3_0/largeinstalltweaks.html.
>>
>> Then, check whether you need all your checks at 5 mins or you can move
>> some of them to 10 mins pace.
>>
>> Then, review your check plugins: Perl plugins eat more memory and CPU
>> cycles then C compiled checks. If they support EPN
>> http://nagios.sourceforge.net/docs/3_0/embeddedperl.html, use it, it makes
>> your plugin faster and lighter.
>>
>> Then, check your checks. Some checks return data slower then others. Let's
>> say, SNMP checks are not lightning fast.
>>
>> Then, check your graphs. Graphing perfdata takes CPU cycles and uses
>> memory. Do you need all your graphs?
>>
>> Then, get rid of NDOUtils. They are chocking all the way, not efficient,
>> clumsy, old and heavy. If you want to store your data in MySQL, use Merlin
>> instead.
>>
>> Anyway, did you tune your MySQL? Is it causing too much I/O? Is it
>> munching too much RAM or CPU cycles?
>>
>> Did you tune your Apache or http server? Does it cope with your needs? Is
>> it munching too much RAM or CPU cycles?
>>
>> If you want live infos about your hosts and services, let's say to use
>> with Navis, grab MKlive: it's blazing fast and gives you access to the
>> core Nagios process.
>>
>> Are you using a virtualized environment? If so, remember that I/O layer in
>> virtualized environments has a poor performance, use fast and real disks
>> and your I/O will drop dramatically.
>>
>> Try to move status.dat to /dev/shm. The latter is a ram disk ready to use
>> and writing in ram is always faster then writing on disk.
>>
>> Avoid logging too much, it increases I/O and takes CPU and RAM.
>>
>> What iotop and iostat are telling you?
>>
>> What do you see in top or htop?
>>
>> If you can or wish, compile all from sources, it will go faster on your
>> system.
>>
>> You can use passive checks with NSCA or NRDP to reduce load, even though I
>> do not like them a lot.
>>
>> These are just few ideas that came to my mind.
>>
>>
>> Let's talk about sharing load.
>>
>> You can use different methods:
>>
>> Merlin
>> (http://www.op5.org/community/plugin-inventory/op5-projects/merlin): gives
>> you loadbalancing and redundancy. I use it for Ninja, never used for load
>> balancing and redundancy.
>>
>> DNX (http://dnx.sourceforge.net/): Something new, it's gaining momentum,
>> good to offload the checks. Worth to give a try.
>>
>> Mod_gearman (http://labs.consol.de/lang/de/nagios/mod-gearman/): Love at
>> first site :-) Easy, powerful, load balancing and fault tolerant. Compile
>> gearmand with memcached support and all the result checks will go directly
>> to ram, avoiding I/O on disk. It's really simple to setup, if one of the
>> workers go down, the others will share its work. Be careful: security is a
>> problem, there is not a good auth system, but using a VPN will solve the
>> problem. Efficient, I use a virtual machine with 2 cores and 2 gb of ram
>> to make about 5K checks. And the load is not a concern. You need more
>> horse power? Add a worker. You have some checks timing out due to poor
>> connections to the targets? Put a worker close to the target, but be
>> careful, the timing, let's say the rta of a ping, will be from the worker
>> perspective.
>>
>> Well, hope it helps.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Nagios-users mailing list
>> Nagios-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nagios-users
>> ::: Please include Nagios version, plugin version (-v) and OS when
>> reporting any issue.
>> ::: Messages without supporting info will risk being sent to /dev/null
>>
> 
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 
> 
> 
> _______________________________________________
> Nagios-users mailing list
> Nagios-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null


-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null