<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40"> <head> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"> <meta name=ProgId content=Word.Document> <meta name=Generator content="Microsoft Word 10"> <meta name=Originator content="Microsoft Word 10"> <link rel=File-List href="cid:filelist.xml@01C29537.33AC1250"> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="time"/>  <style>  </style>  </head> <body lang=EN-US link=blue vlink=purple style='tab-interval:.5in'> <div class=Section1> Hello All,<o:p></o:p> <o:p> </o:p> Is anyone out there successfully using Nagios with a Postgres backend for a large installation under RedHat (7.3)? I have been trying like heck to get it to work nicely and I haven’t had much luck. Here are the numbers:<o:p></o:p> <o:p> </o:p> Central Server #1:<o:p></o:p> Nagios 1.0b6 using a Postgres 7.2.3 (thanks to the recent postgres timestamp patch) (also tried 1.0b3 with pg7.1 and pg7.2.3)<o:p></o:p> Redhat 7.3 on a quad processor PII 550 with 2.5 Gb RAM, 200+ Gb of hardware Raid 5 disk space.<o:p></o:p> <o:p> </o:p> Central Server #2:<o:p></o:p> Nagios 1.0b3 using flat text file backend.<o:p></o:p> Redhat 6.2 on a Uni-processor PIII 800 with 512 Mb Ram and ~12 GB of disk space (non-raid)<o:p></o:p> <o:p> </o:p> Neither of the above servers are performing active checks.<o:p></o:p> <o:p> </o:p> I have 4 Data Collector machines each polling and reporting via NSCA to both servers above for 500 services (2000 total) with a 5 minute check interval for each service.<o:p></o:p> <o:p> </o:p> Here is what I am seeing on the machine with the postgres backend:<o:p></o:p> <o:p> </o:p> <st1:time Hour="9" Minute="21">9:21am</st1:time> up <st1:time Hour="22" Minute="8">22:08</st1:time>, 6 users, load average: 53.54, 726.96, 844.05<o:p></o:p> 14336 processes: 14334 sleeping, 2 running, 0 zombie, 0 stopped<o:p></o:p> CPU0 states: 44.5% user, 5.11% system, 0.0% nice, 49.7% idle<o:p></o:p> CPU1 states: 26.5% user, 10.5% system, 0.0% nice, 63.1% idle<o:p></o:p> CPU2 states: 14.3% user, 2.3% system, 0.0% nice, 83.5% idle<o:p></o:p> CPU3 states: 24.7% user, 34.10% system, 0.0% nice, 40.7% idle<o:p></o:p> Mem: 2582276K av, 2572180K used, 10096K free, 0K shrd, 66476K buff<o:p></o:p> Swap: 2097112K av, 4K used, 2097108K free 1062052K cached<o:p></o:p> <o:p> </o:p> At this point the machine is of course only somewhat usable. The other machine, accepting the exact same passive service checks looks like this:<o:p></o:p> <o:p> </o:p> <st1:time Hour="9" Minute="25">9:25am</st1:time> up 34 days, <st1:time Hour="20" Minute="19">20:19</st1:time>, 3 users, load average: 1.94, 2.15, 2.07<o:p></o:p> 149 processes: 147 sleeping, 2 running, 0 zombie, 0 stopped<o:p></o:p> CPU states: 0.0% user, 0.6% system, 0.0% nice, 0.2% idle<o:p></o:p> Mem: 516816K av, 265816K used, 251000K free, 230400K shrd, 25960K buff<o:p></o:p> Swap: 530104K av, 9840K used, 520264K free 144908K cached<o:p></o:p> <o:p> </o:p> My major concern with the first machine is that all but about 80 of those 14336 processes are nagios and nsca, with the majority being nsca. That number continues to increase until I have to reboot the machine. Using strace and lsof, it appears that the nsca processes are all waiting to write to nagios.cmd but the multiple nagios processes also appear to be waiting on or writing to a pipe as well. I’m trying to understand to overall process… I understand that nsca must wait until the 4k pipe is cleared before writing more data to it, but I don’t understand the 100 or so nagios processes that also appear to be spawned by the master process. Do each of those also read commands from nagios.cmd and then attempt to insert those into the DB?<o:p></o:p> <o:p> </o:p> Here are the optimizations I have done on the central server:<o:p></o:p> <o:p> </o:p> Nagios – <o:p></o:p> command_check_interval=-1<o:p></o:p> max_concurrent_checks=150 ; does this even apply?<o:p></o:p> service_reaper_frequency=2<o:p></o:p> service_check_timeout=60<o:p></o:p> host_check_timeout=30<o:p></o:p> event_handler_timeout=30<o:p></o:p> notification_timeout=30<o:p></o:p> ocsp_timeout=5<o:p></o:p> perfdata_timeout=5<o:p></o:p> execute_service_checks=0<o:p></o:p> accept_passive_service_checks=1<o:p></o:p> process_performance_data=0<o:p></o:p> aggregate_status_updates=1<o:p></o:p> status_update_interval=15<o:p></o:p> <o:p> </o:p> Postgres –<o:p></o:p> tcpip_socket = true<o:p></o:p> max_connections = 100<o:p></o:p> port = 5432 <o:p></o:p> hostname_lookup = false<o:p></o:p> show_source_port = false<o:p></o:p> shared_buffers = 200 # 2*max_connections, min 16<o:p></o:p> max_fsm_relations = 100 # min 10, fsm is free space map<o:p></o:p> max_fsm_pages = 10000 # min 1000, fsm is free space map<o:p></o:p> max_locks_per_transaction = 64 # min 10<o:p></o:p> wal_buffers = 24 # min 4<o:p></o:p> sort_mem = 49152 # min 32<o:p></o:p> vacuum_mem = 49152 # min 1024<o:p></o:p> effective_cache_size = 30000 # default in 8k pages<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Linux - <o:p></o:p> kernel.shmmax = 536870912<o:p></o:p> kernel.shmall = 536870912<o:p></o:p> ulimit -u 15000<o:p></o:p> ulimit -n 3000<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Does anyone have any suggestions? I would appreciate any help I can get.<o:p></o:p> <o:p> </o:p> Thanks,<o:p></o:p> <o:p> </o:p> Marc<o:p></o:p> </div> </body> </html>