A NDOUtils replacement : CentreonBroker

[Andreas Ericsson]

Matthieu Kermagoret wrote:
> On Wed, Jul 1, 2009 at 5:17 PM, Andreas Ericsson<ae at op5.se> wrote:
>> Matthieu Kermagoret wrote:
>> This means there are four major contenders for the spot of being the
>> default database broker for Nagios; NDOUtils, Merlin, IDOUtils and
>> CentreonBroker. This is great, as it means the best technical solution
>> will win. At the same time, it's not so great since it means at least
>> four developers are working toward similar goals, but not working
>> together towards that goal.
>>
> 
> I agree with you that there will certainly be some efforts wasted but
> it seems to me that each broker has its own peculariarity (replication
> for Merlin, performance for CentreonBroker, ...).
> 

Well, Merlin handles 80k hosts and 500k services just fine, no matter
which host the checks are being executed on, so performance is hardly
something only CentreonBroker does well ;-)

>>>  - multiple input ports
>>>  - multiple database output
>>>  - configurable logs
>> These sound neat though, although I fail to see how it can be useful to
>> have a single daemon output to more than one database since all databases
>> worth their salt already have replicating capabilities.
>>
>>>  - TLS connection
>> This doesn't parse, or TLS doesn't mean thread-local storage to you.
>>
> 
> I thought that now TLS is more known as the new name of SSL than of
> the name of a low-level feature.
> 

I'm the stupid one here. Sorry for the noise. This sounds fairly
interesting though. I'll have to see how you've implemented this and
see if I can scrounge it for Merlin some day.

>>>  - optimized DB schema
>> While this is good to hear, as the DB schema of the original NDOutils
>> was largely what prevented it from scaling even close to linearly, it
>> also makes me quite sad since there's now 3 separate projects that try
>> to achieve the same thing, and addon authors (like NagVis, Nagios-BP,
>> PNP4Nagios et al) have to add support for all of them if end-users are
>> to retain freedom of choice in the backend.
>>
>> At a first glance, your schema looks remarkably like that of the merlin
>> database (which in turn is derived from the monitor_gui database, which
>> in turn is derived from the nacoma database), with the major difference
>> that you use pluralistic names for your object tables ("hosts" instead
>> of "host") and some different names for some of your columns. Since
>> NagVis, Nagios-BP and PNP4Nagios already supports the Merlin database,
>> perhaps you should have a look and see what would be simple to change
>> to conform to that at least.
>>
> 
> We discovered your project too recently. But that's almost certainly
> what we will try to do in the near future.
> 

Sounds like a good plan. I see your db schema dates back only to June 15
(at least in SVN), so I guess you're not too fixed on it yet :-)

>>>  - multithreaded (use modern processors)
>> This doesn't buy one so much for an event-daemon running outside Nagios
>> as one may think, since Nagios is more than capable of saturating one
>> core by itself. Why was this determined necessary?
>>
> 
> I don't really see what you mean (sorry for my poor english) but I'll
> try to explain why our daemon is multithreaded.
> 
> Merethis is starting to work with some big client installation with
> many pollers. We just want to be able to collect all the event streams
> these pollers are sending at a single point. But there's so much data
> that the single-threaded ndo2db cannot process all events. So we'd
> like to have a broker daemon that can scale on multi-processor
> multi-core touchy machines.
>

It's perfectly possible to handle several hundred poller systems with
a single multiplexing thread though (and never having to worry about
locking is definitely nice). I found that when I started thinking
about implementing Merlin as a multithreaded daemon it just got messy
for the same reasons I couldn't use a networked message-queue to
implement the cross-host event transport mechanism. Either I'd have
multiple threads listening to the remote hosts and then writing and
sending from a single thread (which would defeat the entire purpose
of having a multithreaded app from the start), or I'd have to
maintain lists of where each event has been sent so that the queue
manager could keep track of when it could let one event drop. Some
really rudimentary testing showed that maintaining the lists would
double the operational cost per event.

I was curious to hear your thoughts on this matter. Especially
since you said you support receiving from multiple instances and
also writing to multiple databases from one and the same daemon
instance, which means you should have the same problems I did
and a solution that works for centreonbroker should work for merlin
too.

Ah well. I guess I can just browse the source and find out.

> 
>> Hmm. So long as it's not a drop-in replacement I think it'd be better
>> to announce it as an NDOUtils alternative. Otherwise you'll most likely
>> get a lot of complaints from people when it doesn't work with their
>> ndo database.
>>
> 
> I guess my words were not careful enough. I apologize. It's not yet
> planned to be a NDOUtils replacement, but our intention was to propose
> this as the new basis of NDOUtils developments.
> 

Right.

>>> Despite its name Centreon-Broker
>>> do not rely on Centreon at all. Also, the name can be changed and more
>>> documentation provided. If you're interested you can checkout the code
>>> at http://svn.centreon.com/trunk/centreon-broker or the homepage at
>>> http://forge.centreon.com/projects/show/centreon-broker .
>>>
>>> With this software, we try to address the main NDOUtils drawbacks in a
>>> modern way. As we have a good code start, we would like to share it
>>> with the community and avoid it to reinvent the wheel. So feel free to
>>> tell us what you think about it.
>>>
>> Where can I find the public repository to pull the latest changes from?
>>
> 
> With Subversion you can directly checkout the repository over HTTP :
>  $> svn co http://svn.centreon.com/trunk/centreon-broker
> 

Thanks. I found the link in the original email after I'd sent my first
reply, and I've imported it into git now so I can browse the history
and see how you've thought :-)

Cheers

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

------------------------------------------------------------------------------