[naemon-dev] Ideas about future features

[Robin Sonefors]

On 2013-12-27 02:06, Matthias Eble wrote:
> 1) have a feature to monitor per metric rather than per check_command.
>     * Today, many plugins check lots of things.
>     * Typical example is check_disk, check_snmp.
>     * Depending on the configuration method, acknowledging a problem
> with /mytmpmount also disables notifications for /var
>       * To fix that, we'd need to create a stricter plugin output
> standard that contains per-metric status codes.
>          * metrics would be /mytmpmount-freespace, TCP-response-time,
> http-status-code or http-match-string
>       * The core would need to create sub-services at run-time and
> populate their results.
>       * Benefit: per metric actions and logging. Especially per metric
> downtime and acknowledgements
>       * Maybe it could also be used for receiving snmp traps or log
> pattern matching checks
>          * different alerting for different patterns/traps
>
>     * Today, many folks wrap the plugin call and submit results to a
> passive check.
>        * works, but all possible services need to be in the config.
>        * That's where folks start generating nagios configs and reload
> the daemon.
>        * Is that what we want? Maybe?
>           * problems arise when there are syntax problems
>
>     * raw proposal:
>    define service {
>    ...
>        check_command  check_disk
>      contact_group  os_admins
>      define metric {
>          metric_name  ^/oracle.*
>          contact_group oracle_admins
>      }
>    }
>
>     * maybe another layer could be added for check_multi-like plugins.
>        * but they could also be forced to structure metric names

I've been thinking about plugins and plugin architecture a bit.

The nagiosplugins project is talking about a new threshold format - 
https://www.nagios-plugins.org/doc/new-threshold-syntax.html - to 
achieve the same thing you want to solve in-core.

I think the nagiosplugins approach - basically, update all plugins to 
support a much more complex (though easier to understand) threshold 
format, because the old one was too complicated - is wrong. Programmers 
write buggy code, and telling programmers to write more code leads to 
more bugs (or at least I write buggy code, and I'm too stupid to write 
plugins already - that's why I stick to the core :P )

But I'm also not sure how far into the core the I'd want to put it. What 
if we, instead of either change the core or the plugins, write a plugin 
wrapper that takes a threshold as described by nagiosplugins and a 
plugin command line? It would simply parse the perfdata from the plugin, 
the threshold from the CLI, throw away the plugin exit code, and send a 
new, "imploved" exit code and stdout to naemon?

I feel this plugin wrapper approach would take the least amount of work 
to implement. Which problems would it leave unsolved?

 > [snip]
> What do you think? What's the focus of the dev-team?

So far, it seems the focus is mostly on cleanup. There's just *so* 
*much* ancient *crap* lying around. Tens of thousands of lines of code 
to create a ugly, useless web UI, which force me to ifdef every second 
line - what? Three (or so, I lost count) different configuration parsers 
for near-identical-yet-subtly-different configuration file formats - 
really? And then the amount of special casing for things that you might 
expect to behave similarly until you find out the hard way that they 
really don't - for instance, I always thought the flapping calculation 
was based on the last 20 (or so) check results, but nooo: 
https://github.com/naemon/naemon-core/blob/master/naemon/flapping.c#L116 
I think the current score is something like -120k lines compared to the 
initial code import, but there's a lot more we could do.

Oh, and testing. One of the scariest things when starting on a new 
codebase is realizing that there are no tests at all. The only thing 
worse than that is finding a directory full of tests - granted, all 
covered in cobweb and dust - and you think (or hope, or whatever you 
call that feeling when you know you must never assume good things but 
still want to) that you might have found The Book of Shadows in the 
attic, but after dusting the code off and flipping through it, it turns 
out that nobody has executed (or even compiled) any code here for 
*years*, and half the tests test features that doesn't even exist 
anymore. It dawns on you that somebody spent days - weeks, even - 
writing tests to avoid regressions - and then didn't run the test and 
thus didn't catch the regressions. See: t-tap, where a few of the tests 
files actually work, and none of them have a working build system ATM.

As far as I'm going to go in terms of longer-term vision and 
the-way-to-go-iness, I'd like to modularize the crap out of the core. 
The nagios "core" is anything but, as explained above. It would be neat 
to lift out a bunch of nagios functionality into a bundle of 
preinstalled modules. This would serve two purposes: it would force us 
to dogfood the broker API and thus help us improve it, and it would 
compartmentalize features (new, and old) to avoid weird interactions 
with other features.

The broker API as it exists is terrible - you're just given all of the 
naemon internals, spotty and inconsistent hooks, and a "good luck". This 
means that, as a core developer, any change I make at all is bound to 
break some module, while as a module author, I need to learn all of the 
core to write a module. And you want to store your own add-on 
configuration/data? Hah! So, in the end, it's just easier to become a 
core contributor, because who has the time not to?

What would happen if, to take an example that sounds weird but makes 
some kind of sense, the flapping functionality was a module? That would 
require some extra module functionality - modules would have to be able 
to add configuration statements to the config (global and per-object) 
for configuring flapping thresholds, and modules would have to be able 
to couple state (is_flapping, last 20 check results) with the object and 
have it persist between restarts. Now, what if this was the easiest, 
most concise, and easiest-to-find-out-how way to do it?

I think a module should be able to do all these things - and if it could 
do that, and if flapping was a module, I would not ever again have to 
worry about flapping in the remaining core, nor would I wonder where all 
special cases for flapping are handled - heck, I could even see if the 
flapping feature has tests and how extensive they are, just from looking 
at github.com/naemon/flapping ! Today, almost all features - including 
flapping - is handled by the pair of ogres known as 
handle_async_service_check_result/handle_async_host_check_result - 
looking at the code, I have no idea what it will actually end up doing 
for each case, but I'm quite sure a few of the code paths are buggy - 
because that many untested if conditions just aren't going to all be 
correct. Modularizing away the if statements (all of them, all over the 
core) should render a more consistent, less buggy monitoring solution.

tl;dr: naemon should allow contributors to write modules that are much 
more powerful than today's broker modules, to make it possible and easy 
to write a module to add seemingly built-in functionality, like metrics 
and exceptions - then, we could start to write such modules, go crazy, 
and see what comes out!