Really slow log2ndo

[Andreas Ericsson]

Mikael Fridh wrote:
> On Wed, Sep 17, 2008 at 7:46 PM, Andreas Ericsson <ae at op5.se> wrote:
>> Benjamin Krein wrote:
>>> Is there a reason log entries aren't split up into multiple fields in
>>> the NDO DB?  It seems kind of silly to put the entire log line in a
>>> single field when there are very clear delimiters in the line.
> 
> These are a few excerpts of the logentries;
> 
> Auto-save of retention data completed successfully.

Junk.

> LOG ROTATION: DAILY

Implementation detail junk.

> CURRENT HOST STATE: bernicla2;UP;HARD;1;PING OK - Packet loss = 0%,
> RTA = 0.50 ms

Useful, but superfluous (especially with LOG_ROTATION: Daily).

> ndomod: Successfully reconnected to data sink!  0 items lost, 88
> queued items to flush.

NEB junk. Will go to separate logfile in a not too far off future.

> CURRENT SERVICE STATE: rt-130-238-128-160-27;PING;OK;HARD;1;PING OK -
> Packet loss = 0%, RTA = 2.77 ms
> 

See comment for CURRENT HOST STATE entry.

> There are only clear delimiters for some types of entries, not all.
> And if you do separate the types into different tables it will stop
> being a Log... A log is where you go to check historically what
> happened during a certain period of time. If you split it up in
> separate tables it will be more complex to get an excerpt on just that
> - all events during a certain period.
> 
>> I imagine the current structure was designed to facilitate displaying
>> the entire log though. Your guess is as good as mine, as it was a long
>> time ago I took a look at the ndoutils code.
> 
> It's a log on disk, thus it's a log in the database. Atleast that's my
> take on it.
> I would use MyISAM MERGE tables or PARTITIONS for this if you want to
> keep an "online" archive of the logs.
> With the MERGE trick you can compress old tables and rejoin them in
> the merge periodically if you'd like.
> 

Or one can just print the logfiles from disk. They're already partitioned
into rotated logfiles so it's not that much of a chore.

>>> I'm contemplating writing my own parser for the archived logs, but I'm
>>> tempted to modify the NDO code to make use of multiple fields.
> 
> You could modify the ndo code to recognize more event types. There are
> a few logentry_type IDs that are duplicate it seems.
> 
> The event history of objects (services and hosts) is already in
> nagios_statehistory, so what else is there in the log that you could
> gain so much from parsing out into separate tables/fields?
> 

Notifications. Program start and stop. Downtime start and stop.
Adding a *raw* log to the database just duplicates the log that's
already saved on disk, so it buys us absolutely nothing while not
taking advantage of the indexing that a database can offer. And
even if the logfiles would ever go away (not likely), it's totally
trivial to concatenate logentries from several tables when one
wants to view them.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/