Really slow log2ndo

[Andreas Ericsson]

Benjamin Krein wrote:
> I posted the following in the Users mailing list, but got to thinking  
> that maybe Developer's list was a better place for it.
> 
> Am I the only one who is experiencing very slow performance importing  
> archived log files into MySQL with log2ndo?  I'm averaging ~1.5 log  
> entries per second being inserted.  This seems slow to me.
> 
> I've tried dividing the parsing & MySQL service across 2 servers as  
> well as move all the parsing/inserting locally on a remote (to Nagios)  
> server which is much beefier than our actual Nagios server.  My  
> insertion speed remains fairly consistent.
> 
> I'm currently using the Unix socket option.  Is this the wrong one?   
> Would TCP be faster (doesn't seem so)?  Should I process the logs to  
> flat text files first then push those to ndo2db?
> 
> At this rate it's going to take FOREVER!  I have ~10 million archived  
> lines to parse and insert.  Maybe I'm expecting too much?
> 

First thing to try is to disable all indexes on the sql table
while running the import, and then re-enabling them afterwards.


If you're feeling hardcore, you could try first using the
importer from op5 reports-module and then convert the imported
table to ndoutils format.

The reports-module importer has done live imports of 800M lines
of log-data in < 6 minutes. You'll find it at http://www.op5.org/
under the 'projects' section. We have no snapshots available,
but the gitweb can generate one for you on-the-fly if you want.

Note that it'll drop quite a lot though, as it only bothers with
host states, service states, downtime start/stop and program start/
stop.

YMMV.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/