Patch submission for comments : CGI speed improvement (XNG)

[Fran=?utf-8?B?w6c=?=ois Laupretre]

> Yes and no. The comments were in french, much of the code wasn't 
> indented and mnemonic identifiers doesn't appear to be in the 
> programming style you use, so it took a while to grok all of it.

Right. The only thing I wanted was to know if it could be interesting from
your point of view. I didn't expect people to look at the code. Actually, I
wanted to know if it was worth the pain to cleanup the code, to comment it,
and to write a documentation. I spent more than one week on this code and,
before going on, I wanted to submit an alpha version.

And if you prefer a different indenting style, it is not a problem.
 
> A simpler solution to the same problem would be to implement real 
> hashing instead of the algorithm in current use. Presently it 
> generates 
> worse hashes than an internet checksum function and the 
> compare_hashdata* functions (called from find_*) are 
> extremely expensive.

I agree but I saw it as another issue. Of course, the current pseudo-hashing
system is not very efficient but, until it gets suppressed, it is better to
transmit the hash tables to the CGIs instead of having them rebuild them.
There are also too many mallocs in the current nagios code, and several
memory leaks. But, once again, I tried to find the best ratio of improvement
versus quantity of changes in code.

> nagios-db isn't the ultimate solution, although using a 
> database is far superior to flat-files (regardless of format).

Certainly for the version where CGIs are replaced by some PHP code. And,
when it is time to work on it, I will be happy to participate. But I have a
problem now, and I need a solution before one or two years. If you tell me
that this version 3 is planned for 3rd quarter 2005, it is OK and I won't
try to add such code.

Yes, storing information in a database can be better as an alternative, but
I am not sure that the flat file option will disappear soon.

> I'm not so sure about this patch you've proposed though. By 
> necessity it 
> involves a fair deal of pointer voodoo. Code such as that is 
> always hell 
> to maintain, and it gets increasingly difficult to ensure support for 
> various compilers and architectures. F.e., gcc aligns components of 
> datastructures on sizeof(register) boundaries with certain 
> optimization 
> options, and squeezes them with others, making pointer-hopping using 
> anything but relative offsets a debugging nightmare. Other compilers 
> have similar symptoms.

Yes. This is the drawback. The current structure of the memory used by the
core makes it quite complex to serialize. Let me just add that we could
assume that the core and CGI code are compiled on the same machine by the
same compiler. I know that the code is complex and I already spent much
debugging time on it. Actually, the really complex part of the code is in
serialize_object_section(). As I wanted to keep it small, I did a generic
function, which takes an object description as parameter. This way, the code
is twice more complex, but 14 times smaller :-).

Another thing is the serialize/unserialize functions for status data. If the
program status data used a structure, and if the core and the CGIs used
common structures to store the host & service status data, this code could
use the object generic code, which would suppress nearly everything from
xsdng.c.

And, another idea, involving more changes, can be to reorganize all these
structures in the core after the configuration is read from the obj cfg
files. At this time, except for strings that would be treated as individual
mallocs, we know how much size we need for the structures, we can allocate a
single block of memory for them, and we can replace the linked lists with
arrays, even for the sub-object lists. The code needed to do that is nearly
the same as serialize_object_section() but, there, it would be used only
once. It would provide a much more efficient way of managing the lists,
would suppress nearly every pointer manipulation, and would maximize the
serialize/unserialize process speed (nearly everything would be ready to be
transferred as is to the CGIs).

In conclusion : I consider the complexity of this code as a rather low price
to pay to speed up the CGI execution by a factor of 10 to 30. Now, the
choice depends of the value you give to this improvement...

Maybe other people with large configurations can say how interesting it
seems to them.

Regards

François
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-lists.org/archive/developers/attachments/20050609/5cc16f3f/attachment.html>
-------------- next part --------------
Ce message et ses pièces jointes (le "message") est destiné à l'usage    
exclusif de son destinataire.                                            
Si vous recevez ce message par erreur, merci d'en aviser immédiatement   
l'expéditeur  et de le détruire ensuite. Le présent message  pouvant  
être altéré à notre insu,  CALYON Corporate and Investment Bank                              
ne peut pas être engagé par son contenu. Tous droits réservés. 
          
This message and/or any  attachments (the "message") is intended for     
the sole use of its addressee.                                            
If you are not the addressee, please immediately notify the sender and    
then destroy the message.  As this message and/or any attachments may 
have been altered without our knowledge,  its content  is not legally 
binding on CALYON Corporate and Investment Bank. All rights reserved.