Make sockets non-blocking

[Stephen Gran]

On Thu, May 06, 2010 at 11:21:36AM +0200, Andreas Ericsson said:
> On 05/06/2010 12:32 AM, Stephen Gran wrote:
> > 
> > I wasn't too worried about dealing with message queueing, as NDO
> > already does that part fairly well (as I'm sure you're aware).
> 
> Actually, I'm not. When we started looking at NDOUtils it wasn't very
> efficient. Diligent effort has been spent to make it decent, but its
> database schema and the fact that it maintains log-entries in a flat
> format in a database makes it rather unusable for us. Hence, we
> decided to make Merlin database-aware and use that instead. I neither
> maintain nor use NDOUtils myself, but I review patches for it whenever
> I come across them.

ndomod itself doesn't queue in a database - it has an internal struct of
outstanding messages to send (actually just a char ** in the message
struct, but close enough).  This arguably could be improved, but it has
the advantage of simplicity.

There is quite a bit that could be improved in the handling of the
sinkbuffer (functions that return int being called in void context and
so on), but it's much lower on my priority list.

> >  I think that short term, the simplest logic may be to scrap the
> >  while loop altogether and just mark the message as failed if you
> >  get a partial write.  That way the recipient can ditch it without
> >  worrying about reassembling partial messages, and NDO can deal with
> >  it with it's usual retry logic.
> 
> Sounds sensible. It should probably re-try immediately on EINTR or
> EAGAIN though.

EINTR makes sense to retry on - EAGAIN is slightly more difficult, since
a wedged socket will just continuously return EAGAIN/EWOULDBLOCK.  This
difficultly in deciding is one of the reasons for opting to leave the
loop in place instead of removing it altogether :)  The major downside
of opting to mark a socket as failed and move on, of course, is that
when you close the blocked socket, you throw away all the messages in
the socket's send buffer.  I'm not convinced there is a way around that,
but I'm not sure quite how glib we should be about marking a socket as
blocked.

To be clear, in our environment, we use ndomod to update a reporting
server - it is not actually crucial to getting pages to engineers.  In
our environment, non-blocking I/O is more important than the downside of
losing messages.  This may not be the case in other environments, where
message delivery may be more critical.  I'm trying to straddle the line
of making I/O non-blocking while being reasonably cautious about
discarding messages, but if you ask me to make a choice, I'll likely
come down on the side of not blocking the main nagios process.

Cheers,
-- 
 --------------------------------------------------------------------------
|  Stephen Gran                  | O, it is excellent To have a giant's    |
|  steve at lobefin.net             | strength; but it is tyrannous To use it |
|  http://www.lobefin.net/~steve | like a giant.   -- Shakespeare,         |
|                                | "Measure for Measure", II, 2            |
 --------------------------------------------------------------------------

------------------------------------------------------------------------------