[PATCH] Re: alternative scheduler

[Andreas Ericsson]

On 12/01/2010 08:55 PM, Adam Augustine wrote:
> On Wed, Dec 1, 2010 at 8:33 AM, Jochen Bern<Jochen.Bern at linworks.de>  wrote:
>> On 12/01/2010 03:14 PM, Andreas Ericsson wrote:
>>> A much better solution would be to spawn workers to handle the checks
>>> and let the master parent just sit and receive results and update status
>>> files though, but that's a quite invasive change so it'll have to wait a
>>> bit.
>>
>> Would DNX or Mod Gearman (which both have a NEB module snatch checks
>> before the Nagios core gets around to executing them itself, and feed
>> them into sort of a distributed batch queue system instead) be close
>> enough to qualify?
>>
>> http://dnx.sourceforge.net/
>> http://labs.consol.de/nagios/mod-gearman/
>>
>> Kind regards,
>>                                                                 J. Bern
>> --
> 
> While DNX and mod_gearman do implement that specific functionality,
> they are still subject to the scheduler/reaper bottlenecks. We (the
> institution that started the DNX project) have played around with the
> check scheduling parameters quite a bit over the years and even with
> our best scheduling parameters and DNX actually executing the plugins,
> we still see checks scheduled such that we have a large number of
> checks scheduled to execute in a single second with several seconds
> (3-5) of nothing scheduled to execute between. That isn't necessarily
> a big problem as long as the DNX and mod_gearman workers can handle
> the peaks.

Handling peaks has always been the problem with performance. Handling
the valleys is never the problem.

> But you then have to provide bigger hardware than if the
> checks were scheduled more smoothly. And really, all things being
> equal, the average number of checks scheduled to execute in any given
> second should be relatively constant.
> 

Agreed. More or less. It should also be relatively straightforward to
implement. The only corner case I can think of that needs special
handling is when a check is scheduled to run very rarely and with a
shorter span in the timeperiod than its check_interval is set to. All
other cases can be handled by grabbing a random number and adding that
number to the next scheduled check interval while adjusting the check
interval downwards by half the "scheduling_flexibility" number of
seconds.

> I suppose it is possible we have something mis-configured because we
> have misunderstood the inner workings of the scheduler, but I am at a
> loss as to where, and we have spent a lot of time in the past looking
> at how checks are scheduled and executed.
> 
> I haven't used Merlin yet (I intend to do some testing), but the model
> of distributed schedulers each handling smaller numbers of checks
> works around that problem. But if it works that way then it really
> just hides the issue.

Yes and no. No matter what scheduler you're using you'll sooner or later
run into networks too large for one system to handle. When that happens,
you'll have to expand sideways, and merlin lets you do just that. It's
orthogonal to having a scheduler that distributes load somewhat evenly
on a single system.

Merlin also solves a different problem than DNX and mod-gearman in that
it allows one to have a full UI on all the nodes in the network. AFAIU,
DNX doesn't do that, and neither does mod-gearman.

> DNX actually posts results directly to the
> circular results buffer to bypass some of the reaper issues. I noticed
> Andreas' blog posting includes breaking the reaper into its own thread
> to get the level of performance shown, which makes sense, and I think
> an attempt to do that was posted some time ago on this list by Steve
> Morrey.
> 

It was, and AFAIR he got stuck on the same issue I ran into, although I
patched Nagios to have thread-safe API's for handling performance-data
and checks and whatnot. Currently, only ctime() remains to be changed
to use ctime_r() instead.

> Anyway, the scheduler was, I think, originally designed to be very
> conservative in terms of CPU use, and back in the day that made sense
> with the limited hardware that was available. I think now the
> expectation is that large installations will be dedicating hardware
> and wanting Nagios to consume as much as has been allocated to it.

Yes. The current problem is that Nagios has problems saturating the
CPUs on powerful hardware. It's quite commonplace to see a system
with high latency and low load. While it's often down to pilot error
and a failure to understand how the scheduler works and how the
various variables the user can set affects it, there's still a limit
to how much cpu one can get Nagios to eat.

> Clearly changing some of the sleeps to sched_yield would be a good
> beginning. Putting the reaper into another thread (as Andreas blog
> posting indicates) is another massive improvement.
> 

Doing so would probably be a rather straightforward change since the
thread-safety changes were introduced. Provided there's only one
reaper thread running at a time there shouldn't even be any concurrency
issues.

> But I think the scheduler still needs to be looked at, unless we are
> one of a small group seeing that behavior.
> 

It does, but it's not so bad as this thread sometimes make it seem.
And no matter what we do, there'll always be room for improvement.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev