[PATCH] Re: alternative scheduler

[Fredrik Thulin]

On Fri, 2010-12-03 at 14:28 +0100, Andreas Ericsson wrote:
> ...
> > I meant to say that N is calculated when the list of checks is
> > (re)loaded. As I don't even try to have retry_intervals and such, a
> > steady tick interval works great as long as I can finish initiating
> > another service check in between ticks.
> > 
> 
> Ah, right. And initiating a check is quite cheap until they start
> piling up when the network goes bad, which you sort of avoid by using
> a constant stream of executing checks, so you always know there'll be
> constant load on the system you're monitoring from. 

Right, but initiating checks doesn't get more expensive just because the
checks require more CPU cycles to complete (because of retries). Other
resources might suffer though - I guess the first one to be depleted
would be file descriptors.

> I'm wondering if
> that doesn't sort of solve the problem in the wrong direction though,
> since the monitoring system is supposed to serve the other systems and
> endure the inconveniences it suffers itself as best it can. Sort of.

Hmm. The goal here is to scale sideways as you put it. To evolve to more
cores and more schedulers thus reaching higher number of checks possible
per time unit, per server.

If a given server can only take on 1000 checks per time unit and you
typically run it around 900, nothing good will come out of
retry_interval suddenly trying to get the server to do 1100 checks per
minute. That is over-subscription and the result is undefined at best.

I would rather dynamically figure out that I'm very probable to be able
to run 1000 checks per time unit, and then either 

  * use my current approach of always doing 950 and not having   
    retry_interval and similar, or
  * do 800 per time unit, and allow retry_interval etc. to push it up to
    900-1000 but never more

Funny. I've always preferred Ethernet to ATM but here I am talking about
time slots and guarantees =).

> >> That's still "doing more than you did before", on a system level, so the
> >> previous implementation must have been buggy somehow. Perhaps erlang
> >> blocked a few signals when the signal handler was already running, or
> >> perhaps you didn't start enough checks per tick?
> > 
> > I agree it is more work for the scheduler, but that is better than
> > having under-utilized additional CPUs/cores, right?
> > 
> 
> So long as the net effect is that you can run more checks with it, yes, but
> an exponential algorithm will always beat a non-exponential one, so with a
> large enough number of checks you'll run into the reverse situation, where
> the scheduler eats so much overhead that you no longer have cpu power left
> to actually run any checks.

That would be absolutely true for a single scheduler, but think about
many schedulers!

My current PoC has a single scheduler too, with a single list of all
checks. That worked good enough for the PoC, but I could very easily
turn it into as many as one scheduler process per core, each having a
subset of the total set of checks.

I'm not sure the ideal is one scheduler per core, but do think the
architecture should be along those lines. The future looks like it will
give us many many MANY cores. To be able to utilize them, you must
ultimately leave the single scheduler situation.

Even if you offload the scheduler thread of everything else, a single
CPU won't have time to even loop through a huge enough list of services.
This last paragraph is a bit academic, I agree ;).

/Fredrik



------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev