Scheduler as a Thread
By templedf on Nov 03, 2007
If you were at the Sun Grid Engine Workshop '07 in September, you heard Andy talk about the new development efforts around Sun Grid Engine 6.2. One of those development projects was to make the scheduler process into a thread in the qmaster.
When we made the transition from 5.3 to 6.0, one of several major changes was that Grid Engine became multi-threaded. Remember the commd? In 6.0 the commd went away because it was replaced with a multi-threaded communications library. The qmaster itself went from being a single-threaded processing loop to being a heavily multi-threaded daemon. The idea of the qmaster also absorbing the scheduler was discussed back then, but the transition was already complicated enough.
Now that the multi-threaded qmaster has had time to settle, we've decided it's time to shake the snow globe again. Why? Well, every scheduling run, the qmaster has to pass the scheduler everything it needs to know to make its scheduling decisions. Bringing the scheduler into the qmaster as a thread allows the scheduler direct access to the data without sending it over the wire. It's not all hugs and puppies, though. Both the qmaster and the scheduler make high demands on the same set of data. Locking will be an issue that will take a little time to optimize. Don't expect the first release that has the scheduler as a qmaster thread to be an order of magnitude faster. We may eventually get there, but it will take some time.
The potential performance increase from reduced data traffic is nice, but for me the really exciting part is that once the scheduler is a thread in the qmaster, it's a small additional step to make the scheduler into 2 threads. Once there is more than one scheduler thread, going from 2 to 4 to 8 is just a matter of tuning.
What good are multiple scheduler threads? Well, the most obvious answer is that they can share the task of scheduling jobs, potentially improving scheduling time by parallelizing the operation. Even more interestingly, multiple scheduler threads could potentially each manage a different scheduler domain. Have you ever wanted to enable the fair share (share tree) policy during working hours and the functional policy at night? Having two scheduler threads with two different scheduler domains and two different scheduler configurations would let you do exactly that. No promises on when or if scheduler domains will become a product feature, but the ground work has been laid by making the scheduler into a qmaster thread.
Why bring this up now? Well, the engineer working on the project is going to do his first check-in today. It's not yet 100%, but it should work well enough to try. (It still fails some of our regression tests.) To get a build, check out the EB_PRE_SAAT tag and compile. Feedback is appreciated. You can send comments to the email@example.com mailing list.