Java Thread Priority Revisited in JDK7

In an earlier blog entry I described how Java thread priorities relate to native thread priority. We recently ran into bug 6518490 that forced me to change the default mapping of priorities on Solaris™ as previously described.

Briefly, in the Solaris IA(interactive) or TS(timeshare) scheduling classes, the effective priority of a thread is function of the thread's assigned priority and a thread-specific system-managed behavioral factor. (That's a gross oversimplification. If you're curious, I recommend Solaris Internals, 2nd Edition Section 3.7 followed by a skim of ts.c. In the case of Java the JVM maps the logical Java priority set via Thread.setPriority() to an TS- or IA-specific priority, and then calls priocntl() to set the assigned priority for the thread. The scheduler avoids indefinite starvation of low priority threads by periodically applying a boost (this is the system-managed behavioral factor) for threads that have languished too long on the ready queue without having been run. In addition, threads that block voluntarily -- as opposed to exhausting their time slice and requiring involuntary preemption -- may receive a boost to their effective priority by way of changes to the previously mentioned behavioral factor.

Lets say we have a 2x multiprocessor system, otherwise idle, with threads T1 and T2 at high assigned priority and T3 at low assigned priority. T1 and T2 loop, yielding, or alternatively they might pass a "token" back and forth via a pipe. Critically, they block often, so they receive the aforementioned boost to their effective priorities. Thread T3 simply loops, computing the digits of π. T3 will languish on the ready queue. Over time the scheduler will boost T3's effective priority, but unfortunately that boost isn't sufficient to push T3's priority beyond that of the boosted effective priorities of T1 and T2, so T3 can starve indefinitely.

To avoid the problem I've inverted the default for the -XX:[+/-]UseThreadPriorities flag from True to False on Solaris. The defaults setting for the flag remains unchanged on other platforms. Running all Java threads at the same priority sidesteps the problem and avoid starvation. I considered just posting a work-around and leaving the flag polarity unchanged, but the bug is difficult to diagnose properly. A work-around, albeit simple, is useless if a customer is forced to diagnose and identify a rather opaque and exotic bug. Put another way, a work-around is no better than the diagnostic procedure used to identify the associated malady. The change to UseThreadPriorities is in JDK7 and is propagating back through JDK6 and JDK5. If you desperately want Java threads priorities to map to underlying native priority you can use -XX:+UseThreadPriorities, but in a sense you'll be accepting the risk of the hang. You have to explicitly opt-in to enable priority mapping. Priorities are a quality-of-implementation concern, not a correctness concern. Recall that thread priorities are strictly advisory, and the JVM has license and latitude to implement or change them as needed. They should never be depended upon to ensure progress (relative to other threads) or lack of progress (i.e., synchronization).

Finally, this changed isn't as significant as it might seem. Under IA and TS, the upper half of the logical priority range was previously mapped to native "high" and the lower half was mapped over the range of native priorities. The only observable difference between the old behavior and running with thread priorities disabled is that Java threads assigned priorities in the lower half of the logical range will now run at the same effective (native) priority as threads in the upper half. When competing for CPU cycles with other threads, these low priority threads will receive relatively more cycles from the scheduler than they did in the past.

Comments:

So basically, you're working around a thread-starvation bug in Solaris. That sounds weird. I would have thought that Sun would have fixed something like that a long time ago.

An alternate work-around might be to map the logical Java thread priorities to 3 or 4 Sun thread priorities, with all of the Sun thread priorities at one end of the Sun priority range.
Having small gaps would make it less likely that you'd hit the problem induced by the sun "behavioural" adjustment.

Posted by Noel Grandin on January 10, 2008 at 10:13 PM EST #

Hi Noel, Yes - arguably the JVM change is a work-around for a kernel issue and ideally I'd prefer to have seen this addressed in the kernel. The anti-starvation policy in the kernel scheduler is a QoI optimization, but provides no guarantees of eventual progress. But with a work-around in the JVM we can provide relief to the customers in the field in a timely manner. The conditions where the hang will manifest are rather exotic and contrived, but we've seem it with increasing frequency.

I experimented with restricting priority bands as you'd mentioned, but the bug still recurs. In my judgement it was best to simply eliminate the hang instead of decreasing the frequency.

Regards, -Dave

Posted by Dave Dice on January 11, 2008 at 01:39 AM EST #

Hi Dave, it sounds like the 'effective boosting' algorithm that determines which Thread gets preference is incorrect. If it is indeed counting the number of actual cycles that each thread gets then eventually the lower priority thread will get some cycles on some core, irrespective of which cores or how many cores are running the other (higher priority) threads. If the logic somehow became incorrect when the VM threads are running on multiple cores (or processors) then what is this effective boosting algorithm actually counting to determine which thread gets to go?

Posted by Matt on January 27, 2008 at 06:45 PM EST #

Hi Matt, Folks on the Solaris team indicated that the anti-starvation facility -- which provides transient boosts to threads that have languished for long periods on the dispatch queues -- doesn't absolutely guarantee eventual progress (non-starvation) in the presence of mixed assigned priorities. In our case we have some threads at high(er) assigned priority that deschedule themselves voluntarily and frequently, and are thus beneficiaries of a behavioral "bonus". These threads are competing with threads at a lower assigned priority. The anti-starvation adjustment isn't sufficient to push the effective priority of this 2nd group of threads above that of the 1st.

With respect to CPU cycle accounting, older kernels used the 100Hz "tick" interrupt and were vulnerable to quantization problems but that's changing with the arrival of cycle-accurate accounting.

Also, under Solaris each CPU has its own dispatch queue and makes local decisions but the policies and parameters are such that these local decisions collectively achieve the desired global scheduling policy. There is no centralized scheduler. As a consequence of this design it's possible encounter situations where CPU#0 is running thread T5 and T5's effective priority is less than that of thread T6, which resides on CPU#1's dispatch queue. (You can avoid that scenario by placing threads in the real-time "RT" scheduling class).

Posted by Dave Dice on January 28, 2008 at 02:47 AM EST #

As a former employee at one of the big4 firms I realized that my options are pretty hard to find, until I heard about: Big4.com. Now, I have a new great job and I’m looking ahead in the future with my head held up high.

Posted by Michael on October 16, 2008 at 05:29 AM EDT #

David, hi;

Can you please expose JVM Thread Policy as JDK management interface instead of forcing people to mess with
-XX:[+/-]UseThreadPriorities, etc options?

I would rather detect at run time in java code where I am and then use -XX:ThreadPriorityPolicy analog for policy level, -XX:JavaPriority10_To_OSPriority=0 analog for mapping contract, etc, rather then to mess with external start-up scripts;

Currently I have to use hacks such as these via JNI or bash:
http://www.akshaal.info/2008/04/javas-thread-priorities-in-linux.html

Finally, please make jvm thread policy values consistent across os; for example -XX:ThreadPriorityPolicy=1 means one thing on windows and other on linux, and third on solaris.

Thanks; Andrei.

Posted by Andrei Pozolotin on October 19, 2009 at 03:45 AM EDT #

Hi Alex,

Your point is well-taken that the current mapping policy and set of switches leaves something to be desired.

The issue you point out highlights a philosophical fissure: developers sometimes want direct access to platform-specific APIs for which there's an imperfect (or no) mapping in the Java platform. Historically, the Java community resists directly exposing platform-specific facilities because we lose portability.

Thread priorities are a classic example. Technically, thread priorities are completely advisory, and, at least for Java, serve only as hints to optimize performance. It's up to the discretion of the JVM implementor to provide the mapping between the logical priority abstraction in Java and the underlying platform. (Completely ignoring the mapping and doing nothing is entirely reasonable, and in fact is used by some JVMs).

Unfortunately there's not always an easy and obvious 1:1 numerical mapping between logical and actual priorities. Often, platform priorities are a tuple including assigned priority, scheduling class, and other attributes. As such, we clearly can't satisfy the needs of every developer with a simple mapping.

Furthermore, the platform default priorities (which may be high, low, or somewhere in the middle) have a significant impact on mappings as we don't want the java default priority to be different from the platform default.

The situation becomes even murkier because of scheduling anomalies which arise on different platforms, and, at the request of the OS implementors, we try to avoid in the JVM by fine-tuning the mapping mechanisms.

Given that, in some cases you're better off using JNI calls to platform-specific code to assign the priority (and scheduling class, etc) as you see fit. We officially discourage this -- it assumes a particular thread model, for instance -- but in practice it's workable, at least with today's JVM implementations.

Regards, Dave

Posted by David Dice on October 22, 2009 at 02:49 AM EDT #

Sorry! Mis-addressed, I meant Andrei instead of Alex.

Dave

Posted by David Dice on October 22, 2009 at 02:51 AM EDT #

Dave,
thanks for the very polite "NO";
BUT, how about a compromise: expose what I ask as part of tools.jar / com.sun.xxx management package?
cheers,
Andrei.

Posted by Andrei Pozolotin on November 08, 2009 at 06:15 AM EST #

It would be great if thread priorities were turned on by default on Linux. This is fully doable, as a user is allowed to lower a thread's priority by way of setting the a positive nice value/level on it.

Currently, the JVM checks whether you are root, and simply disables the entire thing if not, emitting a warning if you tried to enable it. The reasoning behind this is obviously that only root can heighten the nice value (use negative values), thus getting higher than normal thread priorities. However, as mentioned, normal users can still lower the value, which in many circumstances is actually what is wanted (e.g. lower priority batch jobs running in background should "yield" for higher priority interactive work).

But, due to a bug in the check, it is actually possible to get what everyone wants. Check out this "Closed - Fix delievered" bug (which obviously is not fixed, and not delivered):
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4813310

I ran some tests verifying the hack, and wrote some words on it:
http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workaround.html

Please just take away those checks, and enable the thread priorities on Linux by default.

Posted by Endre Stølsvik on January 30, 2010 at 01:36 PM EST #

YES, I agree: "It would be great if thread priorities were turned on by default on Linux" - Andrei.

Posted by Andrei Pozolotin on February 15, 2010 at 08:32 AM EST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Dave

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today