Analysis against JVM Thread Dump - Out Of Thread

Out Of Thread means there are many tasks (requests) are pending to run, but only a few Threads are available. Many queued tasks have to wait for running threads to be idle. It is a much more complicated case and it is very hard to be identified from Thread Dump log. To dig out the root cause it requires the analyst have much knowledge on the Thread Pool technology. We will have a much more complex source code to demonstrate this.

Demonstration

We define a class ThreadPool which creates a group of a limited number of threads that are used to execute queued tasks. It contains below functions

  • ThreadPool(): The constructor to initialize a group of threads (several PooledThread instances).
  • runTask(): Requests a new task to run. This method returns immediately and the task executes on the next available idle thread in this ThreadPool.
  • getTask(): Returns the next pending task to run.
  • close(): Force to stop all threads (PooledThread instances), no matter the thread is running or queued to run.
  • join(): Closes this ThreadPool and waits for all running threads to finish. Any waiting tasks are executed.

ThreadPool also has an inner class PooledThread which extends Thread abstract class, it is used to create the concrete running thread and invoke the target task.

private class PooledThread extends Thread {

        public PooledThread() {
            super(ThreadPool.this, "Zigzag_Thread-" + (threadID++));
        }

        public void run() {
            while (!isInterrupted()) {

                // get a task to run
                Runnable task = null;
                try {
                    task = getTask();
                } catch (InterruptedException ex) {
                }

                // if getTask() returned null or was interrupted,
                // close this thread by returning.
                if (task == null) {
                    return;
                }

                // run the task, and eat any exceptions it throws
                try {
                    task.run();
                } catch (Throwable t) {
                    uncaughtException(this, t);
                }
            }
        }
    }
	

So put it together to see how the ThreadPool is defined, like below.

	
class ThreadPool extends ThreadGroup {

    private boolean isAlive;
    private LinkedList taskQueue;
    private int threadID;


    public ThreadPool(int numThreads) {
        super("Zigzag_ThreadPool");
        setDaemon(true);

        isAlive = true;

        taskQueue = new LinkedList();
        for (int i = 0; i < numThreads; i++) {
            new PooledThread().start();
        }
    }

    
    public synchronized void runTask(Runnable task) {
        if (!isAlive) {
            throw new IllegalStateException();
        }
        if (task != null) {
            taskQueue.add(task);
            notify();
        }

    }

    protected synchronized Runnable getTask() throws InterruptedException {
        while (taskQueue.size() == 0) {
            if (!isAlive) {
                return null;
            }
            wait();
        }
        return (Runnable) taskQueue.removeFirst();
    }

    
    public synchronized void close() {
        if (isAlive) {
            isAlive = false;
            taskQueue.clear();
            interrupt();
        }
    }


    public void join() {
        synchronized (this) {
            isAlive = false;
            notifyAll();
        }

        Thread[] threads = new Thread[activeCount()];
        int count = enumerate(threads);
        for (int i = 0; i < count; i++) {
            try {
                threads[i].join();
            } catch (InterruptedException ex) {
            }
        }
    }


    private class PooledThread extends Thread {

        public PooledThread() {
            super(ThreadPool.this, "Zigzag_Thread-" + (threadID++));
        }

        public void run() {
            while (!isInterrupted()) {

                // get a task to run
                Runnable task = null;
                try {
                    task = getTask();
                } catch (InterruptedException ex) {
                }

                // if getTask() returned null or was interrupted,
                // close this thread by returning.
                if (task == null) {
                    return;
                }

                // run the task, and eat any exceptions it throws
                try {
                    task.run();
                } catch (Throwable t) {
                    uncaughtException(this, t);
                }
            }
        }
    }
}

And next we have the client program defined to ask the ThreadPool to create a group of PooledThreads to run many more tasks.

public class OutOfThread {
    public static void main(String[] args) {
        if (args.length != 2) {
            System.out.println("Tests the ThreadPool task.");
            System.out.println("Usage: java OutOfThread numTasks numThreads");
            System.out.println("  numTasks - integer: number of task to run.");
            System.out.println("  numThreads - integer: number of threads in the thread pool.");
            return;
        }
        int numTasks = Integer.parseInt(args[0]);
        int numThreads = Integer.parseInt(args[1]);

        // create the thread pool
        ThreadPool threadPool = new ThreadPool(numThreads);

        // run example tasks
        for (int i = 0; i < numTasks; i++) {
            threadPool.runTask(createTask(i));
        }

        // close the pool and wait for all tasks to finish.
        threadPool.join();
    }

    /**
     * Creates a simple Runnable that prints an ID, waits a long time, then
     * prints the ID again.
     */
    private static Runnable createTask(final int taskID) {
        return new Runnable() {
            public void run() {
                System.out.println("Task " + taskID + ": start");

                // simulate a long-running task
                try {
                    int i = 0;
                    while (i<9999999L*2000)
                        i++;

                } catch (Exception ex) {
                }

                System.out.println("Task " + taskID + ": end");
            }
        };
    }

}

We ask the ThreadPool to create only 3 threads in its pool, and these 3 threads will run 5 tasks. The expected result could be:

Task 2: start
Task 0: start
Task 1: start
Task 0: end
Task 3: start
Task 2: end
Task 4: start
Task 1: end
Task 3: end
Task 4: end

However the actual result is:

E:\>java -classpath . zigzag.research.threaddump.OutOfThread 5 3
Task 2: start
Task 0: start
Task 1: start
Task 2: end
Task 3: start

In a long period, we find Task 4 are not started yet and the client program is continuously running and waiting, poor performance occurs.

Thread Dump Analysis

In the Thread Dump we find below information, Zigzag_Thread-3, Zigzag_Thread-1 and Zigzag_Thread-0 are "RUNNABLE". Definitely you cannot find any problem because they are running. After you take several other Thread Dump, it still show these 3 threads running. Zigzag_Thread-2 does not show because it is done. But where is the expected Zigzag_Thread-4? Zigzag_Thread-4 is still waiting in the queue and cannot get any idle thread.

"Zigzag_Thread-3" prio=6 tid=0x00000000069d5800 nid=0x26b4 runnable [0x000000000744f000]
   java.lang.Thread.State: RUNNABLE
        at zigzag.research.threaddump.OutOfThread$1.run(OutOfThread.java:45)
        at zigzag.research.threaddump.ThreadPool$PooledThread.run(OutOfThread.java:183)

"Zigzag_Thread-1" prio=6 tid=0x00000000069d5000 nid=0x11b4 runnable [0x000000000734f000]
   java.lang.Thread.State: RUNNABLE
        at zigzag.research.threaddump.OutOfThread$1.run(OutOfThread.java:45)
        at zigzag.research.threaddump.ThreadPool$PooledThread.run(OutOfThread.java:183)

"Zigzag_Thread-0" prio=6 tid=0x00000000069d2000 nid=0x1a34 runnable [0x000000000724f000]
   java.lang.Thread.State: RUNNABLE
        at zigzag.research.threaddump.OutOfThread$1.run(OutOfThread.java:46)
        at zigzag.research.threaddump.ThreadPool$PooledThread.run(OutOfThread.java:183)

Modern Application Server (Web Container) has the interface for user to monitor and modify the Thread Pool size. For example below is Oracle Application Server and Weblogic Server.



Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About


Jie Chen is the L3 memeber in Oracle Agile Support.
This blog focuses on the Maintenance, Diagnosis and Tuning related technical skills.
The technology covers Java/JavaEE, Weblogic, Security, Clustering, and Database of course.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today