StackOverflowError and threads waiting for ReentrantReadWriteLock

If the terms 'Stuck Threads' or 'Hang' sound familiar and scary to you, please read on. We can sometimes come across situations where the Weblogic Server (WLS) reports some threads as being 'Stuck' or 'not making any progress', but the stack traces of those threads and others in the process look absolutely normal, showing no signs why and how those threads could get stuck. I got to analyze such a situation recently. The threads in the application were reported to be Stuck....

Wednesday, February 28, 2018 | Read More

Can young generation size impact the application response times?

I was recently involved in a performance issue where the application's response times were decreasing over time, and I was tasked to determine if GC was playing any role in that. Well, as we all know that misconfiguration of the java heap space and its generations can lead to unexpected/frequent/longer garbage collection cycles causing inferior performance and longer response times of the application, and that indeed was the cause of this performance issue. I want to share some...

Monday, December 11, 2017 | Poonam Parhar | Read More

Clarifying some confusion around Java Flight Recordings

In this blog, I am going to talk about two confusions around Java Flight Recordings that I have come across from multiple users/customers. Let's take a look at them one by one. Flight Recordings Interval A number of Java Flight Recorder users seem to have a confusion around the 'start-time' and the 'interval' of the flight recordings. The complaint is that the flight recordings in the Java Mission Control show longer recording interval than their actual recorded data. For...

Tuesday, November 14, 2017 | Read More

How do I find what's getting promoted to my old generation?

This question comes up every now and then that how can I find what the minor/young collections are promoting to the old generation of my Java application. Recently, there was a case where the customer wanted to know the types of objects getting promoted to the old generation at different input loads for their application. Well, one longer and a little bit tedious way is to use -XX:+TraceScavenge option. This is a non-product JVM option and hence you would need to build a debug...

Wednesday, May 10, 2017 | Read More

HotSpot JVM throwing OOM even when there is memory available

I was recently asked a question that why my Java application runs out of Java Heap when there is still so much memory available to use? Here is the question in detail - I am running my application with -Xmx1600m but the logs show that it runs out of memory without fully using the 1600mb. 2017-03-21T13:15:39.478+0000: 289274.599: [Full GC [PSYoungGen: 338944K->0K(425472K)] [ParOldGen: 1092073K->1055276K(1092096K)] 1431017K->1055276K(1517568K) [PSPermGen:...

Thursday, April 27, 2017 | Read More

WeakReferences and GC

I was recently working on a memory growth issue involving WeakReferences, and I came across this great blog post on how non-strong references can affect the heap usage and the garbage collection: https://plumbr.eu/blog/garbage-collection/weak-soft-and-phantom-references-impact-on-gcIt has a demo application that demonstrates an increase in the number of objects getting promoted from young to old generation when WeakReferences are used. That made me deeply interested...

Thursday, February 16, 2017 | Read More

Hung JVM due to the threads stuck in pthread_cond_timedwait()

In this post, I am going to discuss a couple of scenarios where the Java process hangs are actually NOT because of a Java/JVM issue but happen due to the OS problems. Both the hangs being discussed here occur on the Linux operating system. Lets take a look at the first case where a thread seems to be stuck in the Java Thread.sleep() call. From the process stack trace, the two threads of interest are: Thread 26755: (state = IN_VM)  - java.lang.Thread.interrupt0() @bci=0...

Wednesday, November 2, 2016 | Read More

G1 and Flight Recorder's -XX:FlightRecorderOptions=stackdepth option

Recently there was a report from a customer that they observed performance degradation while using JFR with G1 when they used it along with -XX:FlightRecorderOptions=stackdepth=512, but didn't see any performance impact when using the same setting for the stackdepth with the Parallel Collector. The flight recordings revealed that the poor performance was due to the long pauses introduced by the JFRCheckpoint operations. The long JFRCheckpoints occurred only for the recordings...

Thursday, October 27, 2016 | Read More

Crashes in ZIP_GetEntry

Recently, received quite a few reports of the application crashes from different customers and product groups with the stack trace looking like this: Stack: [0xb0c00000,0xb0c80000], sp=0xb0c7c890, free space=498kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)C [libc_psr.so.1+0x700] memcpy+0x2f8C [libzip.so+0xd19c]C [libzip.so+0x2380] ZIP_GetEntry+0xe4C [libzip.so+0x2800] Java_java_util_zip_ZipFile_getEntry+0xc4j ...

Wednesday, September 28, 2016 | Read More

Eager reclamation of Humongous objects with G1

G1 collector in JDK 8u40 and 8u60 got a very nice enhancement that might come as a surprise to users when they upgrade to either of these releases. As we all know that the objects larger than half of the size of the regions are called Humongous objects, and are allocated in special regions called Humongous regions. Up until 8u40, the Humongous regions were not collected during Evacuation pauses, and were collected only at the end of the marking cycle or during Full GCs.In...

Tuesday, September 27, 2016 | Read More

Using NMT with custom JVM launcher

I recently worked with a customer, facing problem in using Native Memory Tracker with their custom JVM launcher created using the JNI API JNI_CreateJavaVM, and learned how NMT can be enabled when used with the custom JVM launchers (non 'java' launchers). This blog post shares details on how that can be done.Let's start with the following JNI code example where I am passing the JVM option -XX:+NativeMemoryTracking=detail as an argument while creating the JVM with...

Tuesday, February 16, 2016 | Read More

VM warning: PICL (libpicl.so.1) is missing

To increase the compiled code performance, following two fixes were integrated into JDK 8u40 and 7u80. With these changes, HotSpot uses PICL library available on Sparc Platform to get the CPU L2 cache line data size. This information can be used by the Compiler to optimize the generated compiled code and thus improve the runtime performance. 1. JDK-8056124: Hotspot should use PICL interface to get cacheline size on SPARC Changeset: http://hg.openjdk.java.net/jdk9/jdk9/hotspot/r...

Tuesday, December 8, 2015 | Read More

JVM hang with CMS collector

This blog post was long due but better late than never. Sometime back I worked on an interesting GC problem which initially looked like an issue with the JVM runtime system. I am talking about 'JDK-8085965: VM hangs in C2Compiler'. This bug has been fixed in JDK 9 and 8u45+. In this post, I want to share interesting symptoms of this bug, and the troubleshooting steps taken that helped in getting to the root of the problem.This report came in as a JVM hang when running with...

Tuesday, October 27, 2015 | Read More

Why do I get message "CodeCache is full. Compiler has been disabled"?

JVM JIT generates compiled code and stores that in a memory area called CodeCache. The default maximum size of the CodeCache on most of the platforms is 48M. If any application needs to compile large number of methods resulting in huge amount of compiled code then this CodeCache may become full. When it becomes full, the compiler is disabled to stop any further compilations of methods, and a message like the following gets logged: Java HotSpot(TM) 64-Bit Server VM warning:...

Thursday, September 10, 2015 | Read More

Throughput collector and PrintTenuringDistribution option

This is going to be a very short blog post. The inspiration for this post is a customer report that the PrintTenuringDistribution option does not show age information with the throughput collector (Parallel or Parallel Old GC).2.798: [GC Desired survivor size 11534336 bytes, new threshold 7 (max 15) [PSYoungGen: 67584K->9802K(78848K)] 67584K->9810K(254976K), 0.0467029 secs] [Times: user=0.07 sys=0.02, real=0.05 secs] 3.604: [GCDesired survivor size 11534336 bytes, new...

Tuesday, April 21, 2015 | Read More

Longer Young Collections with JDK7 as Compared to JDK6

If you have moved from JDK 6 to 7 and are observing some increase in the times that young collections take to do their job, then here's a remedy: -XX:+JavaObjectsInPerm.In JDK7, class statics and interned strings were moved from PermGen to the Java Heap and that may contribute towards a little increase in the young generation collection times. There is an option JavaObjectsInPerm which when enabled reverts that change i.e. class statics and interned strings get allocated back...

Friday, April 17, 2015 | Read More

Updates to the Java Troubleshooting Guide

Mattis Castegren who is my manager at Oracle, and is the guest writer for this blog post would like to share some details on the Java Troubleshooting Guide. Here's what he has to say:-- With the release of JDK 8, the official Java Troubleshooting Guide got a big overhaul. All pages were looked over and updated for JDK 8, and the two previous guides for the JVM and for Desktop Technologies were merged into one. Inthe last month, with the release of 8u40, we have launched...

Thursday, April 2, 2015 | Read More

Analysis of Strange Hotspot Crashes

In this blog post, I am going to talk about the core file analysis of some strange looking hotspot crashes using gdb and Serviceability Agent debugging tools. At the first glance, these crashes appeared to be caused by some Hotspot Server Compiler issue but in the end these actually turned out to be due to a Linux Kernel bug. This first crash happened in the compiled code of method SafeHashMap.put() # Problematic frame: # J 3575 C2 com.tangosol.util.SafeHashMap.put(Ljava/lang/Ob...

Tuesday, February 3, 2015 | Read More

Long Class-Unloading Pauses with JDK8

Recently, I came across a report where a user was facing long GC pauses with JDK8. They didn't see this issue while running with JDK7. The problem with jdk8 was that sometimes the class unloading step of CMS-remark phase was taking a very long time to do its job. They were running with large pages enabled with both jdk7 and jdk8.2014-11-30T01:51:35.632+0000: [GC (CMS Final Remark) [YG occupancy: 292006 K (1179648 K)][Rescan (parallel) , 0.0397261 secs][weak refs processing,...

Wednesday, December 17, 2014 | Read More

Increased heap usage with G1 GC

G1 garbage collector is a predictable low pause collector that works efficiently to achieve the low pause time goals. To meet the set pause time goals, G1 tries to: pick only those many old heap regions for the collection-set that it thinks could be collected in the specified goal time adapt the young generation size to contain the collection times within the set goal. In that effort, however, G1 may end up causing a side effect of increased usage of the Java Heap. For example,...

Tuesday, November 25, 2014 | Read More

Running on a 64bit platform and still running out of memory

It sounds strange that one is running with 64bit JVM but still running out of native memory or Java Heap space. On a 64bit machine, with 64bit JVM we get almost unlimited memory then how is it possible to run out of memory? Well, it may happen in certain situations. HotSpot JVM has a feature called CompressedOops with which it uses 32-bit (compressed) pointers on 64-bit platforms to have smaller footprints and better performance on 64-bit platforms. 64-bit address values are...

Friday, October 10, 2014 | Read More

About G1 Garbage Collector, Permanent Generation and Metaspace

We received some questions around the G1 garbage collector and the use of Permanent Generation with it. There seems to be some confusion that the Hotspot JVM does not use permanent generation when G1 is used as the garbage collector. Here’s some clarification: JDK 7: PermGen Permanent Generation still exists in JDK 7 and its updates, and is used by all the garbage collectors. In JDK7, the effort to remove thepermanent generation was started and some parts of the data residing...

Wednesday, March 19, 2014 | Read More

UseLargePages on Linux

There is a JVM option UseLargePages (introduced in JDK 5.0u5) that can be used to request large memory pages from the system if large pages memory is supported by the system. The goal of the large page support is to optimize processor Translation-Lookaside Buffers and hence increase performance. Recently we saw few instances of HotSpot crashes with JDK7 on the Linux platform when using the large memory pages. 8013057: assert(_needs_gc || SafepointSynchronize::is_at_safepoint())...

Saturday, January 4, 2014 | Read More

Troubleshooting Long GC Pauses

Low pause times during the application run is the most important goal for many enterprise applications, especially for the transaction-based systems where long latencies can result in the transaction time-outs. For systems running on the Java Virtual Machines, garbage collections can sometimes be the cause of the long pauses. In this post I am going to describe different scenarios where we can encounter long GC pauses and how we can diagnose and troubleshoot these GC pauses. Cau...

Wednesday, July 31, 2013 | Read More

Understanding G1 GC Logs

The purpose of this post is to explain the meaning of GC logs generated with some tracing and diagnostic options for G1 GC. We will take a look at the output generated with PrintGCDetails which is a product flag and provides the most detailed level of information. Along with that, we will also look at the output of two diagnostic flags that get enabled with -XX:+UnlockDiagnosticVMOptions option - G1PrintRegionLivenessInfothat prints the occupancy and the amount of space used...

Monday, June 18, 2012 | Read More

Object Query Language Help

We can use a query language called Object Query Language with JHat and Serviceability Agent to query information from Java Heap. Help on this language is available with the JHat tool. I have put this handy help document on OQL here for quick reference.

Friday, April 6, 2012 | Read More

JRockit hang fix

Fix for the JRockit hang I talked about in my previous blog would be available in R28.2.4.

Friday, April 6, 2012 | Read More

Diagnosis of a JRockit Deadlock

Recently I came across an interesting JRockit JVM deadlock. I am sharing the details of that deadlock, its diagnosis, and how to workaround that deadlock, which might be useful until the fix for that is available in a future JRockit release. This deadlock occurred between an application Java thread and the JVM's Code Generation thread. The call traces for both the threads looked like this: "[ACTIVE] ExecuteThread: '0' for queue:  'weblogic.kernel.Default (self-tuning)'" id=15...

Friday, February 3, 2012 | Read More

JavaOne Presentation on SA Plugin for VisualVM

Here are the slides that I presented at the BOF session of 'Serviceability Agent Plugin for VisualVM' at JavaOne 2011. Serviceability Agent Plugin for VisualVM

Monday, October 24, 2011 | Sun | Read More

SA-Plugin for VisualVM

We all know Serviceability Agent - a great debugging tool for troubleshooting HotSpot VM problems. Now Serviceability Agent is also available in VisualVM through SAPlugin. More deatils here: https://visualvm.dev.java.net/saplugin.html

Sunday, May 16, 2010 | Sun | Read More

Adplus configuration

While working on a Java Plugin(on Windows) issue, I found one useful feature of Adplus. I was having a problem with LoadLibraryEx() and I was trying to load a dll into another process' address space but it was not getting loaded. Adplus report was showing DLL_Load and DLL_Unload First Chance Exceptions for the attempts of loading and then unloading the dll. I wanted to find why the dll was getting unloaded immediately after it was attempted to get loaded. And for this, Iwanted...

Wednesday, November 25, 2009 | Sun | Read More

G1 Collector

In 6u14, a preliminary version of the new Garbage First (G1) garbage collector is included. Garbage First, or G1, is a low pause, server style garbage collector targeted for multi-processors with large memories. G1's primary advantages over the Concurrent Mark-Sweep (CMS) collector include incremental compaction, better predictability and ease of use. For using G1, need to specify following command line options: -XX:+UnlockExperimentalVMOptions -XX:+UseG1GCThe following...

Sunday, July 12, 2009 | Sun | Read More

Important CMS Fixes

In this entry, I would like to talk about some CMS (Concurrent Mark Sweep) issues, their workarounds and the releases these are fixed in. \* 6558100: CMS crash following parallel work queue overflow. This crash is seen when -XX:+ParallelRefProcEnabled is set. Workaround is to use -XX:-ParallelRefProcEnabled. This is fixed in 1.4.2_17, 5.0u14 and 6u4 \* 6578335: CMS: BigApps failure with -XX:CMSInitiatingOccupancyFraction=1 -XX:+CMSMarkStackOverflowALot.For clarity sake, this...

Sunday, July 12, 2009 | Sun | Read More

More on Windows Crash Dumps for Java Processes...

User Mode Process Dumper There is another very good tool 'User Mode Process Dumper' that can be used to collect user dumps for crashing Java processes. You can get it and install it from here: http://www.microsoft.com/downloads/details.aspx? FamilyID=E089CA41-6A87-40C8-BF69-28AC08570B7E&displaylang=en After installation, run it from the Control Panel by clicking 'Process Dumper'. Add the application name to be monitored; java.exe in our case.Set the process monitoring rules...

Saturday, December 27, 2008 | Sun | Read More

Windows crash dumps for Java Processes

Windows Crash Dump is memory dump of a process running on a Windows system. These dumps can be very useful for debugging Java process crashes. In this entry I discuss how to collect sane Crash Dumps for Java process crashes on Windows machines that can later be analyzed using Windbg or other 'Debugging Tools For Windows'. I have a simple java class test which uses native library test.dll using JNI. test.dll implements a 'native' method where it accesses a null pointer and...

Saturday, October 18, 2008 | Sun | Read More

How to implement an interface using Reflection

I was trying to use classes and methods of an existing jar using Reflection. And there was an Interface in that jar which I had to implement. Now how to do that with Reflection ?? We can achieve this by using two classes in java.lang.reflect package; Proxy and InvocationHandler.java.lang.reflect.Proxy class accomplishes implementation of interfaces by dynamically creating a class that implements a set of given interfaces. This dynamic class creation is accomplished with the...

Thursday, July 24, 2008 | Sun | Read More

Utility to compare Class Histograms

We use the Class Histograms generated by -XX:+PrintClassHistogram jvm option quite often to diagnose memory leak issues. But we have to manually compare the histogram entries to analyze the difference between entries and the growth in heap. I have written a utility in Java that reads the histograms from the file containing these histogram dumps and shows them in a graphical interface. One can click on these histograms and view them in thisutility and can also compare any two...

Saturday, April 12, 2008 | Sun | Read More

How to create VC++ project for HotSpot workspace

I am sure very few people would know that on Windows platform, we can create a VC++ project for the HotSpot workspace in JDK sources. It is very useful as it gives me the ability to view and browse HotSpot source code and also build it at a simple click from Visual Studio IDE. It is also very helpful in debugging hotspot issues which can be reproduced on Windows with a simple testcase(and not just at customer site :) )So let's see how we can create this project for OpenJDK...

Sunday, March 9, 2008 | Sun | Read More

Debugging JVM hang with 'dbx'

Recently worked on a JVM hang problem, so posting here the steps followed in debugging this issue with dbx. Customer was facing hard hang on a Solaris-x86 machine with jdk 1.5.0_08 and the process was not even responding to kill -3. They sent us a core file of the hung process.For dbx to have the debug information, I re-built the product version of 1.5.0_08 libjvm.so with -g flag and loaded and used this new libjvm.so with dbx. With that, dbx was able to get all the debug...

Sunday, February 24, 2008 | Sun | Read More

CR 6454676

This fix for CR 6454676 (Need -XX:+HeapDumpOnCtrlBreak to trigger heap dump on ctrl-break or ctrl-\\) was made available in 1.4.2_18 and 5.0u14. But a format string used in this fix got currupted due to SCCS mis-interpretation and caused the following bug. 6615248: SCCS tag interpetation makes code invalid And due to this regression, HeapDumpOnCtrlBreak does not work correctly in 5.0u14. This problem has been fixed in 1.4.2_18 and 5.0u17.

Monday, February 11, 2008 | Sun | Read More

'jhat' sources in OpenJDK

'jhat' source code is available in OpenJDK. Open 'jdk' project in NetBeans and expand 'Sources For All Platforms'. You can find the sources of 'jhat' in these packages: com.sun.tools.hat com.sun.tools.hat.internal.model com.sun.tools.hat.internal.oql com.sun.tools.hat.internal.parser com.sun.tools.hat.internal.server com.sun.tools.hat.internal.utilIt is possile to just build jhat.jar instead of building the whole jdk. In com.sun.tools.hat, you will find ant makefile...

Sunday, December 9, 2007 | Sun | Read More

Heap Dump using -XX:+HeapDumpOnCtrlBreak

Heap Dumps are useful for Java Object Heap analysis. HeapDumpOnCtrlBreak jvm option would be available in upcoming 5.0 and 1.4.2 update releases that can be used to dump heap when a SIGQUIT signal is sent to the running process. One more feature would be available - if this option is used along with -XX:+PrintClassHistogram then both these tasks will be performed in one single VM Operation so that the Class Histogram and the Heap Dump represent the same snapshot of the...

Tuesday, December 4, 2007 | Sun | Read More

Building OpenJDK on Windows with NetBeans

Trying to build OpenJDK on Windows ? Here are some simple steps that may help... Here are the softwares required for the build. - Download OpenJDK from http://download.java.net/openjdk/jdk7/ and unzip the contents - Download and install jdk 6.0 from http://java.sun.com/javase/downloads/index.jsp and jdk 7.0 from http://download.java.net/jdk7/binaries - Download and install openjdk binary plugs from http://download.java.net/openjdk/jdk7/- Install Microsoft Visual Studio .NET...

Friday, November 30, 2007 | Sun | Read More

How to programmatically obtain GC information

Here's a small program which uses GarbageCollectorMXBean MBean to programmatically obtain the GC information of the running java process. Call it's printGCInfo in your applcation whenever you want to get the last GC information. import com.sun.management.GcInfo;import javax.management.MBeanServer;import java.lang.management.ManagementFactory;import com.sun.management.GarbageCollectorMXBean;import java.lang.management.MemoryUsage;import java.util.Iterator;import...

Thursday, July 5, 2007 | Sun | Read More

Collector names for GarbageCollectorMXBean MXBean

While trying to use GarbageCollectorMXBean to obtain GC information about a running process, I had to look around to find out the correct name strings for various Garbage Collectors. This is what I did to obtain an instance of GarbageCollectorMXBean: MBeanServer server = ManagementFactory.getPlatformMBeanServer(); GarbageCollectorMXBean bean = ManagementFactory.newPlatformMXBeanProxy(server, GC_BEAN_NAME, GarbageCollectorMXBean.class); where GC_BEAN_NAME is...

Monday, May 21, 2007 | Sun | Read More

Automatic Imports addition using NetBeans

NetBeans has a very useful keyboard shortcut: Alt+Shift+F. Upon pressing these keys, NetBeans adds all the required import statements at the top of the source file.

Sunday, May 13, 2007 | Sun | Read More

JVMTI agent to print reference paths

To get started on JVMTI please read JVMTI reference guide JVMTI reference guide and a very good article on JVMTI Programming here Here I will briefly describe how I used JVMTI to write an agent which prints the reference paths of instances of a given class in the running process. The global data structure used in this agent: /\* Global agent data structure \*/typedef struct { /\* JVMTI Environment \*/ jvmtiEnv \*jvmti; /\* Data access Lock \*/ jrawMonitorID ...

Friday, May 11, 2007 | Sun | Read More

jmap with CMS

Hotspot Servicability Agent, the backbone of jmap tool had a bug due to which jmap was not able to generate correct histogram and heap dump of applications running with Concurrent Mark Sweep collector. This bug 6311411 has been fixed in 5.0u12 and 6.0u2. So starting with these releases, jmap works fine with CMS collector.

Thursday, May 10, 2007 | Sun | Read More

Understanding CMS GC Logs

CMS GC with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps prints a lot of information. Understanding this information can help in fine tuning various parameters of the application and CMS to achieve best performance. Let's have a look at some of the CMS logs generated with 1.4.2_10:39.910: [GC 39.910: [ParNew: 261760K->0K(261952K), 0.2314667 secs] 262017K->26386K(1048384K), 0.2318679 secs]Young generation (ParNew) collection. Young generation capacity is 261952K and after...

Thursday, March 23, 2006 | Sun | Read More

Integrated Cloud Applications & Platform Services