Thursday Oct 18, 2012

A brief note for customers running SOA Suite on AIX platforms

When running Oracle SOA Suite with IBM JVMs on the AIX platform, we have seen performance slowdowns and/or memory leaks. On occasion, we have even encountered some OutOfMemoryError conditions and the concomittant Java coredump. If you are experiencing this issue, the resolution may be to configure -Dsun.reflect.inflationThreshold=0 in your JVM startup parameters.

https://www.ibm.com/developerworks/java/library/j-nativememory-aix/ contains a detailed discussion of the IBM AIX JVM memory model, but I will summarize my interpretation and understanding of it in the context of SOA Suite, below.

Java ClassLoaders on IBM JVMs are allocated a native memory area into which they are anticipated to map such things as jars loaded from the filesystem. This is an excellent memory optimization, as the file can be loaded into memory once and then shared amongst many JVMs on the same host, allowing for excellent horizontal scalability on AIX hosts.

However, Java ClassLoaders are not used exclusively for loading files from disk. A performance optimization by the Oracle Java language developers enables reflectively accessed data to optimize from a JNI call into Java bytecodes which are then amenable to hotspot optimizations, amongst other things.

This performance optimization is called inflation, and it is executed by generating a sun.reflect.DelegatingClassLoader instance dynamically to inject the Java bytecode into the virtual machine. It is generally considered an excellent optimization. However, it interacts very negatively with the native memory area allocated by the IBM JVM, effectively locking out memory that could otherwise be used by the Java process.

SOA Suite and WebLogic are both very large users of reflection code. They reflectively use many code paths in their operation, generating lots of DelegatingClassLoaders in normal operation. The IBM JVM slowdown and subsequent OutOfMemoryError are as a direct result of the Java memory consumed by the DelegatingClassLoader instances generated by SOA Suite and WebLogic. Java garbage collection runs more frequently to try and keep memory available, until it can no longer do so and throws OutOfMemoryError.

The setting sun.reflect.inflationThreshold=0 disables this optimization entirely, never allowing the JVM to generate the optimized reflection code.

IBM JVMs are susceptible to this issue primarily because all Java ClassLoaders have this native memory allocation, which is shared with the regular Java heap. Oracle JVMs don't automatically give all ClassLoaders a native memory area, and my understanding is that jar files are never mapped completely from shared memory in the same way as IBM does it. This results in different behaviour characteristics on IBM vs Oracle JVMs.

Thursday Feb 09, 2012

Analyzing Thread Dumps in Middleware - Part 4

This posting is the fourth and final section in the series Analyzing Thread Dumps in Middleware

In this post, we will see a new version of TDA (Thread Dump Analysis) tool developed by the author of this blog (Sabha Parameswaran in collaboration with his colleague, Eric Gross, also from Oracle A-Team) and its capabilities as well as some real world samples of thread dump analysis before concluding the series.

Analyzing Thread Dumps - Series:

Part 4: TDA A-Team and real world samples

Part 3: TDA tools

Part 2: How to capture and analyze Thread Dumps, navigate lock chains

Part 1: Basics of Thread states and thread locking

[Read More


Wednesday Feb 08, 2012

Analyzing Thread Dumps in Middleware - Part 3

This posting is the third section in the series Analyzing Thread Dumps in Middleware.

In this post, we will discuss some of the tools that can help with thread dump analysis as well as their limitations. In the next and final section in this series, we will look a new tool developed by the author of this blog in collaboration with his colleague in Oracle to fill the gaps mentioned with some of existing TDA tools in general as well as look at some real world thread dump analysis.


Analyzing Thread Dumps - Series:

Part 3: TDA tools

Part 2: How to capture and analyze Thread Dumps, navigate lock chains

Part 1: Basics of Thread states and thread locking

[Read More


Analyzing Thread Dumps in Middleware - Part 2

This posting is the second section in the series Analyzing Thread Dumps in Middleware

This section details with when and how to capture and analyze thread dumps with special focus on WebLogic Application Server related thread dumps as well as some tips and pointers and what to look for and optimize. The next post will go into tools that can help automate thread dumps analysis, some real world examples and limitations of thread dumps.

Analyzing Thread Dumps - Series:

Part 2: How to capture and analyze Thread Dumps, navigate lock chains

Part 1: Basics of Thread states and thread locking

[Read More


Analyzing Thread Dumps in Middleware - Part 1

How to analyze Thread dumps, for improving Middleware Performance (at App Server or Application level) as well as for general troubleshooting? 

This is part 1 of a series of posts on how to analyze thread dumps. In this post, we will go over basics of thread states and thread locking. In the following posts, we will drill deeper into capturing and analyzing Thread Dumps with special look into WebLogic Application Server specific thread dumps. 

Please see my blog for more details.


Wednesday Dec 14, 2011

Functional Programming in Java

How to use java as a functional programming language. Part One.[Read More]
About


This is the blog for the Oracle FMW Architects team fondly known as the A-Team. The A-Team is the central, technical, outbound team as part of the FMW Development organization working with Oracle's largest and most important customers. We support Oracle Sales, Consulting and Support when deep technical and architectural help is needed from Oracle Development.
Primarily this blog is tailored for SOA issues (BPEL, OSB, BPM, Adapters, CEP, B2B, JCAP)that are encountered by our team. Expect real solutions to customer problems, encountered during customer engagements.
We will highlight best practices, workarounds, architectural discussions, and discuss topics that are relevant in the SOA technical space today.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today