Thursday Feb 01, 2007

WebSphere v6.1 tuning simplified on Solaris 10

I have been working on some simplification for the WebSphere App Server V6.1 performance tuning on Solaris 10 platform and it came to my attention that instead of playing with too many flags there is only one flag should be more than good enough to give the boost needed for performance. However this may not apply to all the work load but I would rather try and determine that and start overriding some of the defaults:

Previously the tuning knobs of interest used to be like following:

  • initialHeapSize= eg. 2048
  • maximumHeapSize= eg. 2048
  • -server
  • -Xmn1024m At least 50% of the heap size; more may be better
  • -XX:-ScavengeBeforeFullGC
  • -XX:MaxTenuringThreshold=
  • -XX:+UseParallelGC
  • -XX:ParallelGCThreads= either equal to number of cpu or on multi core systems set it to somewhere between .5-1xNumber of cores.
  • -XX:+AggressiveHeap
  • -XX:PermSize=
  • -XX:+UseParallelOldGC

All of this can be reduced to just couple of flags and it will give more throughput than this one:
So the new reccomendations are:

  • initialHeapSize= eg. 2048
  • maximumHeapSize= eg. 2048
  • -server May not be needed in most cases
  • -XX:+AggressiveOpts -XX:+UnlockDiagnosticVMOptions -XX:-EliminateZeroing

This will take care of JVM tuning and you should spend time setting the websphere thread pools right.
Let me know if you find it other way...

Wednesday Jan 10, 2007

WebSphere Application Server (WAS) packaging on different Solaris platform

By this time you may be already aware that WebSphere Application Server(WAS) 6.0.2.X and beyond is available on Solaris 10 which gives you flexibility to exploit the greatest operating system with different kind of h/w eg. SPRAC, AMD . There are few enhancement which has been done to improve the packaging of WebSphere Software. Going forward may be this is the way WebSphere sofware will be packaged:
V6.1 of WebSphere Application Server has introduced the concept of non-root installer on Solaris platform and I think that's a good decision the IBM engineering has taken so you don't have to give the root privilages to WAS administartors.
Here I will try to explain what are the differences you will notice due to this changes:

  • When you use the root installer the installation will add the WAS software as Solaris packages which will make an entry into operating system registery.

  • When you use the non-root installer it will not add the Solaris packages

However for Solaris x64 processor-based systems WebSphere has taken this to another level and wether you use the root or non-root installer it won't add any packages to the Solaris package registery. Which I think is a very feature for people doing some engineering work and does the install/uninstall many times. During uninstallation if the package removal fails for some reason then you have to go to package registry and do some work to clean the system called manual uninstallation. But in the cases like no package registery you can simply delete the directory and remove the appropriate installation specific information which is usually found in the home directory of the user with which the WAS was installed eg ~/.ibm. You can also use installRegistryUtils also to find out the registry information.

Tuesday Nov 14, 2006

Using Sun Studio with WebSphere Application Server

Profiling WebSphere Servers Using Sun Studio Performance Tools

This page describes profiling servers being run under IBM's WebSphere Application Server system with the Sun Studio Performance Tools.

To profile WebSphere Application Server(WAS), you will need to have a release of the Sun Studio Performance Tools, picked from the release list, properly installed and configured.
All uses of the Sun Studio Performance Tools require a supported version of Solaris including any required patches. They also need a supported version of Java installed which can be used from WAS installation, this is bundeled and gets installed with WAS. Running the collect command with no arguments will run a script that will verify that all the appropriate patches are installed.
A server run under IBM's WebSphere is written in Java; (see "Profiling Java Applications"). A server is launched by running a script which invokes the JVM to run it. To profile a server, you must ensure that the JVM command launching the server is prepended with a collect command, with appropriate arguments, to invoke the Sun Studio Collector. In the discussion below, the shell variable ${COLLECTOR} is used to refer to that command and arguments.

In the next section, we describe setting ${COLLECTOR}, that is, determining the data collection options you will want to use. In the following section we describe the scripts used to launch a server, and how to edit them to insert ${COLLECTOR}. In the third section, we describe navigating through the measured performance data. In the last section, we give links to the IBM WebSphere documentation.

Data Collection Options
This section describes the setting of collection options for profiling, that is determining what to use for ${COLLECTOR} in WebSphere startup scripts.

Data Options
The command collect is the first token in the ${COLLECTOR}
variable. You may be specify it either as an absolute path to wherever you have installed the tools, or as the command name only, provided you have set your path to include the installation.
The default experiment is a clock-profiling experiment. You may also use hardware counter profiling or sychronization tracing. The experiment begins when the server is launched, and terminates when the server exits. If the profiling session is longer than about 20 minutes, low-resolution profiling should be used to avoid high volumes of data. Since WAS servers are Java-based, the -j on option to collect is always needed.

Experiment Naming and Storage
You can simplify the management of experiments by being careful at naming and storing them. You may name experiments with a -o argument specifying a name including the name of the server, followed by "". The first experiment will have the number 1, and subsequent ones will get consecutive numbers.
You can also use the -d argument to specify a directory into which the experiments should be stored. It should be a locally-mounted, as opposed to NFS-mounted, file system.
You may want to use the date command to generate a string representing the current date and time and embed that string in the experiment name.

Signal Controls
You may use the -y <signal> option to collect to control when data is recorded during the session, and minimize the volume of collected data.
You may start the experiment with data collection paused (thus not profiling the server during its startup), and send the signal to resume data collection either just before or shortly after applying a benchmark load to the server.
You would send the same signal to pause data collection either shortly before or just after the benchmark load is terminated.
You may repeat the resume/pause sequence for other loads, and terminate the server after all are done.

To profile server initialization, you would use -y <signal>,r, send the signal after initialization, and then send it again before and after a load is applied, as above.
If data volume is not a problem, but you will be running multiple loads in a single session, you may pass the -l <signal> option to collect, and send the signal to insert sample markers after initialization and then after each benchmark load is applied.
When using either of these techniques, you should disable periodic sampling, using the -S off option to collect.
WebSphere does not interfere SIGPROF, so you can use it to generate samples or to toggle pause/resume, or to generate samples, but not both. Such use does not interfere with SIGPROF usage for clock-profiling. You can use a second signal, SIGUSR1, for example, if you want to both control pause and resume and to insert
sample markers in the experiment.
Descendant Process Controls
Usually WAS Servers do not spawn additional processes, so -F on is not needed. If the server you are profiling does create additional processes
that you wish to profile, you should enable -F to profile them, either for all descendants, or selectively, based on the name or lineage of the descendant of
Archiving Controls
Depending on the environment under which you are profiling WAS Servers, you may examine the data on the machine on which it was recorded.
If you want to examine it on a different machine, you should set -A copy to make the experiment more portable.

Initiating Data Collection

This section describes how you can modify the scripts used to launch WAS servers to enable data collection.

Launching the Servers with Collector/Analyzer
WAS servers are simply launched by explicitly invoking the following script from the bin directory of the profile under which this server has been created.  <server name>
eg. server1
where "server1" is the name of the server.

To start WAS with Sun Studio Collector/Analyzer tool we need to start server with "collect -j on" option so that data collection can be enabled.

To do so follow the following steps:

  1. Go to the $PROFILE_DIR/bin dir.
  2. Issues the command: <server_name> -script <script_name_to_be_used_with_collector>
  3. This will create a script name <script_name_to_be_used_with_collector>
  4. Now the script need to be modified to point to Sun Studio Collector/Analyzer tool binary location and start the java process with "collect -j on"
  5. Edit the script to have the Sun Studio bin dir in the PATH settings
    1. PATH=$PATH:/java/devtools/sparc/SUNWspro/SS11/bin
    2. You can put it either before or after . $binDir/ line.
  6. Then replace "exec" in the script with "collect -j on". Save and quit the editor.
    1. exec "/opt/IBM-startup/WebSphere/AppServer/java/bin/java"
      will become collect -j on
    2. This is really a long line.
    3. You may not find exec if you are using 6.0.X based WAS
      installation then in that case you can just append "collect -j on " to
      the last line where the java commnad is launcehd.

  7. The server is now ready to be run with collector.

The data will be collected in the directory from where you will invoke the <script_name_to_be_used_with_collector>
script. You can invoke this script from anywhere as it doesn't have any relative path info.

To force it to collect data in particular directory you mya want to use "-d" option in the script:

-d <directory>        specify experiment directory

An example of such a complex script can be found in the file

More information about the collect and analyze can be found in collect.1 man page.

Server Profiles
This section describes tips for examining the server profiles.
Unless your server has created additional descendant processes which are profiled, and single run creates a single experiment. No experiment filtering of selectivity is needed.
If you have used either of the signal mechanisms described above you may use sample filtering in the Analyzer or er_print to examine the profile for only part of the run. For example, you may want to look at the startup only, or you may want to look at the data for the individual benchmark loads that were run.

When looking at the Timeline in Analyzer, you may want to color methods from the WAS infrastructure all one color, and use other coloring to look at specific sections of you code. You can also show or hide clock profile events that are not CPU-time (that is, they are system CPU-time, or any of the wait states).
This is live document and we will keep on updating this more current updates.

WAS Documentation
Documentation for WAS 6.1 can be found on IBM's web site.
Some topics of interest are:

Thursday Oct 26, 2006

WebSphere V6.0.2 and JVM 1.4.2 tuning on Solaris 10

I have been asked this information many time so here is how you should tune the JVM for WebSphere Application Server on Solaris 10.

The tuning knobs of interest:

  • initialHeapSize= eg. 2048
  • maximumHeapSize= eg. 2048
  • -server
  • -Xnoclassgc May not be needed for every other app
  • -Xmn1024m At least 50% of the heap size; more may be better
  • -XX:-UseAdaptiveSizePolicy
  • -XX:MaxTenuringThreshold=4
  • -XX:+UseParallelGC
  • -XX:ParallelGCThreads= either equal to number of cpu or on multi core systems set it to somewhere between .5-1xNumber of cores.
  • -XX:+AggressiveHeap

Couple of things to note here when doing experimental work:
You can try scaling up MaxTenuringThreshold to find out really at what time the object need to be promoted. Then similar things can be tried for Xmn or new generation size if you know for sure that you are going to generate lot of short lived objects then set it to higher percentage of Maximum Heap size.
The other usual WebSphere App Server tuning would include following params:
ORB Thread Pool
WebContaine Thread Pool
JDBC Connections
EJB Cache
HTTP Transport






More about java performance check out:

Tuesday Oct 24, 2006

JSP pages compilation failure with WebSphere Application Server with java.lang.NoClassDefFoundError

I happen to run into java.lang.NoClassDefFoundError when trying to run one of the sample JSP pages(hitcount) on WebSphere Application Server V6.1. Here is the exception trace(first few lines only):
[10/13/06 16:32:51:941 PDT] 00000026 WebApp E [Servlet

As we know that V6.1 has Eclipse based JSP compiler which uses JDT(Java Development Tooling) and AST(Abstract Syntax Tree) Parser for parsing and generating the java code.

This compiler depends on some of the user env settings and if not propogated well to the compiler during the compiler initialization it may fail to initilize properly. There are two workarounds to this problem:

  • The easiest one is to install as any non-root user and everything should work as expected.
  • The other route you can take is modify your web archive descriptor to have JDK compiler or you can use jikes as well.

To modify the compiler used for you JSP compilation you can edit your ibm-web-ext.xmi file under the web module deployment directory and can have a line similar to this:

<jspAttributes xmi:id="JSPAttribute_XXXXX" name="useJDKCompiler" value="true"/>

For XXXXX you can put any number all you have to do is just make sure JSPAttribute_XXXXX is unique. There are bunch of other as well which can be changed for you web-application.
IBM web-site looks like the fix for this is also going to be released very soon...

Tuesday Sep 26, 2006

Does memory usage really increase when upgrading to WebSphere Application Server V6.1 on Solaris ?

In reference to IBM Flash(Alert) that has been posted by IBM does not provide the
technical details and does not discuss about why one would observe this
growth in memory if not used in way they have been using it with J2SE
1.4.   In nutshell, this is not a growth in memory.  It
is just the technology changes from J2SE 1.4.2 to 1.5.  In J2SE
5.0, we all know that a lot of things has been

Some of the defaults(configuration) that were used with J2SE 1.4.2
before has been changed in 1.5.  As a result, the Java Virtual
Machine (JVM) if started on the same hardware or server will have
different settings with 1.5 JVM which is tuned mostly for the server
class application as opposed to 1.4.2 which is tuned for client class
uses.  When you start your JVM without supplying any JVM options
(defaults) with 1.4.2 and 1.5, you will find lot of difference in the
JVM settings in the two JVM's. We need to understand a bit about this
as to why this is changed.  To understand it well, we have to look
at the changes in J2SE 1.5 that has introduced following new feature
which is relevant in this case:

I am sure if you are interesetd you will follow the link and get the
details but here is a summary whats happening in here. When 1.5 JVM is
started on a particular classe of machine, it adjusts the JVM
parameters based on resources on the machine e.g. memory and cpu. It is
not just the heap size settings (e.g. Xms and Xmx), there are other
options as well which will have different settings. If your machine is
deteced to be the server class machine, it is going to start the server
JVM otherwise it will start the client  JVM. While in J2SE 1.4.2,
client JVM has always been the case unless you specify the "-server"
excplicitly in your settings. Then, the Garbage Collector is also
different on the class of machine you have.

The IBM Flash tuning reccomendation is just to override the Garbage
Collector Ergornomics and Server-Class machine detection feature of the
J2SE 1.5 and bring it to the same options level as 1.4.2 JVM. It makes
sense here as now we will be comparing the two JVM at the same level
and you will find that the 1.5 JVM will have the lesser footprint. You
may find people using the term like de-tuning for this but actually you
may by this time will understand that this is actually not de-tuning
but bringing the JVM 1.5 to 1.4.2 level and doing an apple to apple
comparision. Not many production system that drive really heavy
workloads will use in these type of constrained environment and will
favour the ergonomics feature of the 1.5 JVM.

If you are interested in more details about the Sun JVM tuning, there
are plenty of resources to help you such as Java SE Performance is great place to start.

Thursday Aug 17, 2006

WebSphere SpecJAppServer2004 benchmark on Sun Fire T2000

Sun Fire T2000 so far has been the best J2EE app server platform [Read More]

Wednesday Aug 16, 2006

WebSphere Application Server V6.X in Solaris 10 Zone container

Virtualization techniques for WebSphere [Read More]

Monday Aug 14, 2006

WebSphere Application Server V6.1 on Sun Fire T2000

How to tune WebSphere on Niagara servers[Read More]

My first blog


[Read More]



« February 2015