Wednesday Jan 06, 2010

WebSphere Hypervisor Edition and VirtualBox

Finally I got some time to check on the WebSphere Application Server Hypervisor Edition which according to the IBM Web-Site is optimized to run on Cloudburst appliances. I don't have appliance h/w access and I needed to check on how its packaged. Here is what I found in my first walk-through, more details will be added if I found them.
The software is packaged and available in several forms:
1. Appliance Image which is OVA formatted virtual image and intended to be used with Cloudburst appliance
2. ESX images to be used with VMWare ESX Servers
3. OVF formatted virtual machine image, which can be used on any virtualization platform where OVF is supported

The OVF image download file is composed of several files which will appear as different disk in VirtualBox and they hold different software and profiled such as WebSphere App Server, IHS and so on...
So I picked up the OVF image and decided to run it on the VirtualBox on my OpenSolaris box. As VirtualBox supports the OVF format and it can be imported using just few mouse clicks.
When I did that this is what I found:
The OS form some reason, mostly due to absence of Novell SUSE in my VirtualBox install appears as Ubuntu. So I continued with that and booted it. Everything went fine except I ran into a display issues as my VirtualBox did not like the display drivers that is part of the image currently available. More on that in next blog for the moment I just wanted to check how the install/startup and service configuration goes so I continued with a command line console access. After going through the license acceptance and select the right profile everything came back up as expected and I had a running instance of WebSphere Application Server with two profiles running in my VirtualBox. One of the profiles is for the "nodeagent" while other is regular "Application Server Profile".
This is what you need to try this:

  1. Obtain/install a copy of VirtualBox
  2. Download the HyperVisor Edition image in OVF format
  3. Import the OVF file
This should get you going. Later I decided to change it to OpenSuse option in VirtualBox and did not find any difference in behaviour so I assume its OK with either option.

Tuesday Jun 09, 2009

Speed up your SSL operation for IBM HTTP Server on Ultra SPARC T2 systems

Sometime back we have published a Sun Blueprint (Accelerating IBM HTTP Server Cryptographic Operations Using Sun Servers with CoolThreads Technology) detailing the steps needed to get your IBM HTTP Server use the on-chip crypto processor on Ultra SPARC T2 based system for SSL operation. This will give a free SSL operation boost without buying additional hardware for such operations.
The documentation lists all the steps needed to get your IBM HTTP Server and GSKit working  with this on-chip crypto module on Ultra SPARC T2 processor.  In addition to how to configure it, it also has results from some of the performance testing that has been done to measure the performance gain. Your milegae may vary depending on your type of workload but if you are making lot of new client connection and serving "HTTPS" traffic then this would something that is available to you free you want to consider. It wil help you take care of your SSL handshakes operations.
Another important aspect of this is that GSKit is a common library that has been used by IBM in lot of products. And as its evident from the name, Global Security Kit, it is security related implementation to be used across different products such as PKCS#11 or so. Some more details can be found at my prior blog about GSKit. This implies that if you want to hookup with PKCS#11 provider and take advantage of on-chip cryptography for other products that can be done too. You must note that this integration has happened at certain level of IBM HTTP Server so it requires certain version of GSKit embedded with the product for which you will try to take advantage.

Saturday May 30, 2009

IMPACT 2009 notes

It's been really busy since I came back from IMPACT 2009. It was a great event. Since then I have busy working on few stuffs for JavaOne and CommunityOne.
The opening ceremony was great and Billy Crystal was really hilarious.
As usual every company does some launches during these event and IBM followed the tradition by putting up "WebSphere Cloud Brust". It was quite confusing to lot of people including me but when I stopped by the Cloud Brust booth I got fair bit of idea about what it is and how it can provide the value add although I expected the booth demo to be live but is wasn't the case.
Lot of session has talked by processor advancement, memory placement, bus speed and what not..During these there were numerous references to Sun Ultra SPARC T2 line of processor which leads the multi-core revolution. You might have got different perspective of different speaker on this but if you have used it you know the power, processing and power efficiency included, of these systems. If you haven't tried your hand then do and know it for yourself.

I expected this to be on linux/xen combo but surprisingly it was based on VMWare hypervisor that was very interesting. Similar deployment toplogy can be achieved using no additional technology overhead with Solaris such  zones/container or if you are ready to try your hands on OpenSolaris you can use xen. Remember OpenSolaris has tool to get your VMWare images working on OpenSolaris.
Here are few important pictures I took.

WebSphere Cloud Brust announcement slide:

Thursday Apr 30, 2009

IMPACT 2009 @ Las Vegas

Wanted to let you all know that Sun/IBM Team will be hosting the following BOF during this year IMPACT 2009.
Title: Optimizing your IBM WebSphere® Infrastructure with Solaris
Session ID: BOF-3174A
Session Time: Tue, May 5 12:00 PM - 01:00 PM
Location: Venetian Galileo 904
This will give you an unique opportunity to talk to engineers from both Sun and IBM team. We can cover whole range of topics concerning your WebSphere deployment on Sun platform.

For rest of the discussion I suggest you stop by at our booth and we can help you get your question answered.

Tuesday Apr 14, 2009

Scaling up WebSphere on Sun Blade X6270 Server

As always I got chance to try my hands on one of the new blade server that Sun has released based on Intel Xeon processor 5500 series namely, Sun Blade X6270 server module. The new Xeon processor has lot of features to deliver better performance based on the workload by which it is driven eg. automatically increasing processor frequency when needed and possible, as well as the capability to have extra hardware threads or taking advantage of the processor's power and thermal headroom. I would suggest you to visit for more details from domain expert including power consumption to virtualization and everything else supplemented by best OS on the planet "Solaris 10/OpenSolaris". I will leave rest of the discussion to my distinguished colleagues at Sun and will talk about what I observed when running WebSphere Application Server on this system.

As most current version of WebSphere Application Server available was V7.0 so I picked up this installed then made sure I am running with the latest fixpacks and ran it. Everything was breeze from installation, fix-pack upgrade to start/stop and then benchmarking. The Sun Blade X6270 I had was built on 2 sockets of the Intel Xeon L5520 processor and the Solaris 10 10/08 Operating System.

As always I used the same workload that I use for my WebSphere tests and there are other components which included the Database Server which was running DB2, and the test Drivers. For benchmark I used Faban which is very easy to use and run and it does the bookkeeping of benchmark so nicely that I don't have to keep records myself. I don't constrain my DB in terms of performance so I let it run in "/tmp" to just make everything simple so I don't spend too much time on tuning DB as I am working on WebSphere App Server.

The Intel Xeon processor 5500 series has two modes for running namely, Turbo mode and Hyper-Threading mode options in the BIOS setup. Which can be changed during boot by going to BIOS menu and doing the right selection. As Intel has put up "Nehalem, unleashes parallel processing performance enabled by an integrated memory controller and Intel® QuickPath Technology providing high-speed interconnects per independent processing core. The Intel® Turbo Boost Technology taking advantage of the processor's power and thermal headroom. This enables increased performance of both multi-threaded and single-threaded workloads and with Intel HT Technology, bringing high-performance applications into mainstream computing with 1-16+ threads optimized for new generation multi-core processor architecture." I did tried to run in both modes did not notice any significant difference for my tests while running in these modes.

Single instance of server was able to use much of the system capacity and stopped scaling. I was running all the tools and nicstat was one able to point me the moment when the n/w bandwidth was exhausted it stopped taking any more load. Then to work around the network capacity I put a direct connection with the DB Server which resulted in little better throughput and little more system utilization. On the network communication between DB and Server was happening over this interface. Still not enough and server had capacity neither the thread-dumps or system stats pointed out any problem other than network again being saturated. In which the public interface was full loaded and there was a very light load on the point to point connection interface. At this point I don't had any choice other than adding another instance of the server. To do this I created a Solaris Zone/Container and had one instance running into this zone. For this zone I used the point to point connection between the DB and the blade server which eliminated this and I was able to use the server to full capacity.

The network generated interrupts and there distribution can be checked by using the following command:
-bash-3.00# echo ::interrupts | mdb -k | grep igb
49 0x60 6 MSI-X 0 1 - igb_intr_tx_other
50 0x61 6 MSI-X 0 1 - igb_intr_rx
51 0x62 6 MSI-X 4 1 - igb_intr_tx_other
52 0x63 6 MSI-X 7 1 - igb_intr_rx

I looked for igb only becuase my network drivers are as:
igb0 - public (ie. going over GB switch)
igb1 - private (point-to-point)
From the above it is clear that the tx and rx interrupts are going over virtual cpu or more technically threads 0, 4 and 7 which is very well distributed.

So what did I do for tuning the WebSphere ?
My tunings remains same as I was not changing the Operating System and Software which includes Java. I will list them here for reference:

Disabled the Performance Monitoring Infrastructure
Disabled the Application Profiling
I used the DynaCache for the App just as to save some DB overhead
Set the WebContainer thread to the same number of threads as load driver
For Connection Pool I made sure that the number of DB connections are more than my WebContainer thread pool so none of the thread end up waiting for DB connections.
initialHeapSize="2048" maximumHeapSize="2048" -server -Xmn1024m -XX:+AggressiveOpts -XX:+UseParallelGC -XX:ParallelGCThreads=8 -XX:+UseParallelOldGC

Before I conclude I would like you to know that the only 64-bit of WebSphere is available on this platform so if your application happen to be memory hungry you are not limited by a 4GB address space and as you notice from David Dagastine blog that there are lot of things happening with 64-bit JVM and how it is getting optimized compared to 32-bit variant. But you may have to wait for the appropriate fixpacks to get to that. I will update as soon as it becomes available.
There are lot of other advantages of this server and I would encourage you to read more about this @

Monday Mar 23, 2009

Free cryptography for WebSphere on Ultra SPARC T2 and T2 plus systems

If you have been being hit by the processing overhead of SSL stuff and have to buy special purpose hardware to speed up things, then may be its time to leave all this processing to Ultra SPARC T2 and T2 Plus based system. Which has builtin cryptogarphic support on the chip and websphere supports this:

Here are the details from IBM WebSite:

For details of setup you can follow the WebSphere Infocenter.
Please note that this feature will be available only after certain fix pack release so it will not work unless you upgrade your JDK to required level as suggested in the doc.

If you want to learn more about the Cryptographic Acceleration offered by these processor I would encourage to read this Sun Blueprint titled "Using the Cryptographic Accelerators in the UltraSPARC® T1 and T2 Processors".

Ask the expert session for WebSphere on Solaris

We will be hosting an Ask The Experts session on March 24th. 
We invite you all to join where we can discuss and share various ideas around how to optimally use WebSphere software on Solaris platform.
For  details please visit the WebSphere User Group web-site(

Thursday Jan 15, 2009

OpenSolaris and WebSphere

As lot of you may have tried already installing/using WebSphere on OpenSolaris and may have ran into some issues during install so I thought I will cover some of those issues into this blog and will provide possible workaround.
The instruction below applies to WebSphere Application Server v6.1 and v7.0. I havn not  tested 6.0.X but I assume it will work fine. If not then let me know. 

OpenSolaris brings in lot of new feature details can be found at If you don't want to commit a hard disk for OpenSolaris experience you can use VirtualBox and then install OpenSolaris inside VirtualBox. Once you figure out which route you want to take and how you want to install OpenSolaris then you need to grab installer for WebSphere and try installing it over. You will have following few options available to you to install:
Silent installation:

You will have to edit the template response file. After you make all the necessary changes if you try to install it will fail and if you check the log it will complain that this is not one of supported OS. The reason for this can be checked by the output of the command "uname -a" and as you can see from output it is being reported as "5.11".  So this is what you will have to add in your response file:
-OPT disableOSPrereqChecking="true"
Now try again and installation should complete without any error and you will get message about the Operating System version. Just ignore this message and start using.
Installing using the Installer GUI:
When you try to install using it will come up but when it invokes the install command then it will fail complaining the installation cannot be started in GUI and suggest that you use "silent" installer. This error happens only with WAS v6.1 only WAS v7.0 is fine and you can install without any issues. If you want to get into details you can check the niflog  and you will understand that this is happening due to installshield issues. Googling for this will reveal that this is a known issue with lot of platform eg. AIX/Linux/Solaris and it has nothing to do with either the OS or WAS.
So we can use the java based installe by directly invoking the following command:
bash # java -jar setup.jar
This to succeed you must have java runtime in your path which is by default available in OpenSolaris. It will bring up the installer and as said earlier "Prerequisite Check" will fail and if you still continue everything will be installed right.

Just to make sure the "Install Verification Test" and you will observe all the test passing now you are good to go.
If you runinto any display issues it will be worth checking this blog.

Thursday Jan 08, 2009

WebSphere Application Server V7.0

By this time you might have got your hand on new WebSphere Application Server namely WebSphere Application Server v7.0. There are lot of good thing about this version as compared to its predecessors. A list of which can be found easily from 7.0 InfoCenter.
Some of the feature as Solaris user you should be aware are:

  • New JDK/JVM Support

  • DTrace made easy

  • Performance Enhancement

  • New J2EE Spec Level Compliance

All the detail can be found at
Some of the differences you can note from the standard Java Installation with the one that is part of WAS 7 s missing sample directory which contains lot of samples including DTrace scripts which are ready to be used. If you want to try them grab a JDK 6.X installer. Install it and copy the dir YOUR_JAVA_INSTALL_DIR/sample/dtrace and start using the script as it is. There is a README.txt file which included in the dtrace dir. All the sample scripts also has instruction about how to use them. More information about DTrace can be found here.

Monday Oct 13, 2008

WebSphere Application Server and T5440

Today, we are announcing the next generation server based on UltraSPARC T2 Plus processor and this server happens to be a monster in terms of performance. It can consolidate a lot of servers in a single box providing the similar combined throughput. For the server details and other blogs related to this server I suggest you to visit T5440 Blog index(
In this blog, I summarize the performance of IBM WebSphere Application Server (WAS)  on T5440. To test and benchmark the performance, I use the most recent release WAS v7, that is based on Java EE 5 spec and JDK 6, with numerous features and performance enhancements. As I said earlier that this new Sun server is a monster in terms of performance, the WAS software set up and configuration require some planning and appropriate allocation of the system resources.   When you do the psrinfo(1M) command on this server, it will report to you that the system has 256 "processors".  This means that the processing power of this system far exceeds the software scalability of a single instance of the Application Server's. Thus, you will need multiple instances of WAS to drive the system to its full utilization.  Solaris Containers provides the most efficient way to accomplish such configuration as Solaris Containers provides process space isolation among different WAS instance as well as allocating proper system resources.
For maximizing the utilization of the system, I configured the environment as follows.

  • I created 7 Solaris Containers, allocated 32 processor threads to each of the 6 Containers, and allocated the remaining processor threads for the other container and the global zone.  Then, I used the following WAS configuration:

  • initialHeapSize="2500" maximumHeapSize="2500"

  • -server -Xmn2048m -XX:+AggressiveOpts -XX:+UseParallelGC -XX:ParallelGCThreads=16 -XX:+UseParallelOldGC -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl -Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XML11Configuration

  • Disable the PMI feature.

  • Disable the System.out logging from the admin console if you want you can do the same by editing the config file. by editing the trace service like - startupTraceSpecification="\*=info:SystemOut=off"

  • DynaCache - The DynaCache was used for this benchmark so the default size of the cahce was set to 2000 which wasn't enough for this test which can be adjusted accordingly based on the application need as I set it to 3000.

  • For thread pool tuning the only thing that was relevant in this benchmark was tuning the WebContainer thread pool right so I did set it to 80 and which was more that enough and I think It can be scaled down a bit but havn't tried.

  • Database connection pool was set it to same value as WebContainer pool and set the min and max to the same value.

For database I did not want to run into the network/disk contention so I created the DB in /tmp and I had to use two databases for this purpose. This was just to eliminated the Database configuration headaches and just measure the App Server machine scalability. These database were connected point-to-point with App Server box and the App Server instances were using the point-to-point connection for the DB.
This resulted in getting to 1.8X scalability of its predecessor system(T5140) in terms of throughput(which is measured for this benchmark in terms of requests served per second or more commonly known as req/sec).
So In nutshell if you are looking for another system to consolidate your WebSphere deployment with 1.8X throughput capacity of earlier UltraSparc T2 plus system then T5440 may be just the right system for you.

Wednesday Apr 23, 2008

Thank you all for conversation during IMPACT 2008, Las Vegas

We would like to thank all those who stopped by and talked to us during IMPACT. It was a great conference and it was great to talk to you all about your Solaris 10 deployments and how well your deployment is running on Solaris 10.
There were few really great session and I would suggest you get a copy of some of the interesting presentation eg.
1. IBM WebSphere and multicore platforms: Performance, scalability, and best practices
2. Exploring IBM Java 5 and Java 6: New features and enhancements
3. Understanding and tuning the Java Virtual Machine for IBM WebSphere
for you reference. They contain lots of good information which you may found useful later.

Albert has already posted the slides which we used during our presentation. They can be found here:
SIG2730: WebSphere Deployment on Solaris
TSM-2447: WebSphere Performance On Solaris

There was one other session which was on DumpAnalyzer which is not available on Solaris or HP-UX I will put another blog which will list out all the equivalent tools that already available in the Java you have on your system with WebSphere.

Saturday Apr 05, 2008

Using JConsole

Using the jconsole tool
J2SE 5.0 has comprehensive monitoring and management support. Among the tools
included in the JDK download is a Java Management Extensions (JMX)-compliant monitoring
tool called jconsole. The tool uses the built-in JMX instrumentation in the Java Virtual
Machine to provide information on performance and resource consumption of running
applications. Although the tool is included in the JDK download it can also be used to monitor
and manage applications deployed with the Java 2 Platform Standard Edition Runtime
Environment 5.0 (JRE 5.0).
Jconsole can attach to any application that is started with the JMX agent. A system property
defined on the command line enables the JMX agent. Once attached jconsole can be used to
display useful information such as thread usage, memory consumption, and details about
class loading, runtime compilation, and the operating system.

Figure jconsole Summary tab
The Summary tab provides the following information
– Uptime: how long the JVM has been running
– Total compile time: the amount of time spent in runtime compilation
– Process CPU time: the total amount of CPU time consumed by the JVM

Figure jconsole Memory tab
The Memory tab shown in above Figure provides the following information.
– Current heap size: Number of Kbytes currently occupied by the heap
– Committed memory: Total amount of memory allocated for use by the heap
– Maximum heap size: Maximum number of Kbytes occupied by the heap
– Objects pending for finalization: Number of objects pending for finalization
– Garbage collector information: Information on GC, including the garbage collector
names, number of collections performed, and total time spent performing GC
A view of the memory pictured in above Figure shows two events of interest. At
02:20 a garbage collection was triggered from the Jconsole tool. At 02:50 an
OutOfMemoryError was thrown by an application.

Figure jconsole Threads tab

Figure Thread detail

The Threads tab shown in above Figure provides the following information.
– Live threads: Current number of live daemon threads plus non-daemon threads
– Peak: Highest number of live threads since JVM started
– Daemon threads: Current number of live daemon threads
– Total started: Total number of threads started since the JVM started (including
daemon, nondaemon, and terminated)
The Classes tab provides the following information.
– Current classes loaded: Number of classes currently loaded into memory

– Total classes loaded: Total number of classes loaded into memory since the JVM
started, including those subsequently unloaded
– Total classes unloaded: Number of classes unloaded from memory since the JVM

Figure jconsole VM tab
The VM tab shown in above Figure provides the following information.
– JVM type and version
– Uptime
– VM command line arguments
– Class, Library, and Boot Class paths
– JIT Compiler
– Operating System information
In addition to monitoring, jconsole can be used to dynamically change several parameters in
the running system. For example, the setting of the -verbose:gc option can be changed so
that garbage collection trace output can be dynamically enabled or disabled for a running
application. You can also perform a GC by clicking the Perform GC button on the Memory
Getting started with jconsole
1. Start the application with the option. This option sets
the system property which enabled the JMX agent.
2. Start Jconsole with the jconsole command. Jconsole ships as a binary in the
$JAVA_HOME/bin directory. (JAVA_HOME indicates the home directory of the J2SE
installation. For example, /opt/IBM/WebSphere/AppServer/java/bin)

Figure Jconsole Connect to Agent dialog box
3. When Jconsole starts it shows a window with the list of managed VMs on the machine as
shown in Figure 9-12. The process-id (pid) and command line arguments for each VM are
printed. Select a VM, click Connect, and Jconsole will attach to it.
For more information on the monitoring and management features, and details on how to use
Jconsole, read the following document.

Tuesday Mar 18, 2008

Coming to IMPACT 2008

This year IBM is having IMPACT 2008 event in Las Vegas, NV from 6 April 2008. If you are planning to be there then here are the few sessions which we are presenting you may find interesting. BOF is going to be very very interesting and we are planning to have a open discussion where we can go through whatever is most needed by customers/WAS admins for their WAS deployments on Solaris 10. If you are looking for some collaterals or white-papers which is not available today we can certainly take down your request, prioritize them and then make them available as soon as possible. So don't miss this opportunities to talk to us and let us know what are the information needed for worry free deployment of WAS on Solaris. The details of the sessions are as follows which we will be presenting :

1) Technical Session: Performance Best Practices for WebSphere Application Server Version 6.1 on Solaris 10
Session #: TSM-2447
Session Time: Thu, 10/Apr, 04:45 PM - 06:00 PM
Location: Room 118

2) BOF: Deployment Best Practices for WebSphere Application Server Version 6.1 on Solaris 10
Session #: SIG-2730A

3) IT Executive Session: Sun and IBM WebSphere Message Broker: Configuration, Deployment and tuning on Solaris 10
Session #: ITE-2695

Wednesday Jan 16, 2008

Redbook for WebSphere on Solaris

Last couple of week we don't had any update over here as I was working with some of the IBM folks(really great ones..) on a Redbook project. The redbook is titled something like "IBM WebSphere on Sun Solaris" and will be available in couple of weeks. Just wanted to let everyone know that this is going to be a onestop information for your Websphere deployment on Solaris. I am sure some of the chapters are really great and will answer may of the questions being asked frequently.

Here is the link to the internet draft:

Monday Oct 22, 2007

UltraSPARC T2 and WebSphere Application Server licensing update

Following the release of this new processor and server based on this IBM has updated the PUV(Processor Unit Value) licensing information.
I will not elaborate on the cost benefit analysis but you can make out from the data that this seems to be very cost effective server in terms of licensing cost. The details are here for your reference:
Processor Value Unit table




« April 2014