Monday Mar 28, 2011

Configuring your Linux server to use largepages for JRockit

JRockit can use the largepages on Linux platform and it can yield better performance for memory/gc intensive application. However on platform like Solaris and with Hotspot the largepages are already enabled you just have to use them, while on Linux it would require following to get to use the LargePages:
Enable the LargePages in the kernel (using CONFIG_HUGETLB_PAGE and CONFIG_HUGETLBFS).
In most cases we don't have to do the above and we can check if its available for use:

# cat /proc/meminfo | grep Huge
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
This tells us that this system support largepages of 2 MB in size

No we need to tell the kernel how much memory we need to reserve which can be done by:

# echo 4096 > /proc/sys/vm/nr_hugepages
This would reserve 8 GB of memory for largepages.

This change can be made permanent in /etc/sysctl.conf by adding followig line:
vm.nr_hugepages = 4096  

nodev            /mnt/hugepages        hugetlbfs    rw,auto,uid=9500,user,sync    0 0

This line in /etc/fstab will give the user with id 9500 r/w permission on /mnt/hugepages.
This can be activated without reboot:
# mount -a
Its all setup now and JRockit can be started with -XlargePages:exitOnFailure=true , after using this
flag if the vm start successfully then again doing
# cat /proc/meminfo | grep Huge
will tell you how many pages are taken away by your JRockit. If the VM doesn't start that would mean you might have missed some steps.

Wednesday Jan 06, 2010

WebSphere Hypervisor Edition and VirtualBox

Finally I got some time to check on the WebSphere Application Server Hypervisor Edition which according to the IBM Web-Site is optimized to run on Cloudburst appliances. I don't have appliance h/w access and I needed to check on how its packaged. Here is what I found in my first walk-through, more details will be added if I found them.
The software is packaged and available in several forms:
1. Appliance Image which is OVA formatted virtual image and intended to be used with Cloudburst appliance
2. ESX images to be used with VMWare ESX Servers
3. OVF formatted virtual machine image, which can be used on any virtualization platform where OVF is supported

The OVF image download file is composed of several files which will appear as different disk in VirtualBox and they hold different software and profiled such as WebSphere App Server, IHS and so on...
So I picked up the OVF image and decided to run it on the VirtualBox on my OpenSolaris box. As VirtualBox supports the OVF format and it can be imported using just few mouse clicks.
When I did that this is what I found:
The OS form some reason, mostly due to absence of Novell SUSE in my VirtualBox install appears as Ubuntu. So I continued with that and booted it. Everything went fine except I ran into a display issues as my VirtualBox did not like the display drivers that is part of the image currently available. More on that in next blog for the moment I just wanted to check how the install/startup and service configuration goes so I continued with a command line console access. After going through the license acceptance and select the right profile everything came back up as expected and I had a running instance of WebSphere Application Server with two profiles running in my VirtualBox. One of the profiles is for the "nodeagent" while other is regular "Application Server Profile".
This is what you need to try this:

  1. Obtain/install a copy of VirtualBox
  2. Download the HyperVisor Edition image in OVF format
  3. Import the OVF file
This should get you going. Later I decided to change it to OpenSuse option in VirtualBox and did not find any difference in behaviour so I assume its OK with either option.

Thursday Dec 03, 2009

Using OpenSolaris as Amazon EC2 VPC Gateway

Sometime back Amazon has launched their Virtual Private Cloud(VPC) services which will bridge between existing company IT and AWS cloud. It is very nice feature which really enables hybrid cloud and it make sense where you data center need grows for sudden burst only. Cases like that it makes lot of sense to use VPC. As you can seamlessly continue to use OpenSolaris and related AMI's in your VPC there is some restriction on how and what can connect to these VPC. According to the documentation right now the requirement states that you will need either CISCO or Juniper routers to do the JOB and there is a Generic Device configuration which lets you use the IPSec based Encryption/Decryption so you it can tunnel your Virtual Private Network Connnection(VPN). You can easily get to the details here.
Using OpenSolaris as Customer Gateway we explored this option with some of the great colleagues from Sun(Glenn, Dan) and we made it easier to work. So an OpenSolaris box can do all that for you without a router. Glenn has put up kennai web page which provides the easy to use tool that will setup the connectivity and start your VPC within minutes. Or as Glenn suggests in his blog a VirtualBox OpenSolaris client running on any derskotp can do the job for test purpose and get you started. Use it and provide us the feedback.

Suggested additional reading:
OpenSolaris Crossbow
Amazon VPC

Tuesday Jun 09, 2009

Speed up your SSL operation for IBM HTTP Server on Ultra SPARC T2 systems

Sometime back we have published a Sun Blueprint (Accelerating IBM HTTP Server Cryptographic Operations Using Sun Servers with CoolThreads Technology) detailing the steps needed to get your IBM HTTP Server use the on-chip crypto processor on Ultra SPARC T2 based system for SSL operation. This will give a free SSL operation boost without buying additional hardware for such operations.
The documentation lists all the steps needed to get your IBM HTTP Server and GSKit working  with this on-chip crypto module on Ultra SPARC T2 processor.  In addition to how to configure it, it also has results from some of the performance testing that has been done to measure the performance gain. Your milegae may vary depending on your type of workload but if you are making lot of new client connection and serving "HTTPS" traffic then this would something that is available to you free you want to consider. It wil help you take care of your SSL handshakes operations.
Another important aspect of this is that GSKit is a common library that has been used by IBM in lot of products. And as its evident from the name, Global Security Kit, it is security related implementation to be used across different products such as PKCS#11 or so. Some more details can be found at my prior blog about GSKit. This implies that if you want to hookup with PKCS#11 provider and take advantage of on-chip cryptography for other products that can be done too. You must note that this integration has happened at certain level of IBM HTTP Server so it requires certain version of GSKit embedded with the product for which you will try to take advantage.

Saturday May 30, 2009

IMPACT 2009 notes

It's been really busy since I came back from IMPACT 2009. It was a great event. Since then I have busy working on few stuffs for JavaOne and CommunityOne.
The opening ceremony was great and Billy Crystal was really hilarious.
As usual every company does some launches during these event and IBM followed the tradition by putting up "WebSphere Cloud Brust". It was quite confusing to lot of people including me but when I stopped by the Cloud Brust booth I got fair bit of idea about what it is and how it can provide the value add although I expected the booth demo to be live but is wasn't the case.
Lot of session has talked by processor advancement, memory placement, bus speed and what not..During these there were numerous references to Sun Ultra SPARC T2 line of processor which leads the multi-core revolution. You might have got different perspective of different speaker on this but if you have used it you know the power, processing and power efficiency included, of these systems. If you haven't tried your hand then do and know it for yourself.

I expected this to be on linux/xen combo but surprisingly it was based on VMWare hypervisor that was very interesting. Similar deployment toplogy can be achieved using no additional technology overhead with Solaris such  zones/container or if you are ready to try your hands on OpenSolaris you can use xen. Remember OpenSolaris has tool to get your VMWare images working on OpenSolaris.
Here are few important pictures I took.

WebSphere Cloud Brust announcement slide:

Thursday Apr 30, 2009

IMPACT 2009 @ Las Vegas

Wanted to let you all know that Sun/IBM Team will be hosting the following BOF during this year IMPACT 2009.
Title: Optimizing your IBM WebSphere® Infrastructure with Solaris
Session ID: BOF-3174A
Session Time: Tue, May 5 12:00 PM - 01:00 PM
Location: Venetian Galileo 904
This will give you an unique opportunity to talk to engineers from both Sun and IBM team. We can cover whole range of topics concerning your WebSphere deployment on Sun platform.

For rest of the discussion I suggest you stop by at our booth and we can help you get your question answered.

Tuesday Apr 14, 2009

Scaling up WebSphere on Sun Blade X6270 Server

As always I got chance to try my hands on one of the new blade server that Sun has released based on Intel Xeon processor 5500 series namely, Sun Blade X6270 server module. The new Xeon processor has lot of features to deliver better performance based on the workload by which it is driven eg. automatically increasing processor frequency when needed and possible, as well as the capability to have extra hardware threads or taking advantage of the processor's power and thermal headroom. I would suggest you to visit for more details from domain expert including power consumption to virtualization and everything else supplemented by best OS on the planet "Solaris 10/OpenSolaris". I will leave rest of the discussion to my distinguished colleagues at Sun and will talk about what I observed when running WebSphere Application Server on this system.

As most current version of WebSphere Application Server available was V7.0 so I picked up this installed then made sure I am running with the latest fixpacks and ran it. Everything was breeze from installation, fix-pack upgrade to start/stop and then benchmarking. The Sun Blade X6270 I had was built on 2 sockets of the Intel Xeon L5520 processor and the Solaris 10 10/08 Operating System.

As always I used the same workload that I use for my WebSphere tests and there are other components which included the Database Server which was running DB2, and the test Drivers. For benchmark I used Faban which is very easy to use and run and it does the bookkeeping of benchmark so nicely that I don't have to keep records myself. I don't constrain my DB in terms of performance so I let it run in "/tmp" to just make everything simple so I don't spend too much time on tuning DB as I am working on WebSphere App Server.

The Intel Xeon processor 5500 series has two modes for running namely, Turbo mode and Hyper-Threading mode options in the BIOS setup. Which can be changed during boot by going to BIOS menu and doing the right selection. As Intel has put up "Nehalem, unleashes parallel processing performance enabled by an integrated memory controller and Intel® QuickPath Technology providing high-speed interconnects per independent processing core. The Intel® Turbo Boost Technology taking advantage of the processor's power and thermal headroom. This enables increased performance of both multi-threaded and single-threaded workloads and with Intel HT Technology, bringing high-performance applications into mainstream computing with 1-16+ threads optimized for new generation multi-core processor architecture." I did tried to run in both modes did not notice any significant difference for my tests while running in these modes.

Single instance of server was able to use much of the system capacity and stopped scaling. I was running all the tools and nicstat was one able to point me the moment when the n/w bandwidth was exhausted it stopped taking any more load. Then to work around the network capacity I put a direct connection with the DB Server which resulted in little better throughput and little more system utilization. On the network communication between DB and Server was happening over this interface. Still not enough and server had capacity neither the thread-dumps or system stats pointed out any problem other than network again being saturated. In which the public interface was full loaded and there was a very light load on the point to point connection interface. At this point I don't had any choice other than adding another instance of the server. To do this I created a Solaris Zone/Container and had one instance running into this zone. For this zone I used the point to point connection between the DB and the blade server which eliminated this and I was able to use the server to full capacity.

The network generated interrupts and there distribution can be checked by using the following command:
-bash-3.00# echo ::interrupts | mdb -k | grep igb
49 0x60 6 MSI-X 0 1 - igb_intr_tx_other
50 0x61 6 MSI-X 0 1 - igb_intr_rx
51 0x62 6 MSI-X 4 1 - igb_intr_tx_other
52 0x63 6 MSI-X 7 1 - igb_intr_rx

I looked for igb only becuase my network drivers are as:
igb0 - public (ie. going over GB switch)
igb1 - private (point-to-point)
From the above it is clear that the tx and rx interrupts are going over virtual cpu or more technically threads 0, 4 and 7 which is very well distributed.

So what did I do for tuning the WebSphere ?
My tunings remains same as I was not changing the Operating System and Software which includes Java. I will list them here for reference:

Disabled the Performance Monitoring Infrastructure
Disabled the Application Profiling
I used the DynaCache for the App just as to save some DB overhead
Set the WebContainer thread to the same number of threads as load driver
For Connection Pool I made sure that the number of DB connections are more than my WebContainer thread pool so none of the thread end up waiting for DB connections.
initialHeapSize="2048" maximumHeapSize="2048" -server -Xmn1024m -XX:+AggressiveOpts -XX:+UseParallelGC -XX:ParallelGCThreads=8 -XX:+UseParallelOldGC

Before I conclude I would like you to know that the only 64-bit of WebSphere is available on this platform so if your application happen to be memory hungry you are not limited by a 4GB address space and as you notice from David Dagastine blog that there are lot of things happening with 64-bit JVM and how it is getting optimized compared to 32-bit variant. But you may have to wait for the appropriate fixpacks to get to that. I will update as soon as it becomes available.
There are lot of other advantages of this server and I would encourage you to read more about this @

Monday Mar 23, 2009

Free cryptography for WebSphere on Ultra SPARC T2 and T2 plus systems

If you have been being hit by the processing overhead of SSL stuff and have to buy special purpose hardware to speed up things, then may be its time to leave all this processing to Ultra SPARC T2 and T2 Plus based system. Which has builtin cryptogarphic support on the chip and websphere supports this:

Here are the details from IBM WebSite:

For details of setup you can follow the WebSphere Infocenter.
Please note that this feature will be available only after certain fix pack release so it will not work unless you upgrade your JDK to required level as suggested in the doc.

If you want to learn more about the Cryptographic Acceleration offered by these processor I would encourage to read this Sun Blueprint titled "Using the Cryptographic Accelerators in the UltraSPARC® T1 and T2 Processors".

Ask the expert session for WebSphere on Solaris

We will be hosting an Ask The Experts session on March 24th. 
We invite you all to join where we can discuss and share various ideas around how to optimally use WebSphere software on Solaris platform.
For  details please visit the WebSphere User Group web-site(

Thursday Jan 15, 2009

OpenSolaris and WebSphere

As lot of you may have tried already installing/using WebSphere on OpenSolaris and may have ran into some issues during install so I thought I will cover some of those issues into this blog and will provide possible workaround.
The instruction below applies to WebSphere Application Server v6.1 and v7.0. I havn not  tested 6.0.X but I assume it will work fine. If not then let me know. 

OpenSolaris brings in lot of new feature details can be found at If you don't want to commit a hard disk for OpenSolaris experience you can use VirtualBox and then install OpenSolaris inside VirtualBox. Once you figure out which route you want to take and how you want to install OpenSolaris then you need to grab installer for WebSphere and try installing it over. You will have following few options available to you to install:
Silent installation:

You will have to edit the template response file. After you make all the necessary changes if you try to install it will fail and if you check the log it will complain that this is not one of supported OS. The reason for this can be checked by the output of the command "uname -a" and as you can see from output it is being reported as "5.11".  So this is what you will have to add in your response file:
-OPT disableOSPrereqChecking="true"
Now try again and installation should complete without any error and you will get message about the Operating System version. Just ignore this message and start using.
Installing using the Installer GUI:
When you try to install using it will come up but when it invokes the install command then it will fail complaining the installation cannot be started in GUI and suggest that you use "silent" installer. This error happens only with WAS v6.1 only WAS v7.0 is fine and you can install without any issues. If you want to get into details you can check the niflog  and you will understand that this is happening due to installshield issues. Googling for this will reveal that this is a known issue with lot of platform eg. AIX/Linux/Solaris and it has nothing to do with either the OS or WAS.
So we can use the java based installe by directly invoking the following command:
bash # java -jar setup.jar
This to succeed you must have java runtime in your path which is by default available in OpenSolaris. It will bring up the installer and as said earlier "Prerequisite Check" will fail and if you still continue everything will be installed right.

Just to make sure the "Install Verification Test" and you will observe all the test passing now you are good to go.
If you runinto any display issues it will be worth checking this blog.

Thursday Jan 08, 2009

WebSphere Application Server V7.0

By this time you might have got your hand on new WebSphere Application Server namely WebSphere Application Server v7.0. There are lot of good thing about this version as compared to its predecessors. A list of which can be found easily from 7.0 InfoCenter.
Some of the feature as Solaris user you should be aware are:

  • New JDK/JVM Support

  • DTrace made easy

  • Performance Enhancement

  • New J2EE Spec Level Compliance

All the detail can be found at
Some of the differences you can note from the standard Java Installation with the one that is part of WAS 7 s missing sample directory which contains lot of samples including DTrace scripts which are ready to be used. If you want to try them grab a JDK 6.X installer. Install it and copy the dir YOUR_JAVA_INSTALL_DIR/sample/dtrace and start using the script as it is. There is a README.txt file which included in the dtrace dir. All the sample scripts also has instruction about how to use them. More information about DTrace can be found here.

Monday Oct 13, 2008

WebSphere Application Server and T5440

Today, we are announcing the next generation server based on UltraSPARC T2 Plus processor and this server happens to be a monster in terms of performance. It can consolidate a lot of servers in a single box providing the similar combined throughput. For the server details and other blogs related to this server I suggest you to visit T5440 Blog index(
In this blog, I summarize the performance of IBM WebSphere Application Server (WAS)  on T5440. To test and benchmark the performance, I use the most recent release WAS v7, that is based on Java EE 5 spec and JDK 6, with numerous features and performance enhancements. As I said earlier that this new Sun server is a monster in terms of performance, the WAS software set up and configuration require some planning and appropriate allocation of the system resources.   When you do the psrinfo(1M) command on this server, it will report to you that the system has 256 "processors".  This means that the processing power of this system far exceeds the software scalability of a single instance of the Application Server's. Thus, you will need multiple instances of WAS to drive the system to its full utilization.  Solaris Containers provides the most efficient way to accomplish such configuration as Solaris Containers provides process space isolation among different WAS instance as well as allocating proper system resources.
For maximizing the utilization of the system, I configured the environment as follows.

  • I created 7 Solaris Containers, allocated 32 processor threads to each of the 6 Containers, and allocated the remaining processor threads for the other container and the global zone.  Then, I used the following WAS configuration:

  • initialHeapSize="2500" maximumHeapSize="2500"

  • -server -Xmn2048m -XX:+AggressiveOpts -XX:+UseParallelGC -XX:ParallelGCThreads=16 -XX:+UseParallelOldGC -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl -Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XML11Configuration

  • Disable the PMI feature.

  • Disable the System.out logging from the admin console if you want you can do the same by editing the config file. by editing the trace service like - startupTraceSpecification="\*=info:SystemOut=off"

  • DynaCache - The DynaCache was used for this benchmark so the default size of the cahce was set to 2000 which wasn't enough for this test which can be adjusted accordingly based on the application need as I set it to 3000.

  • For thread pool tuning the only thing that was relevant in this benchmark was tuning the WebContainer thread pool right so I did set it to 80 and which was more that enough and I think It can be scaled down a bit but havn't tried.

  • Database connection pool was set it to same value as WebContainer pool and set the min and max to the same value.

For database I did not want to run into the network/disk contention so I created the DB in /tmp and I had to use two databases for this purpose. This was just to eliminated the Database configuration headaches and just measure the App Server machine scalability. These database were connected point-to-point with App Server box and the App Server instances were using the point-to-point connection for the DB.
This resulted in getting to 1.8X scalability of its predecessor system(T5140) in terms of throughput(which is measured for this benchmark in terms of requests served per second or more commonly known as req/sec).
So In nutshell if you are looking for another system to consolidate your WebSphere deployment with 1.8X throughput capacity of earlier UltraSparc T2 plus system then T5440 may be just the right system for you.

Tuesday May 13, 2008

EC2 blog

Rajesh has put up the EC2 blog where you can check for breaking news about OpenSolaris on EC2. Here are some information which is available for you:
Meet the Sun OpenSolaris on EC2 Team
Information on GlassFish/LifeRay/MySQL AMI
Current beta program status.

Check and leave you comments there if you want to know anything about this.
There are other AMI's are in progress and should be made available as soon as it is ready.

Monday May 12, 2008

Problem bringing up domU

If you run into issues when following my previous blog then you may running into problem with sysevent daemon.
If that happens here is what you need to do:
once done installing all the package and rebooting with the Sun xVM server option in the boot menu from the dom0 issues the following command.

Cut and paste everything in between ---- this line to your terminal and hit return and then you should be able to proceed with domU creation.

The error is like as:

bash-3.2# xm create -c

Using config file "./".
bash-3.2# Error: Device 0 (vif) could not be connected. Hotplug scripts not working.



/usr/sbin/syseventadm list -R $BASEDIR -c EC_xendev > /dev/null 2>&1
if [ $? -ne 0 ]
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xendev \\
/usr/lib/xen/scripts/xpvd-event 'action=$subclass' \\
'domain=$domain' 'vdev=$vdev' 'device=$device' \\
'devclass=$devclass' 'febe=$fob'

/usr/sbin/syseventadm list -R $BASEDIR -c EC_xpvsys > /dev/null 2>&1
if [ $? -ne 0 ]
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xpvsys \\
/usr/lib/xen/scripts/xpvsys-event 'subclass=$subclass' \\

# restart daemon if the package is being added to the running system
if [ "$BASEDIR" = "/" -a $? -eq 0 ]
/usr/sbin/syseventadm restart


Wednesday May 07, 2008

How to create domU in OpenSolaris 2008.05

In this blog I will walk you through the steps of creating a domU within your OpenSolaris 2008.05 installation.
To accomplish this task you must have done your OpenSolaris 2008.05 install and I assume you are familiar with this if not please refer to OpenSolaris web-site.

Once you done installing this you need to use IPS(Image Packaging System) so that your system will have the packages related to xen. Make sure your pkg authority is set right and you are able to download the packages.
If this is set right i.e. if you haven't changed anything then it should look like this:

-bash-3.2# pkg authority
AUTHORITY URL (preferred)

Now you will need to add the following packages(you can cut and paste these commands to your terminal):

pkg install SUNWvirtinst
pkg install SUNWurlgrabber
pkg install SUNWlibvirt
pkg install SUNWxvmhvm
pkg install SUNWxvmdom
pkg install SUNWxvm
pkg install SUNWgccruntime
pkg install SUNWgnutls
pkg install SUNWlibsdl
pkg install FSWxwpft
pkg install FSWxwrtl

Once done doing this now you have all the necessary packages. Now you need to modify you GRUB BOOT Loader so that you have the option to load xVM when you system boots. This can be achieved by doing this:

Open the file "/rpool/boot/grub/menu.lst" in vi and add the following entry:
title Solaris xVM
bootfs rpool/ROOT/opensolaris
kernel$ /boot/$ISADIR/xen.gz
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive
Save your changes and exit out of the editor.
Now reboot your system and when you get to the grub loader select the "Solaris xVM" so that you will have the xVM related modules loaded.

Once the boot is complete you will need to start some of the services to enable you to use the xen management command. The services need to be enabled are as:

svcadm enable svc:/system/xvm/store:default
svcadm enable svc:/system/xvm/xend:default

svcadm enable svc:/system/xvm/console:default

Note: Enabling the console is really necessary otherwise when you run your xm commands you will get into error like following:
/usr/lib/xen/bin/xenconsole: Could not open tty `/dev/pts/7': No such file or directory

Once you have done this follow the instruction listed in MJR's blog.

Which will have your domU up and running.




« December 2016