Monday Feb 28, 2011

GlassFish 3.1 and Manual Synchronization

If you need to setup a GlassFish cluster on machines which are unable to talk to each other, you can do so with the new manual synchronization commands introduced in GlassFish 3.1: export-sync-bundle and import-sync-bundle

Machine A = DAS
Machine B = remote machine

You have an existing cluster cluster1 with instance1 which you want to synchronize on to machine B.
instance1 has already been registered in DAS domain.xml on machine A using create-local-instance.

1. Export cluster1 on machine A
asadmin export-sync-bundle cluster1

This will create the sync bundle which contains the cluster configuration:

2. Copy to machine B.

If machine B can talk to machine A, you can use the --retrieve option.
On machine B do

asadmin --host machineA export-sync-bundle --target cluster1 --retrieve=true

This will download the sync bundle from machine A onto machine B.

3. On machine B import the sync bundle

asadmin import-sync-bundle --instance instance1

The instance1 filesystem is created using the sync bundle under the node that is provided in the domain.xml. You can specify a different node by using the --node option.

instance1 must already have been registered on DAS domain.xml on machine A, using create-local-instance or create-instance. If it is not registered, import-sync-bundle will give an error.

GlassFish 3.1 and create-local-instance

GlassFish Server Open Source Edition 3.1 is just released with clustering capability! If you have SSH setup on your machines, you can create the clusters and instances just from the DAS (Domain Administration Server) machine using the asadmin commands: create-cluster and create-instance. However, if you don't have SSH setup, you can still create a cluster. You will need to go to each machine and run the create-local-instance command locally.

The create-local-instance command is actually a hybrid command, meaning it's a local command which creates the local filesystem structure, then calls a remote hidden command (_register-instance) to register the instance with the DAS. Registering the instance with the DAS entails creating a server element in the DAS domain.xml.

For example if you want to setup a clustered instance on machine B with the DAS on machine A,

0. Make sure you can ping machine A from machine B. Note the host name or IP address you use for this.

1. Install GlassFish on each machine, machine A and machine B.

Machine A = DAS
Machine B = remote instance

2. Start DAS on Machine A
glassfish3/glassfish/bin/asadmin start-domain

3. On DAS (machine A) create cluster1
glassfish3/glassfish/bin/asadmin create-cluster cluster1

4. On machine B create instance1 (The default das port is 4848. Specify --port to use a different port.)
glassfish3/glassfish/bin/asadmin --host machineA create-local-instance --cluster cluster1 instance1

A few things will happen...

1. This will create a node, machineB.

a) On machine B, a directory for machineB is created under
glassfish3/glassfish/nodes/machineB (if the instance is on the same machine as DAS this will be localhost-domain1).

b) A node element for machineB is created in DAS domain.xml.

<node node-host="machineB" name="machineB" type="CONFIG" install-dir="${com.sun.aas.productRoot}"></node>

2. The DAS connection properties for node machineB is created

Under directory machineB is agent and instance1. Under agent is the config/ The holds the DAS host and port information used by the node, machineB, to connect to the DAS.


The is created the first time an instance is created for a particular node. The node machineB will now always connect to the DAS using this Any instance which uses this node machineB, will talk to the same DAS using the same host and port. If you want to connect to a different DAS, you can either edit manually (all instances under that node would be affected), or create a new node using create-node-config and calling create-local-instance with a different DAS host/port.

3. A directory for instance is created under node machineB.

The directory for instance 1 is created at glassfish3/glassfish/nodes/machineB/instance1. The instance1/config will contain the keystore information. Once you start the instance using start-local-instance, this instance1 directory will get synchronized with the DAS.

4. The instance1 is registered on DAS.

In DAS domain.xml:

<server name="instance1" node-ref="machineB" config-ref="cluster1-config">

5. The instance1 is also registered to all the other instance within the cluster.

The instance1 is also registered to all the other instances in the cluster if they are up and running. All the instances in the cluster will know about each other.

Now you can start the newly created instance1 on machineB
asadmin start-local-instance instance1

Performance Tuner in Oracle GlassFish Server 3.1

The Performance Tuner is now again available in Oracle GlassFish Server 3.1 (docs). This handy tool was first introduced in GlassFish v2.1 as one of the value-add features. You'll enjoy the same performance gains as before. Quoting from previous Ken's previous blog, "The Tuner typically improves performance by 300% or more, and on Oracle's T-Series hardware the performance gain can be greater than 600%!" Wow! The code was ported over from v2.1 with some refactoring to get it working for 3.1.

To really get the best performance out of GlassFish Server 3.1, I recommend getting the Performance Tuner. It is very easy to use. The first page will ask you 7 questions about your particular system and resource usage. There is both inline help and online help to guide you. If you need more detailed information about each of the questions, click the Help button in the upper right of the Admin Console to open the online help.

To starting using the Performance Tuner, first, you decide which configuration you want to tune. This could be the DAS ("server"), a cluster, or a stand-alone instance. For example, let's say I have a cluster with several instances, each running on the new SPARC T3-4 server which I would like to tune. First I make sure the cluster is running. Then I gather some information about my system and the application I'm running.

The SPARC T3-4 server has 64 cores - that's 4 SPARC T3 processors with 16 cores each, and up to 512 GB of memory. I'll allocate something less than 64 cores for the GlassFish server instances - let's say 32 cores, so I enter 32 for the Number of Threads (Logical CPUs).

And I'll specify the heap memory for the GlassFish Server's JVM server instance, to be something less than 512 GB. Let's say 12 GB.

Let's also say the application deployed on the cluster spends a lot of time in the database tier, so I'll select 'HIGH {>=20%) for Allocation Time for Database Access.

I'll keep the default Garbage Collection Strategy of Maximum Peak Throughput (Higher GC Pause.)

The next two questions only apply if the Security Manager or Access Logger is enabled. Since I already have Security Manager and Access Logger disabled, those checkboxes are greyed out.

Since I'm running on a SPARC T-Series Server, I can check the last box.

When I click on the Next button, the second page will display the proposed configuration changes. The table will show what would be added, removed, or changed. If you do not want the Performance Tuner to apply all the changes for you, you can selectively make manual updates using the manual instructions that is provided for each proposed configuration change. If you are satisfied with the proposed changes and want the Performance Tuner to apply the changes click on the Confirm button. For the new changes to take effect the cluster needs to be restarted. A backup of the domain.xml is made in domain1/config/tuner should you need to revert the changes. Just copy the domain_1.xml to domain1/config to go back to the original configuration.

Here's what the first page looks like:

Here's what the second page looks like:

This second page is a table of all the items to be added, removed, or changed. From here you can see the first two items to be changed: Maximum Connections attribute of the http element. If we confirm to apply the changes, the value of Maximum Connections attribute would change from 256 to 10000.

Also, you will see on the right-hand column, the manual steps if you want to do this yourself using either the Admin Console or the asadmin set command.

Once we confirm to apply the changes, we can see that the DAS domain.xml has been updated. The max-connections for http-listener-1 and http-listener-2 are "10000".

<config name="cluster1-config"> ............................... <network-config> <protocols> <protocol name="http-listener-1"> <http max-connections="10000" default-virtual-server="server"> <file-cache></file-cache> </http> </protocol> <protocol security-enabled="true" name="http-listener-2"> <http max-connections="10000" default-virtual-server="server"> <file-cache></file-cache> </http> <ssl ssl3-enabled="false" classname="" cert-nickname="s1as"></ssl> </protocol> <protocol name="admin-listener"> <http max-connections="250" default-virtual-server="__asadmin"> <file-cache></file-cache> </http> </protocol> </protocols> .................................. </network-config>

Restart the cluster for changes to take effect.

Another tip is that if one of the instance's configuration in the cluster needs to be tweaked for JVM Heap size, GC threads, or max-thread-pool-size, you can override the value that was set by the Performance Tuner easily. The Performance Tuner creates a system property in both the server and the cluster config elements. The system properties in the server will override the system properties in the cluster config. If we need to tweak the JVM Heap size for instance1, we can do it here

<server name="instance1" node-ref="localhost-domain1" config-ref="cluster1-config"> <system-property name="JVM_HEAP_SIZE" value="1200m"></system-property> <system-property name="JVM_HEAP_SIZE_XMN" value="600m"></system-property> <system-property name="JVM_PARALLEL_GC_THREADS" value="24"></system-property> <system-property name="MAX_THREAD_POOL_SIZE" value="128"></system-property> </server>

You can watch a brief video of the Performance Tuner - Performance Tuner Demo.


Jennifer Chou


« February 2011