Monday Oct 19, 2015

WLST Support for Dynamic Clustering

Credits - Developed by:

Rehana Tabassum
Byron Nevins


The Value of WebLogic Dynamic Clustering



Dynamic Clusters

Today we would like to talk about a cool new feature that we added to WebLogic.You will definitely want to know all about it if you are interested in working with Dynamic Clusters but without the hassle.

 

Sometimes you would like to have multiple managed servers in a cluster. All of these servers can have identical configurations – other than, say, the specific host machines, port numbers and so on.It’s time-consuming to do the dull and error-prone work of adding configuration manually for identical servers. In WebLogic 12.1.2 Dynamic Cluster capability was added to address these concerns. Dynamic Clusters allow you to specify any number of servers in a cluster with the minimum amount of individual configuration.It also added support to add servers (expand the cluster/scaleUp) and remove servers (shrink the cluster/scaleDown). This can be done in the Admin Console GUI. Servers in the Dynamic Cluster can be started and stopped just like regular servers.

Elasticity Service

A great new feature available in WebLogic 12.2.1 is the Elasticity Service.ADD LINK HERE The Elasticity Service leverages the Dynamic Cluster Technology by expanding and shrinking the Dynamic Clusters. And it does it completely automatically based on usage or many other criteria you can choose from.

WLST Support

What we are introducing is the ability to use WLST to easily expand and shrink dynamic clusters and to start and stop servers quickly.

 

A Use Case

You have a Corporate App for Taxes that results in a large increase in traffic between March 31 and April 30. The rest of the year it is active but only 20% as active.

Let’s say you have a Dynamic Cluster, taxCluster, that you’ve setup for 2 managed servers. Which works fine for 11 months of the year. On March 31 you want to scaleUp the cluster to 10 servers.

  • Startup WLST and connect (see References for details)
  • Run the scaleUp command to increase from 2 to 10 servers in the dynamic cluster:
  • scaleUp(‘taxCluster’, 8, true) See Examples

The result is that the cluster is permanently scaled upfrom 2 to 10 servers. The 8 new servers are all automatically started. On May 1 you restore the cluster to 2 servers

  • Startup WLST and connect (see References for details)
  • Run the scaleDown command to decrease from 10 to 2 servers in the dynamic cluster:
  • scaleDown(‘taxCluster’, 8, true) See Examples

New Functionality

New commands have been added to WLST that have made all of these actions easy to perform:

  • Start Servers in the Cluster
    • Start 1 or more or even all servers that are not running currently
    • Create new server(s) and start them
  • Stop Servers in the Cluster Gracefully
    • Shutdown a given number of servers
    • Shutdown a given number of servers and also permanently remove them -- i.e. shrink the cluster

WLST Commands

To do these actions two new commands were added to WLST.

note: arguments inside square brackets are optional. Defaults are noted

scaleUp (clusterName, numServers, [updateConfiguration=false], [block=true], [timeoutSeconds=600], [type='DynamicCluster'])

clusterName Name of the dynamic cluster
numServers Number of servers to start
updateConfiguration Boolean value specifying whether WLST should increase the maximum size of the cluster if there are not enough non-running servers already configured.
block Boolean value specifying whether WLST should block user interaction until the servers finish starting.
timeoutSeconds Time (in seconds) that WLST waits for the server(s) to start before canceling the operation
type If specified, the argument value must be DynamicCluster This is for future enhancements.

scaleDown(clusterName, numServers, [updateConfiguration=false], [block=true], [timeoutSeconds=300], [type='DynamicCluster'])

clusterName Name of the dynamic cluster
numServers Number of servers to shutdown
updateConfiguration Boolean value specifying whether WLST should both stop the give number of servers and decrease the maximum size of the cluster
block Boolean value specifying whether WLST should block user interaction until the server(s) complete shutdown
timeoutSeconds Time that WLST waits for the server(s) to shutdown before canceling the operation
type If specified, the argument value must be DynamicCluster. This is for future enhancements.

Examples

Below are example code snippets on how to perform scaling actions by invoking the scaleUp and scaleDown commands.
Since they are online commands you must first connect to the administration server (see References for instructions).

Start One Existing Server

wls:/...>scaleUp('mydyncluster', 1)
Remote ScaleUp started successfully after 25 seconds.
Waiting for 1 servers to reach the running state. The timeout is 600 seconds.
All servers are now running.

 

Even if all the servers under the dynamic cluster are already running, you can add one or more new server(s) to the cluster and automatically start them. Thus you can expand the size of the cluster dynamically if needed. To do so you need to specify the value of updateConfiguration as true in the scaleUp command.
Important: The cluster will be expanded only if needed. In the following example if there are 2 servers configured that are not running, they will be started and the configuration won't change. On the other hand if all servers in the cluster are already running then 2 servers will be added permanently to the cluster.

Scaleup and Start 2 New Servers

wls:/...>scaleUp('mydyncluster', 2, true)
Remote ScaleUp started successfully after 25 seconds.
Waiting for 2 servers to reach the running state. The timeout is 600 seconds.
All servers are now running.

 

The following example shows that the scaleUp command fails when there are not enough non-running servers allocated in the configuration and the updateConfiguration attribute is set to false. In this example the scaleUp command is asking to start 2 servers, but only 1 non-running server is available in the configuration.

Not Enough Servers Available

wls:/...>state('mydyncluster')
There are 2 server(s) in cluster: mydyncluster
States of the servers are
mydyncluster-1---RUNNING
mydyncluster-2---SHUTDOWN
wls:/... > scaleUp('mydyncluster', 2)
... (stack trace removed for clarity)
WLSTException: Error occurred while performing scaleUp : There are not enough shutdown servers available in the cluster (1) to scaleUp by 2 -- rerun the command with the updateConfiguration flag set to true.

 

The command will succeed when updateConfiguration is set to true. The config will be expanded by the number of servers needed. In this example 1 server is added.

Expand By 1 Server

wls:/...>state('mydyncluster')
There are 2 server(s) in cluster: mydyncluster
States of the servers are
mydyncluster-1---RUNNING
mydyncluster-2---SHUTDOWN
wls:/... > scaleUp('mydyncluster', 2, true)
Remote ScaleUp started successfully after 39 seconds.Waiting for 2 servers to reach the running state. The timeout is 300 seconds.
All servers are now running.
wls:/> state('mydyncluster')
There are 3 server(s) in cluster: mydyncluster
States of the servers are
mydyncluster-1---RUNNING
mydyncluster-2---RUNNING
mydyncluster-3---RUNNING

 

To gracefully shutdown n servers, you can invoke the scaleDown command. If updateConfiguration is set to false (the default), the configuration will not be changed. In that case a scaleDown command will simply shutdown servers. In this example 2 servers are shutdown gracefully.

Shutdown 2 Servers

wls:/...>state('mydyncluster')
There are 5 server(s) in cluster: mydyncluster
States of the servers are
mydyncluster-1---RUNNING
mydyncluster-3---RUNNING
mydyncluster-2---RUNNING
mydyncluster-5---RUNNING
mydyncluster-4---RUNNING
wls:/... > scaleDown('mydyncluster', 2)
Remote ScaleDown started successfully after 2 seconds.
The servers were stopped successfully.
wls:/... > state('mydyncluster')
There are 5 server(s) in cluster: mydyncluster
States of the servers are
mydyncluster-1---RUNNING
mydyncluster-3---RUNNING
mydyncluster-2---RUNNING
mydyncluster-5---SHUTDOWN
mydyncluster-4---SHUTDOWN

 

If you wanted to shrink the cluster in additrion to stopping 2 servers, you can make the same call except this time set updateConfiguration to true. This will shutdown the servers and remove them from the configuration. I.e. this will shrink the cluster. Notice that 2 servers are gracefully shutdown and removed from the dynamic cluster configuration.

Shutdown 2 Servers and Shrink Cluster

wls:/...>state('mydyncluster')
There are 5 server(s) in cluster: mydyncluster
States of the servers are
mydyncluster-1---RUNNING
mydyncluster-3---RUNNING
mydyncluster-2---RUNNING
mydyncluster-5---RUNNING
mydyncluster-4---RUNNING
wls:/... > scaleDown('mydyncluster', 2)
Remote ScaleDown started successfully after 2 seconds.
The servers were stopped successfully.
wls:/... > state('mydyncluster')
There are 5 server(s) in cluster: mydyncluster
States of the servers are
mydyncluster-1---RUNNING
mydyncluster-3---RUNNING
mydyncluster-2---RUNNING

 

References

Start an interactive WLST session

wlst.sh
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline>

Connnect to an administration server using WLST

wls:/offline> connect('weblogic','welcome1234','t3://localhost:7001')
Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server "admin" that belongs to domain "dynClusterScalingDomain".

Thursday Mar 15, 2012

GlassFish Clustering with DCOM on Windows

DCOM - Distributed COM, a Microsoft protocol for communicating with Windows machines.

Why use DCOM?

  • In GlassFish 3.1 SSH is used as the standard way to run commands on remote nodes for clustering.  It is very difficult for users to get SSH configured properly on Windows.  SSH does not come with Windows so we have to depend on third party tools.  And then the user is forced to install and configure these tools -- which can be tricky.
  • DCOM is available on all supported platforms.  It is built-in to Windows.
The idea is to use DCOM to communicate with remote Windows nodes.  This has the huge advantage that the user has to do minimal, if any, configuration
on the Windows nodes.

Implementation Highlights

Two open Source Libraries have been added to GlassFish:
  • Jcifs – a SAMBA implementation in Java
  • J-interop – A Java implementation for making DCOM calls to remote Windows computers.  

Note that any supported platform can use DCOM to work with Windows nodes -- not just Windows.
E.g. you can have a Linux DAS work with Windows remote instances.

All existing SSH commands now have a corresponding DCOM command – except for setup-ssh which isn’t needed for DCOM.  
validate-dcom is an all new command.

New DCOM Commands

  • create-node-dcom
  • delete-node-dcom
  • install-node-dcom
  • list-nodes-dcom
  • ping-node-dcom
  • uninstall-node-dcom
  • update-node-dcom
  • validate-dcom
  • setup-local-dcom (This is only available via Update Center for GlassFish 3.1.2)

These commands are in-place in the trunk (4.0).  And in the branch (3.1.2)

Windows Configuration Challenges

There are an infinite number of possible configurations of Windows if you look at it as a combination of main release, service-pack, special drivers, software, configuration etc.  Later versions of Windows err on the side of tightening security be default.  This means that the Windows host may need to have configuration changes made.
These configuration changes mostly need to be made by the user.  setup-local-dcom will assist you in making required changes to the Windows Registry.  See the reference blogs for details.

The validate-dcom Command

validate-dcom is a crucial command.  It should be run before any other commands.  If it does not run successfully then there is no point in running other commands.
The validate-dcom command must be used from a DAS machine to test a different Windows machine.  
If  validate-dcom runs successfully you can be confident that all the DCOM commands will work.  Conversely, the opposite is also true:  If validate-dcom fails, then no DCOM commands will work.

What validate-dcom does

  1. Verify that the remote host is not the local machine.
  2. Resolves the remote host name
  3. Checks that the remote DCOM port is being listened on (135, 139)
  4. Checks that the remote host’s File Sharing is enabled (port 445)
  5. It copies a file (a script) to the remote host to verify that SAMBA is working and authorization is correct
  6. It runs a script that it copied on-the-fly to the remote host.


Tips and Tricks

The bread and butter commands that use DCOM are existing commands like create-instance, start-instance etc.   All of the commands that have dcom in their name are for dealing with the actual nodes.


The way the software works is to call asadmin.bat on the remote machine and run a command.  This means that you can track these commands easily on the remote machine with the usual tools.  E.g. using AS_LOGFILE, looking at log files, etc.  It’s easy to attach a debugger to the remote asadmin process, “just in time”, if necessary.

How to debug the remote commands:
Edit the asadmin.bat file that is in the glassfish/bin folder.  Use glassfish/lib/nadmin.bat in GlassFish 4.0+
Add these options to the java call:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1234
  Now if you run, say start-instance on DAS, you can attach your debugger, at your leisure, to the remote machines port 1234.  It will be running start-local-instance and patiently waiting for you to attach.

Thursday Mar 01, 2012

GlassFish DCOM Configuration Utility

Version 3.1.2 of GlassFish adds support for communicating with Windows remote hosts over DCOM.  DCOM is a communications protocol from Microsoft.  All Windows versions that GlassFish supports come with DCOM support built-in.  

In these days of high-security on Windows,  using DCOM may well require a few configuration steps.  One of the more challenging steps is to edit the Windows Registry.

We are using DCOM to run GlassFish processes on the remote Windows machines (we call them nodes).  What we do is the same thing command-line users do -- run asadmin on the remote machine.  Asadmin.bat is a script and in order to remotely run scripts you must have permissions set in the Windows Registry.  In fact you need two Registry entries set to give permission to the Administrators group.  These two are:

  1. WMI - Windows Management Interface
  2. WBEM Scripting

The problem with these 2 registry keys is that in many versions of Windows they are owned by an account named TrustedInstaller.  And by default the Administrators group have no permissions at all.  (Yes , I said the owner of the key -- Registry keys have owners with access control lists exactly analogous to files).  In my humble opinion this is a Windows Bug! 

Setting-up the registry keys correctly is fairly involved.  There are significant complications for 32 bit versus 64 bit Windows.  Windows does automatic redirection of Registry Keys.  You're probably thinking -- "This is just too painful to bear!".  And you are right.  That's why I wrote a C++ Windows program that does all of the registry changes for you.  But that's not all!  I embedded this C++ program inside a jar file that also contains an asadmin extension command.  In order to do the editing of the registry keys you simply need to deploy the extension command and run it one time on each Windows machine.  And then only if you need to run it.

The name of the command is setup-local-dcom.  You can get it from Update Center under contributions.  See my other blog for instructions on exactly how to get it from Update Center.

Instructions for running setup-local-dcom

  • Backup the Registry just in case.
  • Run the command on the Windows machine that will be used as a remote node for GlassFish
  • You need to be logged in to an account that is a member of the Administrators group.
  • run the following to see the manual page:
    asadmin setup-local-dcom --help

  • Run the command to edit the registry keys, if needed.  Use the --verbose option to get a report of what it did.
  • Note that the command makes you type in yes and hit return to give you one last chance to change your mind.  If you find that annoying use the --force option and it will silently run.
  • If the command was successful then go to any other computer that can access it and run the validate-local-dcom command against the machine you just ran setup-local-dcom on.  If validate-local-dcom runs successfully then all clustering commands should work fine in your distributed environment.  If validate-local-dcom fails then see another blog that describes further configuration steps you may need to make on the Windows node.

Example Output of setup-local-dcom --verbose

d:\>asadmin setup-local-dcom -v
Caution: This command might modify the permissions of some keys in the Windows registry.
Before running this command, back up the Windows registry.
The modification allows the Windows user full control over these keys.

Are you sure that you want to edit the Windows registry? If so, type yes in full:  yes
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
>>>>>>>>>>>>>   Set to Verbose Mode    <<<<<<<<<<<<<
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
Administrators group SID: [S-1-5-32-544]
Key: [CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]  Owner: [S-1-5-32-544]
Key: [CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]  Owner: [S-1-5-32-544]
Redirected Key: [Wow6432Node\CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]
Redirected Key: [Wow6432Node\CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]
No need to adjust the Scripting Registry Key.
No need to adjust the WMI Registry Key.
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
>>>>>>>>>>>>>   Set to Verbose Mode    <<<<<<<<<<<<<
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
Administrators group SID: [S-1-5-32-544]
Key: [CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]  Owner: [S-1-5-32-544]
Key: [CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]  Owner: [S-1-5-32-544]
Redirected Key: [Wow6432Node\CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]
Redirected Key: [Wow6432Node\CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]
No need to adjust the Scripting Registry Key.
No need to adjust the WMI Registry Key.

Command setup-local-dcom executed successfully.


Friday Jul 16, 2010

GlassFish V3.1 Offline Configuration

Clustering support is available now in pre-release GlassFish 3.1  In this blog I will show you how to perform Offline Configuration in GlassFish 3.1

I've blogged about  why and how to do Offline Configuration here.

One big difference between GlassFish V2 and GlassFish 3.1 clustering is that node agents are not provided and used in GlassFish 3.1.  Instead ssh is utilized.  Using ssh in GlassFish 3.1 is the subject of a different future blog.  In this blog I'll demonstrate how to setup a distributed GlassFish 3.1 system without node-agents or new ssh features.

The Desired Environment

  • DAS running on improvident
  • One cluster
  • The cluster has one instance
  • DAS and the instance run on different machines

Notes

  • The names are just the names of the machines I happened to use.
  • This is as simple a distributed system as possible for demonstration purposes. 
  1. machine1 = samskritam
  2. machine2 = improvident

  • install the same  GlassFish 3.1installation files on each machine. 
  • DAS (domain1) is used on samskritam. 
  • The other machine never runs DAS.
  • improvident has this set in its environment -- This makes it easy to run asadmin commands.
    • AS_ADMIN_HOST=samskritam
  • In the Procedure section I'm using these conventions:
    1. I put the name of the machine where the command is running at the beginning of the line - you wouldn't type that in.
    2. Comments are in italics
    3. Output from asadmin is bold

Procedure

  1. [samskritam] asadmin start-domain
  2. [samskritam] asadmin create-cluster mycluster
  3. [samskritam] asadmin create-node-config --nodehost improvident imp
  4. [samskritam] asadmin create-instance --cluster mycluster    --node imp mycluster_i1
  5. [improvident] asadmin create-local-instance --node imp mycluster_i1
  6. [improvident] asadmin start-local-instance mycluster_i1

Output

On Samskritam:

samskritam% asadmin start-domain
Waiting for the server to start ....
Successfully started the domain : domain1
domain location: /home/kedar/glassfishv3/glassfish/domains/domain1
Log File: /home/kedar/glassfishv3/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
samskritam% asadmin create-cluster mycluster
samskritam% asadmin create-node-config --nodehost improvident imp
samskritam% asadmin create-instance --cluster mycluster    --node imp mycluster_i1
remote failure:
Successfully created instance mycluster_i1 in the DAS configuration, but failed to create the instance files. Please run:

asadmin --host samskritam.SFBay.Sun.COM --port 4848 create-local-instance --node imp mycluster_i1

on improvident to complete the transaction.

____________________________________________________

On improvident:

~>asadmin create-local-instance --node imp mycluster_i1
Attempting to rendezvous with DAS on host samskritam port 4848
Uptime: 1 minutes, 31 seconds

The instance has rendezvoused with the DAS and will be using host samskritam port 4848 for future communication.
servers.server.mycluster_i1.config-ref=mycluster-config
servers.server.mycluster_i1.lb-weight=100
servers.server.mycluster_i1.name=mycluster_i1
servers.server.mycluster_i1.node=imp
servers.server.mycluster_i1.node-agent-ref=improvident

servers.server.mycluster_i1.property.rendezvousOccurred=true

~>asadmin start-local-instance mycluster_i1
Waiting for the server to start ......................................
Successfully started the instance: mycluster_i1
instancelocation: /export/home/bnlocal/glassfishv3/glassfish/nodeagents/improvident/mycluster_i1
Log File: /export/home/bnlocal/glassfishv3/glassfish/nodeagents/improvident/mycluster_i1/logs/server.log
Admin Port: 24848
~>asadmin list-clusters
mycluster running

~>asadmin list-instances --verbose
Instance Name   Host Name     Admin Port   Cluster     Current State
---------------|-------------|------------|-----------|--------------------------------|
mycluster_i1    improvident      24848      mycluster  Uptime: 15 minutes, 11 seconds




About

Byronnevins-Oracle

Search

Categories
Archives
« May 2016
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    
       
Today