Thursday Mar 15, 2012

GlassFish Clustering with DCOM on Windows

DCOM - Distributed COM, a Microsoft protocol for communicating with Windows machines.

Why use DCOM?

  • In GlassFish 3.1 SSH is used as the standard way to run commands on remote nodes for clustering.  It is very difficult for users to get SSH configured properly on Windows.  SSH does not come with Windows so we have to depend on third party tools.  And then the user is forced to install and configure these tools -- which can be tricky.
  • DCOM is available on all supported platforms.  It is built-in to Windows.
The idea is to use DCOM to communicate with remote Windows nodes.  This has the huge advantage that the user has to do minimal, if any, configuration
on the Windows nodes.

Implementation Highlights

Two open Source Libraries have been added to GlassFish:
  • Jcifs – a SAMBA implementation in Java
  • J-interop – A Java implementation for making DCOM calls to remote Windows computers.  

Note that any supported platform can use DCOM to work with Windows nodes -- not just Windows.
E.g. you can have a Linux DAS work with Windows remote instances.

All existing SSH commands now have a corresponding DCOM command – except for setup-ssh which isn’t needed for DCOM.  
validate-dcom is an all new command.

New DCOM Commands

  • create-node-dcom
  • delete-node-dcom
  • install-node-dcom
  • list-nodes-dcom
  • ping-node-dcom
  • uninstall-node-dcom
  • update-node-dcom
  • validate-dcom
  • setup-local-dcom (This is only available via Update Center for GlassFish 3.1.2)

These commands are in-place in the trunk (4.0).  And in the branch (3.1.2)

Windows Configuration Challenges

There are an infinite number of possible configurations of Windows if you look at it as a combination of main release, service-pack, special drivers, software, configuration etc.  Later versions of Windows err on the side of tightening security be default.  This means that the Windows host may need to have configuration changes made.
These configuration changes mostly need to be made by the user.  setup-local-dcom will assist you in making required changes to the Windows Registry.  See the reference blogs for details.

The validate-dcom Command

validate-dcom is a crucial command.  It should be run before any other commands.  If it does not run successfully then there is no point in running other commands.
The validate-dcom command must be used from a DAS machine to test a different Windows machine.  
If  validate-dcom runs successfully you can be confident that all the DCOM commands will work.  Conversely, the opposite is also true:  If validate-dcom fails, then no DCOM commands will work.

What validate-dcom does

  1. Verify that the remote host is not the local machine.
  2. Resolves the remote host name
  3. Checks that the remote DCOM port is being listened on (135, 139)
  4. Checks that the remote host’s File Sharing is enabled (port 445)
  5. It copies a file (a script) to the remote host to verify that SAMBA is working and authorization is correct
  6. It runs a script that it copied on-the-fly to the remote host.

Tips and Tricks

The bread and butter commands that use DCOM are existing commands like create-instance, start-instance etc.   All of the commands that have dcom in their name are for dealing with the actual nodes.

The way the software works is to call asadmin.bat on the remote machine and run a command.  This means that you can track these commands easily on the remote machine with the usual tools.  E.g. using AS_LOGFILE, looking at log files, etc.  It’s easy to attach a debugger to the remote asadmin process, “just in time”, if necessary.

How to debug the remote commands:
Edit the asadmin.bat file that is in the glassfish/bin folder.  Use glassfish/lib/nadmin.bat in GlassFish 4.0+
Add these options to the java call:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1234
  Now if you run, say start-instance on DAS, you can attach your debugger, at your leisure, to the remote machines port 1234.  It will be running start-local-instance and patiently waiting for you to attach.

Thursday Mar 01, 2012

GlassFish DCOM Configuration Utility

Version 3.1.2 of GlassFish adds support for communicating with Windows remote hosts over DCOM.  DCOM is a communications protocol from Microsoft.  All Windows versions that GlassFish supports come with DCOM support built-in.  

In these days of high-security on Windows,  using DCOM may well require a few configuration steps.  One of the more challenging steps is to edit the Windows Registry.

We are using DCOM to run GlassFish processes on the remote Windows machines (we call them nodes).  What we do is the same thing command-line users do -- run asadmin on the remote machine.  Asadmin.bat is a script and in order to remotely run scripts you must have permissions set in the Windows Registry.  In fact you need two Registry entries set to give permission to the Administrators group.  These two are:

  1. WMI - Windows Management Interface
  2. WBEM Scripting

The problem with these 2 registry keys is that in many versions of Windows they are owned by an account named TrustedInstaller.  And by default the Administrators group have no permissions at all.  (Yes , I said the owner of the key -- Registry keys have owners with access control lists exactly analogous to files).  In my humble opinion this is a Windows Bug! 

Setting-up the registry keys correctly is fairly involved.  There are significant complications for 32 bit versus 64 bit Windows.  Windows does automatic redirection of Registry Keys.  You're probably thinking -- "This is just too painful to bear!".  And you are right.  That's why I wrote a C++ Windows program that does all of the registry changes for you.  But that's not all!  I embedded this C++ program inside a jar file that also contains an asadmin extension command.  In order to do the editing of the registry keys you simply need to deploy the extension command and run it one time on each Windows machine.  And then only if you need to run it.

The name of the command is setup-local-dcom.  You can get it from Update Center under contributions.  See my other blog for instructions on exactly how to get it from Update Center.

Instructions for running setup-local-dcom

  • Backup the Registry just in case.
  • Run the command on the Windows machine that will be used as a remote node for GlassFish
  • You need to be logged in to an account that is a member of the Administrators group.
  • run the following to see the manual page:
    asadmin setup-local-dcom --help

  • Run the command to edit the registry keys, if needed.  Use the --verbose option to get a report of what it did.
  • Note that the command makes you type in yes and hit return to give you one last chance to change your mind.  If you find that annoying use the --force option and it will silently run.
  • If the command was successful then go to any other computer that can access it and run the validate-local-dcom command against the machine you just ran setup-local-dcom on.  If validate-local-dcom runs successfully then all clustering commands should work fine in your distributed environment.  If validate-local-dcom fails then see another blog that describes further configuration steps you may need to make on the Windows node.

Example Output of setup-local-dcom --verbose

d:\>asadmin setup-local-dcom -v
Caution: This command might modify the permissions of some keys in the Windows registry.
Before running this command, back up the Windows registry.
The modification allows the Windows user full control over these keys.

Are you sure that you want to edit the Windows registry? If so, type yes in full:  yes
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
>>>>>>>>>>>>>   Set to Verbose Mode    <<<<<<<<<<<<<
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
Administrators group SID: [S-1-5-32-544]
Key: [CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]  Owner: [S-1-5-32-544]
Key: [CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]  Owner: [S-1-5-32-544]
Redirected Key: [Wow6432Node\CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]
Redirected Key: [Wow6432Node\CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]
No need to adjust the Scripting Registry Key.
No need to adjust the WMI Registry Key.
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
>>>>>>>>>>>>>   Set to Verbose Mode    <<<<<<<<<<<<<
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
Administrators group SID: [S-1-5-32-544]
Key: [CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]  Owner: [S-1-5-32-544]
Key: [CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]  Owner: [S-1-5-32-544]
Redirected Key: [Wow6432Node\CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]
Redirected Key: [Wow6432Node\CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]
No need to adjust the Scripting Registry Key.
No need to adjust the WMI Registry Key.

Command setup-local-dcom executed successfully.

How to Setup Update Center in GlassFish 3.1.2 in Windows

1. start glassfish3\bin\updatetool.bat

Get a cup of coffee -- step 1 will take a while the first time it is called as it bootstraps itself.!

2. Run the newly created updatetool.exe application

The GUI appears after a lengthy wait.  Click on Available Add-ons on the left side of the window and go get another cup of  coffee.  It's going to take a long time.

3. How to get the  DCOM Configuration Command

If you came here from my other blog about setting up DCOM for GlassFish you will need to get the binaries from Update Center.

  1. Click on available updates
  2. Choose the one named setup-local-dcom asadmin subcommand
  3. Accept the license
  4. Update Center will download and install it for you.
  5. It is ready to use now!

Sunday Feb 27, 2011

Automatic Starting Implementation Details for GlassFish 3.1

Process Nuts and Bolts

When you start a GlassFish 3.1 Server Service notice that there are 2 JVM processes running.  The 2 process are these:

  1. The GlassFish Server
  2. A JVM running asadmin start-[local-instance|domain] --verbose

Why?  We want to leverage the start command.  Asadmin does a great deal of work for you when you start a server.  The end-product of that work is mainly (but not only) the generation of a big complex JVM invocation command to start the server with.  I could have implemented the Services to call asadmin and have asadmin exit.  But then the Platform would see that asadmin exited and would report the service as "stopped".  It would not know in the future when the server really stops. 

When you start a server with the --verbose flag then the asadmin process waits for the server to die before exiting.  When the server stops -- asadmin stops and the Platform will update the state to "stopped".

Note that restart-[domain|instance] will still work perfectly.  In that case the server process dies but the asadmin process does not - not even for a moment.  The platform will not "see" anything happening.


I discovered that there seems to be a special way to create services for every different flavor of Linux.  And they are all different from each other.  Basically all of these different ways end up with writing out special files into a special area.  This one common denominator is to use the ancient UNIX way of setting up services which is, not surprisingly, editing lots of configuration files in a special area.  Even the exact details on the special area is different among the different flavors.  The code is flexible and will handle all known Linux flavors. The script can be found in /etc/init.d and links to it in the runlevel directories:  /etc/init.d/rc?.d

Automatic Starting of Servers in GlassFish 3.1

Note: At the end of this blog are links to older blogs from V3.0 about this feature.  You may want to look them over as well.

The way we offer the ability to automatically start GlassFish servers is via "services" on the platform.  "Services" are an ancient technology, available on all of our supported platforms (and I would imagine is available on every serious OS).  Services allow applications to run automatically when a computer boots up.  Nobody needs to login -- the service just starts automatically.  A simple example of this is the FTP server.  When you boot up your computer you can access FTP from any other computer without touching the FTP computer.  This is very important in the event of a power failure.  On Windows you can setup automatic security updates and Windows will automatically reboot your computer.  If you have a Subversion server that is not configured as a service and you are halfway around the world depending on it -- you will become a Services Specialist in the near future when you return home!

GlassFish 3.1 Platform Services now supports all Linux versions, Windows and Solaris10/SMF

Services can be quite complex.  For instance you might want to have it do this:

  1. Automatically start a GlassFish server upon booting
  2. If the server crashes (no way!) restart it
    What if it crashes every time?  The machine will be in trouble with an infinite loop so:
  3. Try to restart 3 times and then give up.

Setting up this level of services-granularity is very difficult and expensive to implement perfectly.  For one - we are supporting several platforms and they all do these things completely differently. The hard part is setting up services at all to do the basic things.  Once that is complete you can easily use the platform's native tools for adjusting things just the way you like.  What we support is the classic services model -- which is to start the service upon booting, unattended.

I worked hard to make it easy to setup services.  Here's how you would create them for an installation that has one Domain:

asadmin create-service

If you want to make sure it would work but you aren't yet ready to pull the trigger, or you just want to see what it would do -- then run this command:

asadmin create-service --dry-run

If your domain is named, say. domain1 then the service's name will be domain1.  Simple.

 After the service is created you are greeted with a platform-specific message giving you the details on how to start the service.  We do NOT automatically start it.  I personally recommend rebooting immediately -- and it will start.

Windows is particularly easy and flexible to work with.  You can manage the GlassFish service(s) by:

  1. GUI -- right-click on Computer/Manage/Services
  2. sc.exe  very handy tool for querying/starting/stopping/restarting etc.
  3. In the bin directory of the domain or instance run the your-service-nameService.exe application that we put in there. 

How to setup automatic starting for an instance?

Easy!  Simply give the name of the instance as the final argument to create-service.  The following command will create a service for myinstance:

asadmin create-service myinstance

There is an undocumented, unsupported command that will delete a service for you.  It is very very easy to delete a service on SMF and Windows.  On Linux it can be tedious.  I've personally tested the command successfully but it has not been through the merciless testing of QA at Oracle.  Use it at your own risk.

asadmin _delete-service 

Important blog for making sure GlassFish doesn't stop when you logout.

I have another blog with implementation details if you are interested.

Some older blogs that pertain to V3.0:

V3.0 Platform Services for Windows and SMF

DIY Cookbook for Configuring Linux Services

How to Run on Linux as a Non-Root User

Saturday Feb 13, 2010

How to make GlassFish 3.1 Platform Services Survive Logoff

Refer to my other blog for general details about setting up Platform Services

A running JVM does not survive the user logging off unless you wave a magic wand over the JVM startup parameters.  It is very simple.  If you give the -Xrs option then the JVM will completely ignore logoff events (as well as others).  The solution in V3 is very simple but first let me give you a little background.  Skip to the end of  this blog to see the solution if you are in a hurry.

 In V2 all you needed to do was add -Xrs as a jvm-option to domain.xml and you were all set.  If you tried this on V3 you'll have noticed that it does not work.  The reason is that V3 running as a service is really two separate JVM's.  There is an asadmin "watchdog" JVM and the V3 JVM.  This arrangement is exactly like running asadmin start-domain --verbose at a command line.  These 2 JVMs go through life together.  If one dies then the other is automatically killed.  That's the problem.  You can set the -Xrs flag in domain.xml but if the asadmin JVM does not also have it set, then logging off will kill the asadmin JVM which, in turn, will automatically take the V3 JVM down with it. 

The solution is simple: set -Xrs on both JVMs.  I verified that V3 running as a service will only survive a logoff if both JVM's have it set.  I tested all 4 combinations:

 domain.xml   -Xrs ?

 asadmin   -Xrs?

 Survives Logoff event?













My recommendation is to set -Xrs for asadmin once and then forget about it.  You can always adjust the behavior exclusively from domain.xml.


  1. asadmin create-jvm-options -Xrs

  2. edit install-root/bin/asadmin[.bat] and add the -Xrs option to the java command

To turn off the behavior run this command:

asadmin delete-jvm-options -Xrs

Sunday May 31, 2009

Platform Services Available in V3

Platform Services have just been rolled out for the first 2 platforms:

  • Solaris  Management Facility (SMF)
  • Windows

In the simplest case where you have one default domain and you want to run it as a service you run this command:

asadmin create-service

Here are all of the options:

[--passwordfile password-file] [--name service-name] [--serviceproperties (name=value)+] [--dry-run=false] [--domaindir domain_parent_directory] [domain_name]

 Once you have created the service it can be controlled in various ways by operating system and native tools. 


We use Windows Service Wrapper as the tool for installing the service.  It can be used to start stop and uninstall the service.  It can be found in the bin subdirectory of the domain directory.  In the default case the name of the executable will be domain1Service.exe  The directory also contains the configuration file, domain1Service.xml.  Below are the available commands:

  • domain1Service start
  • domain1Service stop
  • domain1Service uninstall
  • domain1Service install
You can also manage the GlassFish service with Windows tools:  Services snap-in and the net command line program.

Solaris  Management Facility (SMF)

You create the service exactly the same as WIndows.  One big difference for most users is that on Windows most users have administration privileges.  I.e. everyone is super-user.  Not so on Solaris.  In order to create a service you must have SMF privileges.  How to set that up is beyond the scope of this blog, but root definitely has these privileges by default 

 Important: If root creates a service then only root can start that domain from then on.  The best way to set this up is to install and/or create the domain as root and then create the service as root.

Once you have created the service you need to start it yourself.  Here are the most typical Solaris commands of interest:

  • /usr/bin/svcs  -a | grep domain1  // status
  • /usr/sbin/svcadm enable domain1 // start
  • /usr/sbin/svcadm disable domain1 // stop
  • /usr/sbin/svccfg delete domain1 // uninstall

Future Directions

If there is interest, demand and time we will add support for controlling the service from CLI:

  • start
  • stop
  • uninstall
  • restart

Friday Oct 05, 2007

How to start JavaDB/Derby as a Windows Service

As I've recently started using the built-in Java DB support in GlassFish, I want the Java DB Network Server to start automatically along with GlassFish.

 I discovered that I can use the exact same method and program that's used for GlassFish-as-a-service.

On Windows, I used this script to setup a Windows Service that automatically starts the Java DB Network Server.  You can change the paths, run it once and the DB will start automatically as a service.


@echo off
if "%1A"=="A" goto usage
if "%2A"=="A" goto usage

echo on
sc create %1 binPath= "C:\\as\\lib\\appservService.exe \\"C:\\as\\bin\\asadmin.bat start-database --dbhome %2\\" \\"C:\\as\\bin\\asadmin.bat stop-database\\""  start= auto DisplayName= %1
goto end

echo usage createDBservice  name dbhome






« April 2014