Thursday Oct 31, 2013

Clustering in GlassFish with DCOM on Windows 7

I've discovered that Windows 7 makes it very difficult to use DCOM and, mainly, the GlassFish clustering commands that rely on DCOM.  I spent a few days trying to solve the problems.  I don't yet have a cookbook for making DCOM work on Windows 7.  But here are a few tips and advice I've found.

  1. run asadmin setup-local-dcom -- It now comes automatically with the open source GlassFish 4.  It will write some critical registry entries for you.
  2.  run asadmin validate-dcom to test dcom

3.   When I ran validate-dcom on my Windows 7 network I saw the problem below:

Successfully resolved host name to: gloin/ Successfully connected to DCOM Port at port 135 on host gloin.

Successfully connected to NetBIOS Session Service at port 139 on host gloin.

Successfully connected to Windows Shares at port 445 on host gloin.

Can not access the remote file system.  Is UAC on? : Access is denied.


I discovered the actual problem is that Windows 7 no longer has the "C$" Administrative file share available by default. If "C$" isn't available then nothing will work. Here is how to expose the "C$" share:

  • Registry Change -- this change allows “C$” to be accessed.  As soon as I set it -- the file copying started working!  [1]

    • regkey: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System

    • create this key, 32 bit word, with value == 1

  • LocalAccountTokenFilterPolicy

    4.  Turn on the Remote Registry Service -- This is critical and it's easy to do. Windows 7 has it turned off by default. MyComputer-right click, manage, services, then turn on Remote Registry Service and set it to start automatically in the fture.

    5. Turn off UAC: %systemroot%\system32\UserAccountControlSettings.exe

    6. This is where I discovered that McAfee virus scanner blocks all the NetBios shares!  It has to be disabled.


Thursday Mar 15, 2012

GlassFish Clustering with DCOM on Windows

DCOM - Distributed COM, a Microsoft protocol for communicating with Windows machines.

Why use DCOM?

  • In GlassFish 3.1 SSH is used as the standard way to run commands on remote nodes for clustering.  It is very difficult for users to get SSH configured properly on Windows.  SSH does not come with Windows so we have to depend on third party tools.  And then the user is forced to install and configure these tools -- which can be tricky.
  • DCOM is available on all supported platforms.  It is built-in to Windows.
The idea is to use DCOM to communicate with remote Windows nodes.  This has the huge advantage that the user has to do minimal, if any, configuration
on the Windows nodes.

Implementation Highlights

Two open Source Libraries have been added to GlassFish:
  • Jcifs – a SAMBA implementation in Java
  • J-interop – A Java implementation for making DCOM calls to remote Windows computers.  

Note that any supported platform can use DCOM to work with Windows nodes -- not just Windows.
E.g. you can have a Linux DAS work with Windows remote instances.

All existing SSH commands now have a corresponding DCOM command – except for setup-ssh which isn’t needed for DCOM.  
validate-dcom is an all new command.

New DCOM Commands

  • create-node-dcom
  • delete-node-dcom
  • install-node-dcom
  • list-nodes-dcom
  • ping-node-dcom
  • uninstall-node-dcom
  • update-node-dcom
  • validate-dcom
  • setup-local-dcom (This is only available via Update Center for GlassFish 3.1.2)

These commands are in-place in the trunk (4.0).  And in the branch (3.1.2)

Windows Configuration Challenges

There are an infinite number of possible configurations of Windows if you look at it as a combination of main release, service-pack, special drivers, software, configuration etc.  Later versions of Windows err on the side of tightening security be default.  This means that the Windows host may need to have configuration changes made.
These configuration changes mostly need to be made by the user.  setup-local-dcom will assist you in making required changes to the Windows Registry.  See the reference blogs for details.

The validate-dcom Command

validate-dcom is a crucial command.  It should be run before any other commands.  If it does not run successfully then there is no point in running other commands.
The validate-dcom command must be used from a DAS machine to test a different Windows machine.  
If  validate-dcom runs successfully you can be confident that all the DCOM commands will work.  Conversely, the opposite is also true:  If validate-dcom fails, then no DCOM commands will work.

What validate-dcom does

  1. Verify that the remote host is not the local machine.
  2. Resolves the remote host name
  3. Checks that the remote DCOM port is being listened on (135, 139)
  4. Checks that the remote host’s File Sharing is enabled (port 445)
  5. It copies a file (a script) to the remote host to verify that SAMBA is working and authorization is correct
  6. It runs a script that it copied on-the-fly to the remote host.

Tips and Tricks

The bread and butter commands that use DCOM are existing commands like create-instance, start-instance etc.   All of the commands that have dcom in their name are for dealing with the actual nodes.

The way the software works is to call asadmin.bat on the remote machine and run a command.  This means that you can track these commands easily on the remote machine with the usual tools.  E.g. using AS_LOGFILE, looking at log files, etc.  It’s easy to attach a debugger to the remote asadmin process, “just in time”, if necessary.

How to debug the remote commands:
Edit the asadmin.bat file that is in the glassfish/bin folder.  Use glassfish/lib/nadmin.bat in GlassFish 4.0+
Add these options to the java call:
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=1234
  Now if you run, say start-instance on DAS, you can attach your debugger, at your leisure, to the remote machines port 1234.  It will be running start-local-instance and patiently waiting for you to attach.

Thursday Mar 01, 2012

GlassFish DCOM Configuration Utility

Version 3.1.2 of GlassFish adds support for communicating with Windows remote hosts over DCOM.  DCOM is a communications protocol from Microsoft.  All Windows versions that GlassFish supports come with DCOM support built-in.  

In these days of high-security on Windows,  using DCOM may well require a few configuration steps.  One of the more challenging steps is to edit the Windows Registry.

We are using DCOM to run GlassFish processes on the remote Windows machines (we call them nodes).  What we do is the same thing command-line users do -- run asadmin on the remote machine.  Asadmin.bat is a script and in order to remotely run scripts you must have permissions set in the Windows Registry.  In fact you need two Registry entries set to give permission to the Administrators group.  These two are:

  1. WMI - Windows Management Interface
  2. WBEM Scripting

The problem with these 2 registry keys is that in many versions of Windows they are owned by an account named TrustedInstaller.  And by default the Administrators group have no permissions at all.  (Yes , I said the owner of the key -- Registry keys have owners with access control lists exactly analogous to files).  In my humble opinion this is a Windows Bug! 

Setting-up the registry keys correctly is fairly involved.  There are significant complications for 32 bit versus 64 bit Windows.  Windows does automatic redirection of Registry Keys.  You're probably thinking -- "This is just too painful to bear!".  And you are right.  That's why I wrote a C++ Windows program that does all of the registry changes for you.  But that's not all!  I embedded this C++ program inside a jar file that also contains an asadmin extension command.  In order to do the editing of the registry keys you simply need to deploy the extension command and run it one time on each Windows machine.  And then only if you need to run it.

The name of the command is setup-local-dcom.  You can get it from Update Center under contributions.  See my other blog for instructions on exactly how to get it from Update Center.

Instructions for running setup-local-dcom

  • Backup the Registry just in case.
  • Run the command on the Windows machine that will be used as a remote node for GlassFish
  • You need to be logged in to an account that is a member of the Administrators group.
  • run the following to see the manual page:
    asadmin setup-local-dcom --help

  • Run the command to edit the registry keys, if needed.  Use the --verbose option to get a report of what it did.
  • Note that the command makes you type in yes and hit return to give you one last chance to change your mind.  If you find that annoying use the --force option and it will silently run.
  • If the command was successful then go to any other computer that can access it and run the validate-local-dcom command against the machine you just ran setup-local-dcom on.  If validate-local-dcom runs successfully then all clustering commands should work fine in your distributed environment.  If validate-local-dcom fails then see another blog that describes further configuration steps you may need to make on the Windows node.

Example Output of setup-local-dcom --verbose

d:\>asadmin setup-local-dcom -v
Caution: This command might modify the permissions of some keys in the Windows registry.
Before running this command, back up the Windows registry.
The modification allows the Windows user full control over these keys.

Are you sure that you want to edit the Windows registry? If so, type yes in full:  yes
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
>>>>>>>>>>>>>   Set to Verbose Mode    <<<<<<<<<<<<<
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
Administrators group SID: [S-1-5-32-544]
Key: [CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]  Owner: [S-1-5-32-544]
Key: [CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]  Owner: [S-1-5-32-544]
Redirected Key: [Wow6432Node\CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]
Redirected Key: [Wow6432Node\CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]
No need to adjust the Scripting Registry Key.
No need to adjust the WMI Registry Key.
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
>>>>>>>>>>>>>   Set to Verbose Mode    <<<<<<<<<<<<<
>>>>>>>>>>>>>                          <<<<<<<<<<<<<
Administrators group SID: [S-1-5-32-544]
Key: [CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]  Owner: [S-1-5-32-544]
Key: [CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]  Owner: [S-1-5-32-544]
Redirected Key: [Wow6432Node\CLSID\{72C24DD5-D70A-438B-8A42-98424B88AFB8}]
Redirected Key: [Wow6432Node\CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}]
No need to adjust the Scripting Registry Key.
No need to adjust the WMI Registry Key.

Command setup-local-dcom executed successfully.




« February 2017