GlassFish 3.1: Using SSH to Manage Instance Lifecycles

GlassFish 3.1 introduces the use of SSH to manage the lifecycle of remote instances. This document gives an overview of how to use SSH with GlassFish and describes some alternatives if SSH is not practical in your environment.

1) SSH use in GlassFish 3.1

The Domain Admin Server (DAS) for GlassFish must be able to communicate with remote instances to perform various administrative operations. For most operations this communication occurs directly between the DAS and the running instance. For example when you deploy an application to an instance the DAS connects to the instance and deploys the application to it. But what about starting a remote instance? Or creating it in the first place? In these cases there is no running instance for the DAS to talk to.

In v2 these operations where handled by another process that had to be installed and running on the remote system -- the GlassFish node agent. To start a remote instance the DAS contacted the node agent, and the node agent started the instance.

In 3.1 there is no node agent. Instead the DAS uses SSH to perform certain operations on the remote system. The use of SSH is optional, and it is only needed for specific commands when those commands are used to manage remote instances (instances on systems other than the one the DAS is running on -- instances local to the DAS are supported without SSH). This table lists the commands that have a dependency on SSH when the instance is remote, and lists a command that can be used as an alternative when SSH is not available. The alternate commands are local commands that must be run directly on the system where the instance resides.

Command
Alternate Local Command
Comment
 create-instance  create-local-instance
 delete-instance  delete-local-instance
 start-instance  start-local-instance Or use the native service support in GlassFish (create-service)
 start-cluster  start-local-instance For each instance. Or use the native service support in GlassFish (create-service)


So that's the first decision you need to make when setting up a GlassFish cluster. Do you want to leverage SSH to manage the instance lifecycle? Or do you want to avoid the dependency and perform those operations directly on the local machines yourself.

The benefit of using SSH is that, once it is set up, it makes your life easier since you can perform the above operations centrally via the DAS. The downside is that it requires some effort to get SSH set up (especially for Windows users).

Note: In GlassFish a "remote" command is a command that talks to the DAS and the DAS performs the operation. A "local" command is a command where the asadmin client performs the operation directly (and therefore the command must be run from the GlassFish installation you want to operate on). Some GlassFish "local" commands, like create-local-instance, are actually a hybrid since it needs to contact the DAS to inform it of the new instance. As you might expect the remote commands listed in the table above use SSH to execute the local version of the command directly on the remote instance.

2) What Is SSH?

If you are a Unix user you are likely already familiar with SSH, but Windows users may not be.

SSH consists of two parts: a client and a server.

The most common client is the ssh(1) command. This lets you securely log into a remote system and gives you a remote shell. Other client commands are scp and sftp which let you perform file transfers. The server is sshd or secshd and it provides the SSH service.

The GlassFish DAS uses its own Java based SSH client for communicating with the instance systems (which must be running the SSH service). But the native SSH client is still needed for things like key generation and helping to verify connectivity. In a typical GlassFish deployment the DAS acts as the SSH client, while the instance systems must be running the SSH service.

3) Getting SSH

Most all Unix based systems include SSH support. On the instance systems you need to make sure the SSH service is running (on the Mac you'll need to turn on Remote Login in the Sharing System Preferences).

On Windows you have more work to do. You'll need to choose an SSH provider and install it on all instance systems and the DAS. GlassFish has been tested with Cygwin and MKS:

Product Version Tested
 Notes
Cygwin  1.7.6 Installing Windows Cygwin sshd
 MKS Toolkit for Developers
 9.2

Once you install your SSH provider, make sure that the provider's "bin" directory is in your (Windows) PATH so that you have easy access to the SSH client commands (ssh, sftp, ssh-keygen, etc).

Once you have SSH installed and the SSH service running, verify you can login from the DAS host to the instance systems using the ssh(1) client: ssh username@instance.hostname.com. It should prompt for your password (unless you already have key authentication set up). If you can't ssh into the remote system then you'll have to sort that out before using SSH with GlassFish.

Note: Yamini K B has a nice blog entry (Using GlassFish v3.1 SSH Provisioning Commands) that does a good job summarizing what needs to be done to get SSH working on different operating systems.

4) Mixing Different OS Types

The DAS and all GlassFish instances in a cluster must be running on the same OS type and version. Running a cluster of mixed OS types is not supported. In practice mixing different flavors of Unix usually works OK (but is not officially supported), but mixing Windows and Unix is asking for trouble.

5) SSH Authentication

When the SSH client connects to the SSH service it needs to authenticate the user. The most basic form of authentication is password authentication. SSH also supports other forms of authentication. In particular it support public key authentication.

GlassFish 3.1 supports 3 forms of SSH authentication. They each have trade-offs:

 Authentication Scheme
Pro Con
 Username / Password
 Easy to configure in SSH.
A bit more work configuring GlassFish to use this in a secure fasion.
 Public Key (unencrypted)
 Easy to configure GlassFish to use.
Can be trickier to get configured correctly in SSH.
 Public Key (encrypted)
More secure than unencrypted public key.
More work to configure both SSH and GlassFish to use this.


To use Username/Password securely with GlassFish you will need to use a GlassFish feature called "password aliases" so that your SSH password is not stored in the clear. This isn't too bad, but does require a little extra fiddling to create and use the password alias.

The unencrypted public key approach requires that you generate an SSH key pair and then copy the public key to the remote systems. GlassFish provides an asadmin command, setup-ssh, to help you do this. Since there is no password you don't need to mess with password aliases, and the DAS looks in the default places to locate the key to authenticate with -- so you often don't even need to provide that info. But folks sometimes run into trouble getting key authentication working with SSH. It usually works great, but if it doesn't it can be tricky to figure out why.

In the encrypted key file approach you encrypt your private key file with a passphrase that must be provided to use the private key. This means you must configure GlassFish to know about the passphrase -- which means using password aliases again. So this method combines the complexity of the first two approaches. You probably won't choose this approach unless it is the preferred SSH authentication scheme at your site.

If you already have SSH deployed at your site you may have the preferred authentication scheme already defined for you.

 At this point you should decide what form of authentication you want to use.

6) What user?

One basic decision you will need to make is concerning what SSH user you want GlassFish to use when it connects to remote systems. Some considerations:

  1. For key authentication the user that the DAS process is running as must be able to read the SSH user's private key file (typically in ~/.ssh).
  2. Remote instances will be started, and therefore be running as, the SSH user.
  3. By default the DAS will assume the SSH user is the same as the user running the DAS process.
So it ends up being easiest if the SSH user is the same as the user that starts the DAS. This doesn't have to be the case -- it just makes things simpler.

7) Setting Up SSH Key Authentication

GlassFish provides a command, setup-ssh, that sets up SSH key authentication for you. You run this command directly on the DAS system. Here is an example run. In this example adc2101159 is the name of the remote host you will be running an instance on, and we assume the user running the command (dipol) is the same user that will start the DAS and the same user that we want GlassFish to use when connecting to remote systems over SSH:

 
  $ asadmin setup-ssh adc2101159
  SSH key not found for user dipol
  Would you like to generate an SSH key pair (without a key passphrase) for dipol to access [adc2101159]? [y/n]: y
  Enter SSH password for dipol@adc2101159
  Created directory /export/home/dipol/.ssh
  /usr/bin/ssh-keygen successfully generated the identification /export/home/dipol/.ssh/id_rsa
  Copied keyfile /export/home/dipol/.ssh/id_rsa.pub to dipol@adc2101159
  Successfully connected to dipol@adc2101159 using keyfile /export/home/dipol/.ssh/id_rsa
  Command setup-ssh executed successfully.
  

As you can see setup-ssh does a couple things:

  1. It checks to see if you have an SSH key already generated. If you don't it gives you the option to generate one.
  2. After generating the key it copies it to the remote host. To do this it needs your SSH password to make the connection to copy the key.
  3. It then connects to the remote system to verify that key authentication is working

After key authentication is set up you can re-run setup-ssh any time and it will detect the key and verify the connection with it:

  $ asadmin setup-ssh adc2101159
  Successfully connected to dipol@adc2101159 using keyfile /export/home/dipol/.ssh/id_rsa
  SSH public key authentication is already configured for dipol@adc2101159
  Command setup-ssh executed successfully.

If setup-ssh fails you need to decide if you want to debug why key authentication is failing, our give up and use password authentication. A couple things to check if key authentication fails:

  1. Make sure on the remote system that the file system permissions for the SSH user are not too permissive. The SSH user's home directory, ~/.ssh directory, and authorized_key file must be writeable only by the SSH user. If they are writeable by others then the secure service may not trust the authorized_key file and may reject public key authentication.
  2. On MKS we have run into a case where fixing #1 required:
    1. Copying the .ssh\\id_rsa key file over to the remote system manually and importing it with the MKS configuration GUI (Passwordless button under Secure Service tab) or
    2. Using the MKS configuration GUI to turn off "enforce strict mode".

8) Installing GlassFish

GlassFIsh must be installed on each system that will host an instance. You have two choices on how to do this:

  1. Install manually on each system
  2. Install manually on the DAS system, then use the install-node command to create a GlassFish install image based on the DAS install and then install that image on the remote instance systems.

Option 1 is best if you want complete control over the installation and install time settings. Option 2 works well if all of your systems are set up the same and you want to duplicate the DAS GlassFish installation.

Here is an example of an install-node run. This is run on the DAS system:

  asadmin install-node --installdir /export/tmp/glassfish3 adc2101159
  Created installation zip /export2/glassfish3/glassfish.zip
  Copying /export2/glassfish3/glassfish.zip (83527743 bytes) to adc2101159:/export/tmp/glassfish3 
  Installing /export2/glassfish3/glassfish.zip into adc2101159:/export/tmp/glassfish3
  Removing adc2101159:/export/tmp/glassfish3/glassfish.zip
  Fixing file permissions of all files under adc2101159:/export/tmp/glassfish3/bin
  Command install-node executed successfully.

Note: Currently install-node works best with SSH key authentication.

9) Nodes and Instances

In GlassFish 3.1 a Node is a configuration object in the DAS that represents a GlassFish installation on a host. There are two types of nodes:

  1. SSH nodes
  2. CONFIG nodes

Both node types contain some basic information about the node like:

  • The hostname of the remote host
  • The location of the GlassFish installation on the remote host

An SSH node also contains information needed to connect to the host via SSH such as the SSH username to use, the SSH port number, etc.

The DAS ships with one node already created named "localhost-<domainname>". This is a CONFIG node that can be used when creating instances local to the DAS. See the next section for more info on creating instances local to the DAS. CONFIG nodes are also used with remote instances when SSH is not going to be used. See section 11 for more info on that. But if you want to manage a remote instance via SSH then you need to first create an SSH node to represent a GlassFish installation on the remote system.

You use the create-node-ssh command to create SSH nodes. If you have SSH key authentication already set up (section 7) and the DAS is running as the sshuser (section 6), then things are really easy:

  $ asadmin create-node-ssh --nodehost adc2101159 --installdir /export/glassfish3 node1
  Command create-node-ssh executed successfully.

You can use list-nodes to see what nodes are defined:

  $ asadmin list-nodes --long
  NODE NAME   TYPE    NODE HOST   INSTALL DIRECTORY    REFERENCED BY
  localhost   CONFIG  localhost   /export2/glassfish3   
  node1       SSH     adc2101159  /export/glassfish3
  Command list-nodes executed successfully.

If you are using SSH password authentication then you need to do a little more work to securely set the SSH password on the SSH node. 3.1 SSH Authentication has some more details on running create-node-ssh with other SSH authentication schemes.

create-node-ssh will validate the parameters you pass it. For example it makes sure it can connect to the remote host using SSH, and it verifies that the installdir you pass it is a GlassFish install. Therefore you should have SSH set up and GlassFish installed on the remote instances before you start creating SSH nodes.

Once you have the SSH node created, you can then create an instance on the node:

  $ asadmin create-instance --node node1 instance1
  Command _create-instance-filesystem executed successfully.
  Port Assignments for server instance instance1:
  JMX_SYSTEM_CONNECTOR_PORT=28686
  JMS_PROVIDER_PORT=27676
  HTTP_LISTENER_PORT=28080
  ASADMIN_LISTENER_PORT=24848
  JAVA_DEBUGGER_PORT=29009
  IIOP_SSL_LISTENER_PORT=23820
  IIOP_LISTENER_PORT=23700
  OSGI_SHELL_TELNET_PORT=26666
  HTTP_SSL_LISTENER_PORT=28181
  IIOP_SSL_MUTUALAUTH_PORT=23920
  The instance, instance1, was created on host adc2101159
  Command create-instance executed successfully.

And start it:

  $ ./asadmin start-instance instance1
  Waiting for instance1 to start ......
  Successfully started the instance: instance1
  instance Location: /export/glassfish3/glassfish/nodes/node1/instance1
  Log File: /export/glassfish3/glassfish/nodes/node1/instance1/logs/server.log
  Admin Port: 24848
  Command start-local-instance executed successfully.
  The instance, instance1, was started on host adc2101159
  Command start-instance executed successfully.

If at any time you want to test the connection to an SSH node you can do it using the asadmin ping-node-ssh command:

 $ ./asadmin ping-node-ssh node1
Successfully made SSH connection to node node1 (adc2101159)
Command ping-node-ssh executed successfully.

10) Local Instances

To create instances on the same system as the DAS (typically using the same GlassFish installation as the DAS) you do not need SSH and you don't need to worry about nodes at all. Run this command from the same installation as the DAS:

  $ ./asadmin create-local-instance localinstance1
  Port Assignments for server instance localinstance1:
  JMX_SYSTEM_CONNECTOR_PORT=28686
  JMS_PROVIDER_PORT=27676
  HTTP_LISTENER_PORT=28080
  ASADMIN_LISTENER_PORT=24848
  JAVA_DEBUGGER_PORT=29009
  IIOP_SSL_LISTENER_PORT=23820
  IIOP_LISTENER_PORT=23700
  OSGI_SHELL_TELNET_PORT=26666
  HTTP_SSL_LISTENER_PORT=28181
  IIOP_SSL_MUTUALAUTH_PORT=23920
  The instance, localinstance1, was created on host localhost
  Command create-local-instance executed successfully.
This creates an instance local to the DAS using the build in localhost node.

11) Doing Remote Instances Without SSH 

If you do not want to use SSH for managing remote instances you can still create and manage them using the local version of asadmin commands as mentioned in section 1. Also, due to an "auto-node creation" feature you don't have to  explicitly create the node before hand. For example if you have this setup:

  • GlassFish installation on two systems: systemA and systemB
  • DAS running on systemA

Then you can create an instance on systemB by doing the following:

Log into systemB and run the following using the GlassFish installation on systemB:

  $ asadmin --host systemA create-local-instance myinstance1
  Rendezvoused with DAS on systemA:4848.
  Port Assignments for server instance myinstance1:
  JMX_SYSTEM_CONNECTOR_PORT=28687
  JMS_PROVIDER_PORT=27677
  HTTP_LISTENER_PORT=28081
  ASADMIN_LISTENER_PORT=24849
  JAVA_DEBUGGER_PORT=29010
  IIOP_SSL_LISTENER_PORT=23821
  IIOP_LISTENER_PORT=23701
  OSGI_SHELL_TELNET_PORT=26667
  HTTP_SSL_LISTENER_PORT=28182
  IIOP_SSL_MUTUALAUTH_PORT=23921
  Command create-local-instance executed successfully.
Note: Since create-local-instance connects to the DAS from a remote system, the DAS must be running with secure admin enabled. See the asadmin enable-secure-admin command -- you will need to restart the DAS after enabling secure admin.

This command does the following:

  1. Creates a CONFIG node in the DAS configuration for systemB if such a node does not already exist. The node name is the hostname of systemB.
  2. Creates a server instance object in the DAS configuration for instance myinstance1
  3. Creates the instance files in the filesystem on systemB for instance myinstance1

You can then start the instance while still logged into systemB:

  $ asadmin start-local-instance myinstance1
  Waiting for myinstance1 to start ......
  Successfully started the instance: myinstance1
  instance Location: /export/glassfish3/glassfish/nodes/adc2101159/myinstance1
  Log File: /export/glassfish3/glassfish/nodes/adc2101159/myinstance1/logs/server.log
  Admin Port: 24849
  Command start-local-instance executed successfully. 

As mentioned in section 1 if you are managing remote instances without SSH then you will not be able to use start-cluster to start instances in a cluster. You will need to use start-local-instance on each instance or use GlassFish's native service support to start the instances.


Comments:

Hello Joe,

Do you know if there is a way to automate the startup of instance with a startup script, without having to enter the password.

I am using something like shown below :
asadmin --user admin --passwordfile /tmp/pswd.txt start-local-instance --node dev-nodeagent instance1

Posted by dilip on December 27, 2011 at 01:18 AM PST #

You may want to look into our native service support. Check out Byron's blog entry: http://blogs.oracle.com/foo/entry/automatic_starting_of_servers_in

Joe

Posted by guest on December 29, 2011 at 10:32 AM PST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

jdipol

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today