GlassFish 3.1: Using SSH to Manage Instance Lifecycles
By Joe Di Pol-Oracle on Nov 23, 2010
GlassFish 3.1 introduces the use of SSH to manage the lifecycle of remote instances. This document gives an overview of how to use SSH with GlassFish and describes some alternatives if SSH is not practical in your environment.
1) SSH use in GlassFish 3.1
The Domain Admin Server (DAS) for GlassFish must be able to communicate with remote instances to perform various administrative operations. For most operations this communication occurs directly between the DAS and the running instance. For example when you deploy an application to an instance the DAS connects to the instance and deploys the application to it. But what about starting a remote instance? Or creating it in the first place? In these cases there is no running instance for the DAS to talk to.
In v2 these operations where handled by another process that had to be installed and running on the remote system -- the GlassFish node agent. To start a remote instance the DAS contacted the node agent, and the node agent started the instance.
In 3.1 there is no node agent. Instead the DAS uses SSH to perform certain operations on the remote system. The use of SSH is optional, and it is only needed for specific commands when those commands are used to manage remote instances (instances on systems other than the one the DAS is running on -- instances local to the DAS are supported without SSH). This table lists the commands that have a dependency on SSH when the instance is remote, and lists a command that can be used as an alternative when SSH is not available. The alternate commands are local commands that must be run directly on the system where the instance resides.
||Alternate Local Command
|start-instance||start-local-instance||Or use the native service support in GlassFish (create-service)|
|start-cluster||start-local-instance||For each instance. Or use the native service support in GlassFish (create-service)
So that's the first decision you need to make when setting up a GlassFish cluster. Do you want to leverage SSH to manage the instance lifecycle? Or do you want to avoid the dependency and perform those operations directly on the local machines yourself.
The benefit of using SSH is that, once it is set up, it makes your life easier since you can perform the above operations centrally via the DAS. The downside is that it requires some effort to get SSH set up (especially for Windows users).
Note: In GlassFish a "remote" command is a command that talks to the DAS and the DAS performs the operation. A "local" command is a command where the asadmin client performs the operation directly (and therefore the command must be run from the GlassFish installation you want to operate on). Some GlassFish "local" commands, like create-local-instance, are actually a hybrid since it needs to contact the DAS to inform it of the new instance. As you might expect the remote commands listed in the table above use SSH to execute the local version of the command directly on the remote instance.
2) What Is SSH?
If you are a Unix user you are likely already familiar with SSH, but Windows users may not be.
SSH consists of two parts: a client and a server.
The most common client is the ssh(1) command. This lets you securely log into a remote system and gives you a remote shell. Other client commands are scp and sftp which let you perform file transfers. The server is sshd or secshd and it provides the SSH service.
The GlassFish DAS uses its own Java based SSH client for communicating with the instance systems (which must be running the SSH service). But the native SSH client is still needed for things like key generation and helping to verify connectivity. In a typical GlassFish deployment the DAS acts as the SSH client, while the instance systems must be running the SSH service.
3) Getting SSH
Most all Unix based systems include SSH support. On the instance systems you need to make sure the SSH service is running (on the Mac you'll need to turn on Remote Login in the Sharing System Preferences).
On Windows you have more work to do. You'll need to choose an SSH provider and install it on all instance systems and the DAS. GlassFish has been tested with Cygwin and MKS:
|Product|| Version Tested
|Cygwin||1.7.6||Installing Windows Cygwin sshd|
| MKS Toolkit for Developers
Once you install your SSH provider, make sure that the provider's "bin"
directory is in your (Windows) PATH so that you have easy access to the
SSH client commands (ssh, sftp, ssh-keygen, etc).
Once you have SSH installed and the SSH service running, verify you can login from the DAS host to the instance systems using the ssh(1) client: ssh firstname.lastname@example.org. It should prompt for your password (unless you already have key authentication set up). If you can't ssh into the remote system then you'll have to sort that out before using SSH with GlassFish.
Note: Yamini K B has a nice blog entry (Using GlassFish v3.1 SSH Provisioning Commands) that does a good job summarizing what needs to be done to get SSH working on different operating systems.
4) Mixing Different OS Types
The DAS and all GlassFish instances in a cluster must be running on the same OS type and version. Running a cluster of mixed OS types is not supported. In practice mixing different flavors of Unix usually works OK (but is not officially supported), but mixing Windows and Unix is asking for trouble.
5) SSH Authentication
When the SSH client connects to the SSH service it needs to authenticate the user. The most basic form of authentication is password authentication. SSH also supports other forms of authentication. In particular it support public key authentication.
GlassFish 3.1 supports 3 forms of SSH authentication. They each have trade-offs:
| Authentication Scheme
| Username / Password
|| Easy to configure in SSH.
||A bit more work configuring GlassFish to use this in a secure fasion.
| Public Key (unencrypted)
|| Easy to configure GlassFish to use.
||Can be trickier to get configured correctly in SSH.
| Public Key (encrypted)
||More secure than unencrypted public key.
||More work to configure both SSH and GlassFish to use this.
To use Username/Password securely with GlassFish you will need to use a GlassFish feature called "password aliases" so that your SSH password is not stored in the clear. This isn't too bad, but does require a little extra fiddling to create and use the password alias.
The unencrypted public key approach requires that you generate an SSH key pair and then copy the public key to the remote systems. GlassFish provides an asadmin command, setup-ssh, to help you do this. Since there is no password you don't need to mess with password aliases, and the DAS looks in the default places to locate the key to authenticate with -- so you often don't even need to provide that info. But folks sometimes run into trouble getting key authentication working with SSH. It usually works great, but if it doesn't it can be tricky to figure out why.
In the encrypted key file approach you encrypt your private key file with a passphrase that must be provided to use the private key. This means you must configure GlassFish to know about the passphrase -- which means using password aliases again. So this method combines the complexity of the first two approaches. You probably won't choose this approach unless it is the preferred SSH authentication scheme at your site.
If you already have SSH deployed at your site you may have the preferred authentication scheme already defined for you.
At this point you should decide what form of authentication you want to use.
6) What user?
One basic decision you will need to make is concerning what SSH user you want GlassFish to use when it connects to remote systems. Some considerations:
- For key authentication the user that the DAS process is running as must be able to read the SSH user's private key file (typically in ~/.ssh).
- Remote instances will be started, and therefore be running as, the SSH user.
- By default the DAS will assume the SSH user is the same as the user running the DAS process.
7) Setting Up SSH Key Authentication
GlassFish provides a command, setup-ssh, that sets up SSH key authentication for you. You run this command directly on the DAS system. Here is an example run. In this example adc2101159 is the name of the remote host you will be running an instance on, and we assume the user running the command (dipol) is the same user that will start the DAS and the same user that we want GlassFish to use when connecting to remote systems over SSH:
$ asadmin setup-ssh adc2101159 SSH key not found for user dipol Would you like to generate an SSH key pair (without a key passphrase) for dipol to access [adc2101159]? [y/n]: y Enter SSH password for dipol@adc2101159 Created directory /export/home/dipol/.ssh /usr/bin/ssh-keygen successfully generated the identification /export/home/dipol/.ssh/id_rsa Copied keyfile /export/home/dipol/.ssh/id_rsa.pub to dipol@adc2101159 Successfully connected to dipol@adc2101159 using keyfile /export/home/dipol/.ssh/id_rsa Command setup-ssh executed successfully.
As you can see setup-ssh does a couple things:
- It checks to see if you have an SSH key already generated. If you don't it gives you the option to generate one.
- After generating the key it copies it to the remote host. To do this it needs your SSH password to make the connection to copy the key.
- It then connects to the remote system to verify that key authentication is working
After key authentication is set up you can re-run setup-ssh any time and it will detect the key and verify the connection with it:
$ asadmin setup-ssh adc2101159 Successfully connected to dipol@adc2101159 using keyfile /export/home/dipol/.ssh/id_rsa SSH public key authentication is already configured for dipol@adc2101159 Command setup-ssh executed successfully.
If setup-ssh fails you need to decide if you want to debug why key authentication is failing, our give up and use password authentication. A couple things to check if key authentication fails:
- Make sure on the remote system that the file system permissions for the SSH user are not too permissive. The SSH user's home directory, ~/.ssh directory, and authorized_key file must be writeable only by the SSH user. If they are writeable by others then the secure service may not trust the authorized_key file and may reject public key authentication.
- On MKS we have run into a case where fixing #1 required:
- Copying the .ssh\\id_rsa key file over to the remote system manually and importing it with the MKS configuration GUI (Passwordless button under Secure Service tab) or
- Using the MKS configuration GUI to turn off "enforce strict mode".
8) Installing GlassFish
GlassFIsh must be installed on each system that will host an instance. You have two choices on how to do this:
- Install manually on each system
- Install manually on the DAS system, then use the install-node command to create a GlassFish install image based on the DAS install and then install that image on the remote instance systems.
Option 1 is best if you want complete control over the installation and install time settings. Option 2 works well if all of your systems are set up the same and you want to duplicate the DAS GlassFish installation.
Here is an example of an install-node run. This is run on the DAS system:
asadmin install-node --installdir /export/tmp/glassfish3 adc2101159 Created installation zip /export2/glassfish3/glassfish.zip Copying /export2/glassfish3/glassfish.zip (83527743 bytes) to adc2101159:/export/tmp/glassfish3 Installing /export2/glassfish3/glassfish.zip into adc2101159:/export/tmp/glassfish3 Removing adc2101159:/export/tmp/glassfish3/glassfish.zip Fixing file permissions of all files under adc2101159:/export/tmp/glassfish3/bin Command install-node executed successfully.
Note: Currently install-node works best with SSH key authentication.
9) Nodes and Instances
In GlassFish 3.1 a Node is a configuration object in the DAS that represents a GlassFish installation on a host. There are two types of nodes:
- SSH nodes
- CONFIG nodes
Both node types contain some basic information about the node like:
- The hostname of the remote host
- The location of the GlassFish installation on the remote host
An SSH node also contains information needed to connect to the host via SSH such as the SSH username to use, the SSH port number, etc.
The DAS ships with one node already created named "localhost-<domainname>". This is a CONFIG node that can be used when creating instances local to the DAS. See the next section for more info on creating instances local to the DAS. CONFIG nodes are also used with remote instances when SSH is not going to be used. See section 11 for more info on that. But if you want to manage a remote instance via SSH then you need to first create an SSH node to represent a GlassFish installation on the remote system.
You use the create-node-ssh command to create SSH nodes. If you have SSH key authentication already set up (section 7) and the DAS is running as the sshuser (section 6), then things are really easy:
$ asadmin create-node-ssh --nodehost adc2101159 --installdir /export/glassfish3 node1 Command create-node-ssh executed successfully.
You can use list-nodes to see what nodes are defined:
$ asadmin list-nodes --long NODE NAME TYPE NODE HOST INSTALL DIRECTORY REFERENCED BY localhost CONFIG localhost /export2/glassfish3 node1 SSH adc2101159 /export/glassfish3 Command list-nodes executed successfully.
If you are using SSH password authentication then you need to do a little more work to securely set the SSH password on the SSH node. 3.1 SSH Authentication has some more details on running create-node-ssh with other SSH authentication schemes.
create-node-ssh will validate the parameters you pass it. For example it makes sure it can connect to the remote host using SSH, and it verifies that the installdir you pass it is a GlassFish install. Therefore you should have SSH set up and GlassFish installed on the remote instances before you start creating SSH nodes.
Once you have the SSH node created, you can then create an instance on the node:
$ asadmin create-instance --node node1 instance1 Command _create-instance-filesystem executed successfully. Port Assignments for server instance instance1: JMX_SYSTEM_CONNECTOR_PORT=28686 JMS_PROVIDER_PORT=27676 HTTP_LISTENER_PORT=28080 ASADMIN_LISTENER_PORT=24848 JAVA_DEBUGGER_PORT=29009 IIOP_SSL_LISTENER_PORT=23820 IIOP_LISTENER_PORT=23700 OSGI_SHELL_TELNET_PORT=26666 HTTP_SSL_LISTENER_PORT=28181 IIOP_SSL_MUTUALAUTH_PORT=23920 The instance, instance1, was created on host adc2101159 Command create-instance executed successfully.
And start it:
$ ./asadmin start-instance instance1 Waiting for instance1 to start ...... Successfully started the instance: instance1 instance Location: /export/glassfish3/glassfish/nodes/node1/instance1 Log File: /export/glassfish3/glassfish/nodes/node1/instance1/logs/server.log Admin Port: 24848 Command start-local-instance executed successfully. The instance, instance1, was started on host adc2101159 Command start-instance executed successfully.
If at any time you want to test the connection to an SSH node you can do it using the asadmin ping-node-ssh command:
$ ./asadmin ping-node-ssh node1 Successfully made SSH connection to node node1 (adc2101159) Command ping-node-ssh executed successfully.
10) Local Instances
To create instances on the same system as the DAS (typically using the same GlassFish installation as the DAS) you do not need SSH and you don't need to worry about nodes at all. Run this command from the same installation as the DAS:
$ ./asadmin create-local-instance localinstance1 Port Assignments for server instance localinstance1: JMX_SYSTEM_CONNECTOR_PORT=28686 JMS_PROVIDER_PORT=27676 HTTP_LISTENER_PORT=28080 ASADMIN_LISTENER_PORT=24848 JAVA_DEBUGGER_PORT=29009 IIOP_SSL_LISTENER_PORT=23820 IIOP_LISTENER_PORT=23700 OSGI_SHELL_TELNET_PORT=26666 HTTP_SSL_LISTENER_PORT=28181 IIOP_SSL_MUTUALAUTH_PORT=23920 The instance, localinstance1, was created on host localhost Command create-local-instance executed successfully.This creates an instance local to the DAS using the build in localhost node.
11) Doing Remote Instances Without SSH
If you do not want to use SSH for managing remote instances you can still create and manage them using the local version of asadmin commands as mentioned in section 1. Also, due to an "auto-node creation" feature you don't have to explicitly create the node before hand. For example if you have this setup:
- GlassFish installation on two systems: systemA and systemB
- DAS running on systemA
Then you can create an instance on systemB by doing the following:
Log into systemB and run the following using the GlassFish installation on systemB:
$ asadmin --host systemA create-local-instance myinstance1 Rendezvoused with DAS on systemA:4848. Port Assignments for server instance myinstance1: JMX_SYSTEM_CONNECTOR_PORT=28687 JMS_PROVIDER_PORT=27677 HTTP_LISTENER_PORT=28081 ASADMIN_LISTENER_PORT=24849 JAVA_DEBUGGER_PORT=29010 IIOP_SSL_LISTENER_PORT=23821 IIOP_LISTENER_PORT=23701 OSGI_SHELL_TELNET_PORT=26667 HTTP_SSL_LISTENER_PORT=28182 IIOP_SSL_MUTUALAUTH_PORT=23921 Command create-local-instance executed successfully.Note: Since create-local-instance connects to the DAS from a remote system, the DAS must be running with secure admin enabled. See the asadmin enable-secure-admin command -- you will need to restart the DAS after enabling secure admin.
This command does the following:
- Creates a CONFIG node in the DAS configuration for systemB if such a node does not already exist. The node name is the hostname of systemB.
- Creates a server instance object in the DAS configuration for instance myinstance1
- Creates the instance files in the filesystem on systemB for instance myinstance1
You can then start the instance while still logged into systemB:
$ asadmin start-local-instance myinstance1 Waiting for myinstance1 to start ...... Successfully started the instance: myinstance1 instance Location: /export/glassfish3/glassfish/nodes/adc2101159/myinstance1 Log File: /export/glassfish3/glassfish/nodes/adc2101159/myinstance1/logs/server.log Admin Port: 24849 Command start-local-instance executed successfully.
As mentioned in section 1 if you are managing remote instances without SSH then you will not be able to use start-cluster to start instances in a cluster. You will need to use start-local-instance on each instance or use GlassFish's native service support to start the instances.