Thursday Mar 01, 2012

GlassFish 3.1.2: Secure By Default Changes

Secure by default is the characteristic of software where its default installation results in a secure configuration. Often there is a trade off between ease of use and the degree of default security.

In GlassFish 3.1.2 we have improved the secure by default behavior, and we've tried to do so without impacting the ease of use that GlassFish is known for (at least not impacting it too much). We had two main goals we were trying to satisfy in 3.1.2:

  1. More actively encourage the user to set an admin password at installation time.
  2. Require an admin password whenever remote administration (aka "secure admin") is enabled.

To achieve this you'll notice the following changes when you use 3.1.2:

  1. The installer now prompts for an admin password even in the default installation mode. You are still allowed to choose no password because remote administration is not enabled out of the box.
  2. Any time remote administration is enabled (by running the enable-secure-admin command for example), you will be required to have an admin password set. Basically GlassFish will do what it can to prevent you from enabling remote administration while not having an admin password.
  3. For the Oracle GlassFish Server commercial zip distributions (where there is no installer) you will be prompted for an admin password the first time you start the default domain (DAS). And just like with the installer, you are still allowed to choose no password because remote administration is not enabled out of the box. The open source / community zips continue to behave as they did in 3.1.1 (no prompting).

One issue our QA organization ran into when running their automated tests on the Oracle GlassFish Server commercial zip bundles had to do with their scripts that automatically installed (unzipped) GlassFish and started the default domain. These scripts started failing because the server was prompting for an admin password at startup.

The solution is to use the change-admin-password command to set an admin password before starting the domain. As part of the 3.1.2 changes we enhanced change-admin-password so that it could be run without the domain (DAS) running if you use the "--domain_name" option.  Here is an example of how to set the admin password on a domain before starting the domain the first time (command output removed for brevity):

$ unzip
$ cd glassfish3/glassfish
$ touch /tmp/password.txt
$ chmod 600 /tmp/password.txt
$ echo "AS_ADMIN_PASSWORD=" > /tmp/password.txt
$ echo "AS_ADMIN_NEWPASSWORD=newadminpassword" >> /tmp/password.txt
$ bin/asadmin --user admin --passwordfile /tmp/password.txt change-admin-password \
    --domain_name domain1
$ rm /tmp/password.txt
$ bin/asadmin start-domain

This does the following:

  • Installs glassfish by unzipping the zip
  • Creates a file (/tmp/password.txt) to supply passwords to the asadmin command. We make sure the file is readable only by the user running asadmin and remove the file when we are done for security purposes. The file has two lines that look like:

  •  Uses the asadmin change-admin-password command to set the admin password on the domain. Now the domain can be started.

For more information about security changes in GlassFish 3.1.2 see Tim Quinn's blog entry, and as always the GlassFish Security Guide is recommended reading.

Sunday Feb 27, 2011

GlassFish 3.1: Clusters Without SSH

GlassFish 3.1 has the ability to use SSH to centrally administer some basic instance lifecycle operations (such as creating and starting an instance). But in some cases you may not want to deal with SSH, and would rather just create and start those instances directly yourself. GlassFish 3.1 supports that too. This blog describes how to set up a two instance cluster across three machines without using SSH.

As part of this procedure you will be enabling secure admin on the GlassFish Domain Admin Server. You can read more about secure admin at Tim's blog.

1) Install GlassFish 3.1

First download install GlassFish 3.1 on three machines. One machine will run the Domain Admin Server (DAS) and the other two machines will each run one GlassFish instance. Let's call these systems dashost, host1 and host2.

Some things to keep in mind:

  1. Install JDK 6 Update 24 or later on each system. You absolutely will have problems if you run GlassFish with anything earlier than update 22.
  2. If you run the installer make sure to choose this JDK to use. If you use the zip distribution of GlassFish you can configure GlassFish to use a particular JDK by adding a line like this to glassfish3/glassfish/config/asenv.conf (asenv.bat for windows):  AS_JAVA="/opt/jdk/jdk1.6.0_24" (make sure to specify the correct path for your system).

2) Start the DAS, enable secure admin and create the cluster

On the system dashost start the domain and verify your JDK version:

dashost$ asadmin start-domain
dashost$ asadmin version --verbose
Version = GlassFish Server Open Source Edition 3.1 (build 43), JRE version 1.6.0_24
Command version executed successfully.

Verify that the JRE version is 1.6.0_24 or newer. If it isn't, make sure asenv.conf has AS_JAVA set to the correct path and restart the domain.

Next, enable secure admin and verify it is working:

dashost$ asadmin enable-secure-admin
dashost$ asadmin stop-domain
dashost$ asadmin start-domain 
dashost$ asadmin version --verbose
. . .
0070: 38 67 51 0D F8 AF 3E 6F   C7 58 02 DB 3B 70 22 66  8gQ...>o.X..;p"f

Do you trust the above certificate [y|N] --> y
Version = GlassFish Server Open Source Edition 3.1 (build 43), JRE version 1.6.0_24
Command version executed successfully.

Note that you are prompted to accept a certificate. This happens the first time you connect to the DAS from a host after enabling secure admin. Answer 'y' and you should see the version information reported.

If you get an error about the server not accepting a secure connection, or if the command hangs then odds are you are not running with JDK 1.6 update 24 or newer.

Next, create the cluster that the instances will be in:

dashost$ asadmin create-cluster cluster1

3) Create instances on the remote systems

Now log into your remote systems and create the instances. We will once again verify the JDK version first. It should look something like this:

$host1 asadmin version --local --verbose
Using locally retrieved version string from version class.
Version = GlassFish Server Open Source Edition 3.1-b43 (build 43)
asadmin Java Runtime Environment version: 1.6.0_24
Command version executed successfully.

The --local option says to report the version of the JDK from the local GlassFish installation that asadmin is running out of and not the version information from the DAS. Now run the version command one more time against the DAS to ensure secure admin is working when we connect from host1:

host1$ asadmin --host dashost version --verbose
. . .
0070: 38 67 51 0D F8 AF 3E 6F   C7 58 02 DB 3B 70 22 66  8gQ...>o.X..;p"f

Do you trust the above certificate [y|N] --> y
Version = GlassFish Server Open Source Edition 3.1 (build 43), JRE version 1.6.0_24
Command version executed successfully.

Once again you may need to accept the certificate if you are connecting from the host for the first time. Now we're ready to create and start the instance:

host1$ asadmin --host dashost create-local-instance --cluster cluster1 instance1
. . .
Command create-local-instance executed successfully.
host1$ asadmin start-local-instance instance2
. . .
Admin Port: 24849
Command start-local-instance executed successfully.

Now repeat this on host2, and your cluster is up and running.

4) Inspect your setup

Now that the cluster is up and running you can take a look at a few things:

dashost$ asadmin list-clusters
cluster1 running
dashost$ asadmin list-instances --long
NAME       HOST   PORT   PID    CLUSTER   STATE         
instance1  host1  24848  17744  cluster1   running
instance2  host2  24848  21768  cluster1   running      
Command list-instances executed successfully.
dashot$ asadmin list-nodes --long
localhost-domain1   CONFIG   localhost     /export/glassfish3
host1               CONFIG   host1         /export/glassfish3    instance1
host2               CONFIG   host2         /export/glassfish3    instance2

Note that in addition to the instances being created, a node was created for each remote GlassFish installation where create-local-instance was run from. This is a feature of create-local-instance -- it will auto-create a node for you when the first instance on the system is created.

If you rather name the nodes yourself (instead of having create-local-instance use the host name as the node name) you can run create-node-config nodename to create an empty node, and then pass this node to create-local-instance using the --node option. Create-local-instance will then populate the node with the correct information.

The third node listed, localhost-domain1, is a built-in node that is used when creating instances local to the DAS (on the same host and using the same GlassFish installation as the DAS).

5) Starting the Instances

Since the instances are remote from the DAS, and SSH is not being used, the DAS can't start the instances. That means the commands start-instance and start-cluster will not work on these instances. Instead you must always use start-local-instance directly on the instance system to start the instance.

A second option is to install the instance as a native OS service, and use the OS service framework to start and stop the instance. See the asadmin create-service command. For more information see Byron's blog on OS services and GlassFish.

Tuesday Nov 23, 2010

GlassFish 3.1: Using SSH to Manage Instance Lifecycles

GlassFish 3.1 introduces the use of SSH to manage the lifecycle of remote instances. This document gives an overview of how to use SSH with GlassFish and describes some alternatives if SSH is not practical in your environment.

1) SSH use in GlassFish 3.1

The Domain Admin Server (DAS) for GlassFish must be able to communicate with remote instances to perform various administrative operations. For most operations this communication occurs directly between the DAS and the running instance. For example when you deploy an application to an instance the DAS connects to the instance and deploys the application to it. But what about starting a remote instance? Or creating it in the first place? In these cases there is no running instance for the DAS to talk to.

In v2 these operations where handled by another process that had to be installed and running on the remote system -- the GlassFish node agent. To start a remote instance the DAS contacted the node agent, and the node agent started the instance.

In 3.1 there is no node agent. Instead the DAS uses SSH to perform certain operations on the remote system. The use of SSH is optional, and it is only needed for specific commands when those commands are used to manage remote instances (instances on systems other than the one the DAS is running on -- instances local to the DAS are supported without SSH). This table lists the commands that have a dependency on SSH when the instance is remote, and lists a command that can be used as an alternative when SSH is not available. The alternate commands are local commands that must be run directly on the system where the instance resides.

Alternate Local Command
 create-instance  create-local-instance
 delete-instance  delete-local-instance
 start-instance  start-local-instance Or use the native service support in GlassFish (create-service)
 start-cluster  start-local-instance For each instance. Or use the native service support in GlassFish (create-service)

So that's the first decision you need to make when setting up a GlassFish cluster. Do you want to leverage SSH to manage the instance lifecycle? Or do you want to avoid the dependency and perform those operations directly on the local machines yourself.

The benefit of using SSH is that, once it is set up, it makes your life easier since you can perform the above operations centrally via the DAS. The downside is that it requires some effort to get SSH set up (especially for Windows users).

Note: In GlassFish a "remote" command is a command that talks to the DAS and the DAS performs the operation. A "local" command is a command where the asadmin client performs the operation directly (and therefore the command must be run from the GlassFish installation you want to operate on). Some GlassFish "local" commands, like create-local-instance, are actually a hybrid since it needs to contact the DAS to inform it of the new instance. As you might expect the remote commands listed in the table above use SSH to execute the local version of the command directly on the remote instance.

2) What Is SSH?

If you are a Unix user you are likely already familiar with SSH, but Windows users may not be.

SSH consists of two parts: a client and a server.

The most common client is the ssh(1) command. This lets you securely log into a remote system and gives you a remote shell. Other client commands are scp and sftp which let you perform file transfers. The server is sshd or secshd and it provides the SSH service.

The GlassFish DAS uses its own Java based SSH client for communicating with the instance systems (which must be running the SSH service). But the native SSH client is still needed for things like key generation and helping to verify connectivity. In a typical GlassFish deployment the DAS acts as the SSH client, while the instance systems must be running the SSH service.

3) Getting SSH

Most all Unix based systems include SSH support. On the instance systems you need to make sure the SSH service is running (on the Mac you'll need to turn on Remote Login in the Sharing System Preferences).

On Windows you have more work to do. You'll need to choose an SSH provider and install it on all instance systems and the DAS. GlassFish has been tested with Cygwin and MKS:

Product Version Tested
Cygwin  1.7.6 Installing Windows Cygwin sshd
 MKS Toolkit for Developers

Once you install your SSH provider, make sure that the provider's "bin" directory is in your (Windows) PATH so that you have easy access to the SSH client commands (ssh, sftp, ssh-keygen, etc).

Once you have SSH installed and the SSH service running, verify you can login from the DAS host to the instance systems using the ssh(1) client: ssh It should prompt for your password (unless you already have key authentication set up). If you can't ssh into the remote system then you'll have to sort that out before using SSH with GlassFish.

Note: Yamini K B has a nice blog entry (Using GlassFish v3.1 SSH Provisioning Commands) that does a good job summarizing what needs to be done to get SSH working on different operating systems.

4) Mixing Different OS Types

The DAS and all GlassFish instances in a cluster must be running on the same OS type and version. Running a cluster of mixed OS types is not supported. In practice mixing different flavors of Unix usually works OK (but is not officially supported), but mixing Windows and Unix is asking for trouble.

5) SSH Authentication

When the SSH client connects to the SSH service it needs to authenticate the user. The most basic form of authentication is password authentication. SSH also supports other forms of authentication. In particular it support public key authentication.

GlassFish 3.1 supports 3 forms of SSH authentication. They each have trade-offs:

 Authentication Scheme
Pro Con
 Username / Password
 Easy to configure in SSH.
A bit more work configuring GlassFish to use this in a secure fasion.
 Public Key (unencrypted)
 Easy to configure GlassFish to use.
Can be trickier to get configured correctly in SSH.
 Public Key (encrypted)
More secure than unencrypted public key.
More work to configure both SSH and GlassFish to use this.

To use Username/Password securely with GlassFish you will need to use a GlassFish feature called "password aliases" so that your SSH password is not stored in the clear. This isn't too bad, but does require a little extra fiddling to create and use the password alias.

The unencrypted public key approach requires that you generate an SSH key pair and then copy the public key to the remote systems. GlassFish provides an asadmin command, setup-ssh, to help you do this. Since there is no password you don't need to mess with password aliases, and the DAS looks in the default places to locate the key to authenticate with -- so you often don't even need to provide that info. But folks sometimes run into trouble getting key authentication working with SSH. It usually works great, but if it doesn't it can be tricky to figure out why.

In the encrypted key file approach you encrypt your private key file with a passphrase that must be provided to use the private key. This means you must configure GlassFish to know about the passphrase -- which means using password aliases again. So this method combines the complexity of the first two approaches. You probably won't choose this approach unless it is the preferred SSH authentication scheme at your site.

If you already have SSH deployed at your site you may have the preferred authentication scheme already defined for you.

 At this point you should decide what form of authentication you want to use.

6) What user?

One basic decision you will need to make is concerning what SSH user you want GlassFish to use when it connects to remote systems. Some considerations:

  1. For key authentication the user that the DAS process is running as must be able to read the SSH user's private key file (typically in ~/.ssh).
  2. Remote instances will be started, and therefore be running as, the SSH user.
  3. By default the DAS will assume the SSH user is the same as the user running the DAS process.
So it ends up being easiest if the SSH user is the same as the user that starts the DAS. This doesn't have to be the case -- it just makes things simpler.

7) Setting Up SSH Key Authentication

GlassFish provides a command, setup-ssh, that sets up SSH key authentication for you. You run this command directly on the DAS system. Here is an example run. In this example adc2101159 is the name of the remote host you will be running an instance on, and we assume the user running the command (dipol) is the same user that will start the DAS and the same user that we want GlassFish to use when connecting to remote systems over SSH:

  $ asadmin setup-ssh adc2101159
  SSH key not found for user dipol
  Would you like to generate an SSH key pair (without a key passphrase) for dipol to access [adc2101159]? [y/n]: y
  Enter SSH password for dipol@adc2101159
  Created directory /export/home/dipol/.ssh
  /usr/bin/ssh-keygen successfully generated the identification /export/home/dipol/.ssh/id_rsa
  Copied keyfile /export/home/dipol/.ssh/ to dipol@adc2101159
  Successfully connected to dipol@adc2101159 using keyfile /export/home/dipol/.ssh/id_rsa
  Command setup-ssh executed successfully.

As you can see setup-ssh does a couple things:

  1. It checks to see if you have an SSH key already generated. If you don't it gives you the option to generate one.
  2. After generating the key it copies it to the remote host. To do this it needs your SSH password to make the connection to copy the key.
  3. It then connects to the remote system to verify that key authentication is working

After key authentication is set up you can re-run setup-ssh any time and it will detect the key and verify the connection with it:

  $ asadmin setup-ssh adc2101159
  Successfully connected to dipol@adc2101159 using keyfile /export/home/dipol/.ssh/id_rsa
  SSH public key authentication is already configured for dipol@adc2101159
  Command setup-ssh executed successfully.

If setup-ssh fails you need to decide if you want to debug why key authentication is failing, our give up and use password authentication. A couple things to check if key authentication fails:

  1. Make sure on the remote system that the file system permissions for the SSH user are not too permissive. The SSH user's home directory, ~/.ssh directory, and authorized_key file must be writeable only by the SSH user. If they are writeable by others then the secure service may not trust the authorized_key file and may reject public key authentication.
  2. On MKS we have run into a case where fixing #1 required:
    1. Copying the .ssh\\id_rsa key file over to the remote system manually and importing it with the MKS configuration GUI (Passwordless button under Secure Service tab) or
    2. Using the MKS configuration GUI to turn off "enforce strict mode".

8) Installing GlassFish

GlassFIsh must be installed on each system that will host an instance. You have two choices on how to do this:

  1. Install manually on each system
  2. Install manually on the DAS system, then use the install-node command to create a GlassFish install image based on the DAS install and then install that image on the remote instance systems.

Option 1 is best if you want complete control over the installation and install time settings. Option 2 works well if all of your systems are set up the same and you want to duplicate the DAS GlassFish installation.

Here is an example of an install-node run. This is run on the DAS system:

  asadmin install-node --installdir /export/tmp/glassfish3 adc2101159
  Created installation zip /export2/glassfish3/
  Copying /export2/glassfish3/ (83527743 bytes) to adc2101159:/export/tmp/glassfish3 
  Installing /export2/glassfish3/ into adc2101159:/export/tmp/glassfish3
  Removing adc2101159:/export/tmp/glassfish3/
  Fixing file permissions of all files under adc2101159:/export/tmp/glassfish3/bin
  Command install-node executed successfully.

Note: Currently install-node works best with SSH key authentication.

9) Nodes and Instances

In GlassFish 3.1 a Node is a configuration object in the DAS that represents a GlassFish installation on a host. There are two types of nodes:

  1. SSH nodes
  2. CONFIG nodes

Both node types contain some basic information about the node like:

  • The hostname of the remote host
  • The location of the GlassFish installation on the remote host

An SSH node also contains information needed to connect to the host via SSH such as the SSH username to use, the SSH port number, etc.

The DAS ships with one node already created named "localhost-<domainname>". This is a CONFIG node that can be used when creating instances local to the DAS. See the next section for more info on creating instances local to the DAS. CONFIG nodes are also used with remote instances when SSH is not going to be used. See section 11 for more info on that. But if you want to manage a remote instance via SSH then you need to first create an SSH node to represent a GlassFish installation on the remote system.

You use the create-node-ssh command to create SSH nodes. If you have SSH key authentication already set up (section 7) and the DAS is running as the sshuser (section 6), then things are really easy:

  $ asadmin create-node-ssh --nodehost adc2101159 --installdir /export/glassfish3 node1
  Command create-node-ssh executed successfully.

You can use list-nodes to see what nodes are defined:

  $ asadmin list-nodes --long
  localhost   CONFIG  localhost   /export2/glassfish3   
  node1       SSH     adc2101159  /export/glassfish3
  Command list-nodes executed successfully.

If you are using SSH password authentication then you need to do a little more work to securely set the SSH password on the SSH node. 3.1 SSH Authentication has some more details on running create-node-ssh with other SSH authentication schemes.

create-node-ssh will validate the parameters you pass it. For example it makes sure it can connect to the remote host using SSH, and it verifies that the installdir you pass it is a GlassFish install. Therefore you should have SSH set up and GlassFish installed on the remote instances before you start creating SSH nodes.

Once you have the SSH node created, you can then create an instance on the node:

  $ asadmin create-instance --node node1 instance1
  Command _create-instance-filesystem executed successfully.
  Port Assignments for server instance instance1:
  The instance, instance1, was created on host adc2101159
  Command create-instance executed successfully.

And start it:

  $ ./asadmin start-instance instance1
  Waiting for instance1 to start ......
  Successfully started the instance: instance1
  instance Location: /export/glassfish3/glassfish/nodes/node1/instance1
  Log File: /export/glassfish3/glassfish/nodes/node1/instance1/logs/server.log
  Admin Port: 24848
  Command start-local-instance executed successfully.
  The instance, instance1, was started on host adc2101159
  Command start-instance executed successfully.

If at any time you want to test the connection to an SSH node you can do it using the asadmin ping-node-ssh command:

 $ ./asadmin ping-node-ssh node1
Successfully made SSH connection to node node1 (adc2101159)
Command ping-node-ssh executed successfully.

10) Local Instances

To create instances on the same system as the DAS (typically using the same GlassFish installation as the DAS) you do not need SSH and you don't need to worry about nodes at all. Run this command from the same installation as the DAS:

  $ ./asadmin create-local-instance localinstance1
  Port Assignments for server instance localinstance1:
  The instance, localinstance1, was created on host localhost
  Command create-local-instance executed successfully.
This creates an instance local to the DAS using the build in localhost node.

11) Doing Remote Instances Without SSH 

If you do not want to use SSH for managing remote instances you can still create and manage them using the local version of asadmin commands as mentioned in section 1. Also, due to an "auto-node creation" feature you don't have to  explicitly create the node before hand. For example if you have this setup:

  • GlassFish installation on two systems: systemA and systemB
  • DAS running on systemA

Then you can create an instance on systemB by doing the following:

Log into systemB and run the following using the GlassFish installation on systemB:

  $ asadmin --host systemA create-local-instance myinstance1
  Rendezvoused with DAS on systemA:4848.
  Port Assignments for server instance myinstance1:
  Command create-local-instance executed successfully.
Note: Since create-local-instance connects to the DAS from a remote system, the DAS must be running with secure admin enabled. See the asadmin enable-secure-admin command -- you will need to restart the DAS after enabling secure admin.

This command does the following:

  1. Creates a CONFIG node in the DAS configuration for systemB if such a node does not already exist. The node name is the hostname of systemB.
  2. Creates a server instance object in the DAS configuration for instance myinstance1
  3. Creates the instance files in the filesystem on systemB for instance myinstance1

You can then start the instance while still logged into systemB:

  $ asadmin start-local-instance myinstance1
  Waiting for myinstance1 to start ......
  Successfully started the instance: myinstance1
  instance Location: /export/glassfish3/glassfish/nodes/adc2101159/myinstance1
  Log File: /export/glassfish3/glassfish/nodes/adc2101159/myinstance1/logs/server.log
  Admin Port: 24849
  Command start-local-instance executed successfully. 

As mentioned in section 1 if you are managing remote instances without SSH then you will not be able to use start-cluster to start instances in a cluster. You will need to use start-local-instance on each instance or use GlassFish's native service support to start the instances.


Joe Di Pol-Oracle


« July 2016