Using GlassFish v3.1 SSH Provisioning Commands

Using GlassFish SSH Provisioning Commands

Introduction

SSH provisioning commands in GlassFish v3.1 are used for setting up hosts that will be used for installing and managing GlassFish instances. Previously, in GlassFish v2.x, administrator would explicitly perform the installations on hosts either manually or through shell scripts and node agent would handle some part of the provisioning (like creating instance) With the introduction of SSH in GlassFish v3.1 as the key communication protocol, GlassFish clusters are now completely secure. Right from installing the software, life cycle of instances can be managed securely.

GlassFish SSH provisioning requires some pre requisites mainly due to the fact that SSH configuration and user management differs based on OS (Unix, Windows) Though it may appear that SSH configuration is complex, it is simple if the correct steps are followed. But once setup, things work like a charm!

GlassFish v3.1 provisioning mainly consists of 5 commands, setup-ssh, install-node, uninstall-node, create-node-ssh, delete-node-ssh. This blog covers each of these commands in detail along with trouble shooting tips and solutions.

SSH pre requisites

There are certain prerequisites before actually getting down to using these commands. Since user running DAS will act as the SSH client, most of these prerequisites apply to remote hosts which run the SSH server (sshd)

Pre requisite
Unix
Windows Cygwin
Windows MKS
Comments
OpenSSH should be installed on all machines hosting GlassFish software SSH utils/packages are usually installed as part of base OS. If not, install the corresponding OpenSSH ssh package for your OS Install OpenSSH that comes along with Cygwin (1.7.6 and above)
Cygwin installation instructions
Install OpenSSH that comes along with MKS Toolkit (9.2 and above)
For MKS installation, get software from http://www.mkssoftware.com/ and follow their instructions

Ensure SSH server (sshd) is running on remote hosts. DAS host will act as the SSH client, so you don't actually need SSH server running on DAS Solaris:
$ svcs status ssh
STATE          STIME    FMRI
online         Dec_14   svc:/network/ssh:default
online         14:10:10 svc:/network/nfs/status:default


Red Hat Linux:
$ service ssh status
 \* sshd is running


Any other flavor of Unix:
Run 'ps -ef|grep sshd' to figure out if SSH server is running
$ cygrunsrv --query sshd
Service             : sshd
Display name        : CYGWIN sshd
Current State       : Running
Controls Accepted   : Stop
Command             : /usr/sbin/sshd -D
$ service query MKSSecureSH
Name:           MKS Secure Shell Service
Service Type:   WIN32_OWN_PROCESS
Current State:  RUNNING
Controls Accepted:      ACCEPT_STOP
Check Point:    0
Wait Hint:      0
Start Type:     AUTO_START
Error Control:  IGNORE
Path:           "C:\\Program Files\\MKS Toolkit\\bin\\secshd.exe"
Dependency:     NuTCRACKERService
Dependency:     tcpip
Service Start Name:     LocalSystem

Cygwin/MKS user home directory should be same as Windows user home directory. If not, there might be errors while running some of the GlassFish commands. Not applicable
Login as Administrator and modify /etc/passwd to edit home directory setting for the SSH user.

(Not sure if there is a better way to do this)
Follow tip #1 in MKS SSHD tips

PATH variable should be correctly configured
Does not require any explicit setting
PATH should have Cygwin bin directory as well as JDK bin directory. MKS bin directory would be automatically added in PATH by the MKS installer but JDK bin directory would need to be explicitly added. It is best to enable this setting as Administrator user so that it applies to all users.
Proper file/directory permissions. Incorrect permissions will cause authentication failures since SSH server may not be able to read/access the SSH config files.

User home directory: 755
.ssh directory: 700
authorized_keys file: 644
Use chmod command
Use chmod command
Use chmod command.
chmod on Windows work differently since Windows files do not have the same attributes as Unix/Posix
If a key is manually generated using ssh-keygen and Windows user home and Cygwin (or MKS) user home is different, then the key should be copied into Windows user home's .ssh directory. Otherwise it will cause SSH authentication within GlassFish to fail (ex create/delete-node-ssh) since GlassFish will look for key under Windows user home .ssh directory. Not applicable
Verify Windows user home .ssh folder's contents.
Verify Windows user home .ssh folder's contents.
SSHD configuration
Settings can be found in /etc/sshd_config or /etc/ssh/sshd_config

StrictModes yes
PubkeyAuthentication yes

Settings can be found in /etc/ssh_config (/cygdrive/c/cygwin/etc/sshd_config)

StrictModes yes
PubkeyAuthentication yes
Programs -> MKS Toolkit -> Configuration -> Configuration Information

Strict mode: Click on "Secure Shell Service" tab. Go to "Advanced" setting, click on "Login" tab, uncheck "Strict Modes"
Password auth: Click on "Secure Shell Service" tab and enable "Password Authentication"

In most cases, the default settings should work.

Also, some settings require a server restart.

Please see here on how to use the various SSH authentication schemes in GlassFish.

setup-ssh

Purpose: To setup public key authentication between DAS host and instance host(s). You don't need to use this command if you intend to use password authentication for SSH

The command first checks if a key is available. It also checks if the key is encrypted. It could also be possible that the SSH password or key passphrase is stored in aliased form in the domain key store. Such an alias is generally passed to the command in non-interactive mode using the --passwordfile option.

Unix: Command generates key pair as well as propagates it to host.
$ asadmin setup-ssh --sshuser yb113654 caitanya.india.sun.com
SSH key not found for user yb113654
Would you like to generate a SSH key pair (without a key passphrase) for yb113654 to access [caitanya.india.sun.com]? [y/n]: y
Enter SSH password for yb113654@caitanya.india.sun.com>
/usr/bin/ssh-keygen successfully generated the identification /home/yamini/.ssh/id_rsa
Copied keyfile /home/yamini/.ssh/id_rsa.pub to yb113654@caitanya.india.sun.com
Successfully connected to yb113654@caitanya.india.sun.com using keyfile /home/yamini/.ssh/id_rsa
Command setup-ssh executed successfully.

Windows Cygwin: Key already exists in following case. So the command just copies the key to remote host.
$ asadmin setup-ssh underpass.india.sun.com
Enter SSH password for Yamini@underpass.india.sun.com>
Copied keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa.pub to Yamini@underpass.india.sun.com
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Command setup-ssh executed successfully.

Windows Cygwin: This example demonstrates the key setup when Windows user home and Cygwin user home is not same. Note that it requires some extra steps. Hence, it is best to keep the home directories same.
$ rm -rf ~/.ssh

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ rm -rf /cygdrive/c/Documents\\ and\\ Settings/Yamini/.ssh

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin setup-ssh underpass.india.sun.com
SSH key not found for user Yamini
Would you like to generate a SSH key pair (without a key passphrase) for Yamini to access [underpass.india.sun.com]? [y/n]: y
Enter SSH password for Yamini@underpass.india.sun.com>
Created directory C:\\Documents and Settings\\Yamini\\.ssh
ssh-keygen successfully generated the identification C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Copied keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa.pub to Yamini@underpass.india.sun.com
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Command setup-ssh executed successfully.
Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ ls ~/.ssh
ls: cannot access /home/Yamini/.ssh: No such file or directory

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ ls /cygdrive/c/Documents\\ and\\ Settings/Yamini/.ssh
id_rsa  id_rsa.pub

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ cp -r /cygdrive/c/Documents\\ and\\ Settings/Yamini/.ssh ~/.ssh

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$

Tip: If you want to run ssh commands manually and Windows home is different from Cygwin user home, the key needs to be copied manually to Cygwin user home (see last command in above example). This is because ssh commands will search for key in Cygwin user home .ssh directory.

Windows Cygwin: Non-interactive mode of key generation using clear text SSH password.
C:\\cygwin\\home\\Yamini\\gf\\glassfish3\\glassfish\\bin>echo AS_ADMIN_SSHPASSWORD=ssh-user-password >/tmp/pass.txt
C:\\cygwin\\home\\Yamini\\gf\\glassfish3\\glassfish\\bin>asadmin --passwordfile ./pass.txt setup-ssh --sshuser Yamini --generatekey underpass.india.sun.com
Copied keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa.pub to Yamini@underpass.india.sun.com
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Command setup-ssh executed successfully.


install-node

Purpose:  To install GlassFish software on remote host(s). Either password or key auth (with/without passphrase) can be used for authentication.

The command first tries to use key authentication  and checks if a key is available. In case key authentication fails, it falls back to using password authentication. It also checks if the key is encrypted. It could also be possible that the SSH password or key passphrase is stored in aliased form in the domain key store. Such an alias is generally passed to the command in non-interactive mode using the --passwordfile option.

Windows Cygwin: Using clear text key passphrase.
$ echo AS_ADMIN_SSHKEYPASSPHRASE=foo123>pass1.txt

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin --passwordfile pass.txt install-node --installdir /home/Yamini/gf underpass.india.sun.com
Created installation zip C:\\cygwin\\home\\Yamini\\gf\\glassfish3\\glassfish\\bin\\glassfish6762824754722728821.zip
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Copying C:\\cygwin\\home\\Yamini\\gf\\glassfish3\\glassfish\\bin\\glassfish6762824754722728821.zip (81745171 bytes) to underpass.india.sun.com:/home/Yamini/gf
Installing glassfish6762824754722728821.zip into underpass.india.sun.com:/home/Yamini/gf
Removing underpass.india.sun.com:/home/Yamini/gf/glassfish6762824754722728821.zip
Fixing file permissions of all files under underpass.india.sun.com:/home/Yamini/gf/bin
Command install-node executed successfully.

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$

Windows Cygwin: Using key passphrase alias.
$ asadmin list-password-aliases
Nothing to list
Command list-password-aliases executed successfully.

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin create-password-alias
Enter the value for the aliasname operand> ssh-keypass
Enter the alias password>
Enter the alias password again>
Command create-password-alias executed successfully.

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin list-password-aliases
ssh-keypass
Command list-password-aliases executed successfully.

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ echo AS_ADMIN_SSHKEYPASSPHRASE=\\$\\{ALIAS=ssh-keypass\\}>pass2.txt

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ cat pass2.txt
AS_ADMIN_SSHKEYPASSPHRASE=${ALIAS=ssh-keypass}

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin --passwordfile pass2.txt install-node --installdir /home/Yamini/gf underpass.india.sun.com
Password is aliased. To obtain the real password, enter master password for domain1's key store>
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Created installation zip C:\\cygwin\\home\\Yamini\\gf\\glassfish3\\glassfish\\bin\\glassfish2993564184607483536.zip
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
GlassFish is already installed on underpass.india.sun.com under /home/Yamini/gf.

Command install-node executed successfully.


create-node-ssh

Purpose: creates a node for a specific host with the GlassFish software installation identified by an installation path. Also contains the connection information for the node.

Windows Cygwin: create-node-ssh will fail if it cannot find a valid installation on remote host.

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin create-node-ssh --nodehost underpass.india.sun.com --sshuser Yamini --installdir c:/home/Yamini/gf/glassfish3 n1
remote failure: Warning: some parameters appear to be invalid.
SSH node not created. To force creation of the node with these parameters rerun the command using the --force option.
Invalid install directory: could not find c:/home/Yamini/gf/glassfish3/glassfish/modules/admin-cli.jar on underpass.india.sun.com
Command create-node-ssh failed.

Using create-node-ssh --install to provision the remote host. This is the same as running install-node first and then create-node-ssh

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin create-node-ssh --nodehost underpass.india.sun.com --installdir /home/Yamini/gf/glassfish3 --install n1
Successfully installed GlassFish on underpass.india.sun.com.
Command create-node-ssh executed successfully.

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin list-nodes
localhost-domain1  CONFIG  localhost
n1  SSH  underpass.india.sun.com
Command list-nodes executed successfully.

Note: In case public key authentication is preferred, setup-ssh should be run prior to create-node-ssh, to setup the communication.

delete-node-ssh

Windows cygwin: Deleting the node configuration along with uninstalling the software.
Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin delete-node-ssh --uninstall n1
Successfully un-installed GlassFish on underpass.india.sun.com.
Command delete-node-ssh executed successfully.

uninstall-node

Purpose:  To un-install GlassFish software on remote host(s). Either password or key auth (with/without passphrase) can be used for authentication.

The command first tries to use key authentication  and checks if a key is available. In case key authentication fails, it falls back to using password authentication. It also checks if the key is encrypted. It could also be possible that the SSH password or key passphrase is stored in aliased form in the domain key store. Such an alias is generally passed to the command in non-interactive mode using the --passwordfile option.

Windows Cygwin: Non-interactive and aliased key passphrase.
Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ echo AS_ADMIN_MASTERPASSWORD=changeit>>pass2.txt

Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin --passwordfile pass2.txt uninstall-node --installdir /home/Yamini/gf underpass.india.sun.com
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Command uninstall-node executed successfully.

Solaris: Interactive mode and aliased key passphrase.
$ asadmin uninstall-node --sshuser yamini1 --installdir /space/yamini1/gf/glassfish3 sunone172.india.sun.com
Key /home/yamini/.ssh/id_rsa is encrypted
Enter key passphrase>
Enter SSH password for yamini1@sunone172.india.sun.com>
Authenticating with password <concealed>
Command uninstall-node executed successfully.

Note: setup-ssh, install-node, uninstall-node are local commands whereas create-node-ssh is a remote command.

Troubleshooting:

  • uninstall-node fails with sftp error.
Yamini@ducati ~/gf/glassfish3/glassfish/bin
$ asadmin --passwordfile pass2.txt uninstall-node --installdir /home/Yamini/gf/glassfish3 underpass.india.sun.com
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
Successfully connected to Yamini@underpass.india.sun.com using keyfile C:\\Documents and Settings\\Yamini\\.ssh\\id_rsa
com.trilead.ssh2.SFTPException: Failure (SSH_FX_FAILURE: An error occurred, but no specific error code exists to describe the failure.)
Command uninstall-node failed.

Reason: One of the directories within installdir is in use on remote host, hence removal fails.

Solution: Make sure installdir (and its children) are not in use before uninstalling.

  • setup-ssh fails on Windows MKS
[C:/yamini/gf/glassfish3/glassfish/bin]  asadmin setup-ssh crux.india.sun.com
Key C:\\Documents and Settings\\Administrator\\.ssh\\id_rsa is encrypted
Enter key passphrase>
Enter SSH password for Yamini@crux.india.sun.com>
SSH key setup failed: SSH password authentication failed for user Yamini on host underpass.india.sun.com
Command setup-ssh failed.

Reason: Password authentication is disabled in MKS ssh server on remote host.

Solution: Enable password authentication using MKS configuration tool. Note that enabling/disabling password authentication requires MKS SSH server restart.

  • setup-ssh fails during connection verification
$ asadmin setup-ssh --sshuser yamini1 sunone172.india.sun.com
Key /home/yamini/.ssh/id_rsa is encrypted
Enter key passphrase>
Enter SSH password for yamini1@sunone172.india.sun.com>
Copied keyfile /home/yamini/.ssh/id_rsa.pub to yamini1@sunone172.india.sun.com
Connection verification failed.
Command setup-ssh failed.

Reason: authorized_keys file permissions are too loose on remote host.

Solution: Follow step 5 in pre requisites section above.


Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

yamini

Search

Top Tags
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today