Thursday Dec 31, 2015

ZDT Rollouts and Singletons

WLS offers messaging, transaction and other system services to facilitate building enterprise grade applications. Typically, services can be either clustered or singleton. Clustered services are available on all nodes of a cluster for example sessions are replicated on secondary node. Whereas singleton services are different in that they don't have any replicated state. Rather, they are services that run on only one server in a cluster at any given point of time so as to offer specific quality of service (QOS) but most importantly to preserve data consistency. Singleton services can be JMS-related, JTA-related or user-defined. During ZDT rollout, servers are restarted in a rolling manner. Taking down a server would bring down the singleton service(s) thus causing service disruptions, services will not be available until the server starts back up. The actual down time would vary depending on server start up time and number of applications deployed. Hence, to ensure that singleton services do not introduce a single point of failure for dependent applications in the cluster, ZDT rollout process automatically performs migrations.[Read More]

Monday Feb 28, 2011

Troubleshooting SSH setup in GlassFish v3.1

Troubleshooting SSH setup in GlassFishv3.1 Centralized administration feature in GlassFish v3.1 uses SSH for provisioning GlassFish nodes and managing cluster (or instance) lifecycle operations securely. Even before DAS can use SSH for communicating with remote hosts, SSH needs to be configured properly. Since SSH server settings vary on different platforms (and even across different flavors of Unix), GlassFish SSH setup command, setup-ssh may fail at times. In this blog, I've tried to capture such failure cases as well as solutions on how to address them.

Before getting into the setup-ssh command, its important to understand SSH authentication schemes. Please check this blog on GlassFish SSH authentication schemes.
The asadmin setup-ssh command uses password authentication to copy over the specified (or generated) public key to the remote host. Before exiting, the command also verifies that SSH public key authentication works as expected.

Please note that all of the SSH server settings mentioned below pertain to the settings on the remote host since DAS host is actually the SSH client. SSH server settings is configured through the SSH server configuration file located as follows:

Unix: /etc/ssh/sshd_config or /etc/sshd_config
Windows with Cygwin: From within cygwin terminal, /etc/sshd_config or actual path on file system would be <Cywin-install-dir>/etc/sshd_config
Windows with MKS Toolkit: Programs -> MKS Toolkit -> Configuration -> Configuration Information.

Also note that SSH server needs to be restarted whenever any of the server attributes are changed.
  • Password authentication error
$ asadmin setup-ssh sunone172
Enter SSH password for yamini@sunone172>
SSH key setup failed: SSH password authentication failed for user yamini on host sunone172
Command setup-ssh failed.

Reason: SSH user password entered is incorrect or SSH server is rejecting password authentication.

Solution: Change the 'PasswordAuthentication' setting to 'yes' in SSH server config and restart the SSH server.

Explanation: Assuming that you are typing in the correct password, first check if manual ssh works:

$ ssh sunone172
Permission denied (publickey).

If you see something like above, it means that SSH server is not even considering password authentication.

It can also be possible that password authentication appears to be working manually:

$ ssh sunone172 pwd

But there is a catch here, the above appears to be using password authentication but its actually using keyboard-interactive authentication. The point is password authentication is different from keyboard-interactive. In the former case, the (SSH) client reads in the password from user and sends it across to the SSH server whereas in the latter case, the password is challenged from the SSH server. A subtle difference is in the way the prompting happens. Also notice the last debug message from 'ssh -v':

debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: password
yamini@sunone172's password:

First three prompts are from the SSH server whereas the last one is from the SSH client.

The attribute that turns on/off keyboard-interactive is 'ChallengeResponseAuthentication' attribute in SSH server configuration. Here is a nice explanation on the difference between password and keyboard-interactive.

In short, it doesn't matter if keyboard-interactive is turned on or not, what matters for us is the password authentication setting.
  • Connection verification failure
$ asadmin setup-ssh sunone172
SSH key not found for user yamini
Would you like to generate a SSH key pair (without a key passphrase) for yamini to access [sunone172]? [y/n]: y
Enter SSH password for yamini@sunone172>
Created directory /home/yamini/.ssh
/usr/bin/ssh-keygen successfully generated the identification /home/yamini/.ssh/id_rsa
Copied keyfile /home/yamini/.ssh/ to yamini@sunone172
Connection verification failed.
Command setup-ssh failed.

Reason 1: Public key authentication is disabled on SSH server.

Solution: Change the attribute 'PubkeyAuthentication' to 'yes' in the SSH server configuration file and restart the SSH server.

Reason 2: On remote host, permissions on $HOME or .ssh or authorized_keys file is too loose.

Solution: Either disable strict checking of file permissions by setting 'StrictModes' to 'no' in SSH server config file and restart the SSH server.
          Or make sure that the following permissions are in place:
          $HOME --> 755
          $HOME/.ssh --> 700
          $HOME/.ssh/authorized_keys --> 600 or 644

Thursday Dec 30, 2010

Using GlassFish v3.1 SSH Provisioning Commands

SSH provisioning commands in GlassFish v3.1 are used for setting up hosts that will be used for installing and managing GlassFish instances. Previously, in GlassFish v2.x, administrator would explicitly perform the installations on hosts either manually or through shell scripts and node agent would handle some part of the provisioning (like creating instance) With the introduction of SSH in GlassFish v3.1 as the key communication protocol, GlassFish clusters are now completely secure.[Read More]

Wednesday Dec 29, 2010

Ubuntu upgrade from 10.04 to 10.10

Finally upgraded my laptop OS to Ubuntu 10.10! Was surprised to see the 'About Ubuntu' screen (first line), however checking the version using CLI shows 10.10 :)

About Ubuntu

Thursday Oct 29, 2009

Diameter Administration on SailFIn 2.0

Diameter addon is a value add feature available only to select customers. This blog is an introduction on how to administer Diameter configuration on SailFin 2.0 [Read More]

Thursday Aug 14, 2008

Call Flow FAQ

Here are some answers to common queries related to SIP call flow monitoring..[Read More]

Wednesday Apr 23, 2008

SailFin Monitoring

Are you interested in knowing how many of the SIP dialogs were successful or how many failed or expired. Enter SIP monitoring![Read More]

Tuesday Mar 18, 2008

Call Flow Monitoring in SailFin

SailFin supports call flow monitoring which is a feature inherited from GlassFish. This feature can be used to debug the request call flow in the application server. In case of SailFin, you will now be able to see how much time a SIP request spends in the SIP container.

Here, I shall demonstrate how to use this feature.

To start with, you will need to enable call flow on a running server instance.

1. Start the DAS

2. Login to administration console (point your browser to http://localhost:4848)

3. In the Common Tasks page, click on 'View Monitoring Data'

4. Then click on 'Call Flow' tab

5. Click on 'Enabled', then 'Save'

You will now see a message like the following in server.log:

[#|2008-03-07T13:37:57.313+0530|INFO|sun-comms-appserver1.0||_ThreadID=16;_ThreadName=httpWorkerThread-4848-1;|ADM11009: CallFlow enable successful.|#]

I'll use the 'subscribe' test from the quicklook tests of the SailFin tests suite. To set up the test environment, do the following:

1. Checkout sailfin-tests

cd <workspace>/sailfin

cvs up -d sailfin-tests

2. Edit proxy host/port in Set environment variables, SF_HOME, ANT_HOME and PATH. Invoke ant

ant setup

3. cd quicklook/publish

4. Run the test

ant all

Now, go back to admin console and click on 'Refresh' to view the call flow data. Here is a sample screenshot.

Tuesday Nov 20, 2007

SailFin Build and Testing Quick Help

If you are looking for some quick reference on how to build and test SailFin, read on..[Read More]

Friday Dec 01, 2006

Auto completing wadm commands

Taking a cue from a blog written by my friend Harsha, I thought why not try the same for wadm, the new CLI in Web Server 7.0 Now that wadm has some 300+ commands to perform various administrative tasks, auto completing (or tab completion) commands is a great help!

[Read More]

Friday Nov 17, 2006

Integrating NetBeans Profiler with Web Server 7.0

While looking into some performance issues, I needed to profile Web Server 7.0 I have used OptimizeIt and Jprobe before but NetBeans had just released its new version of profiler and I thought I'll give it a try and was amazed with the new NetBeans profiler. I've used both 5.0 and 5.5 and am looking forward to the next release 6.0. For now, you can download NetBeans 5.5 from here. Also check out the profiler web site for tutorials, documentation, FAQs, blogs.[Read More]

Wednesday May 24, 2006

Silent Administration Agent Installation in Web Server 7.0

Using the silent installer of Web Server 7.0 does a quick and neat job. This is particularly useful if you have to install on multiple machines to configure a cluster.[Read More]

Thursday May 18, 2006

Starting off with Web Server 7.0 CLI

Web Server 7.0 comes with a new Command Line Interface called wadm for administering the web server. The best part is that its simple to use and comes with a set of man pages for every available command. wadm can be used locally or remotely.[Read More]



« July 2016