Securing Administration in GlassFish Server 3.1 - 3: Admin Message Encryption

If you are an administrator of a GlassFish installation, you will see
some new facets of administrative security in GlassFish Server 3.1.  In
this series of blogs I describe three key aspects: admin
authentication, remote administration, and admin message encryption. 
Some of this will be familiar to you if you have worked with earlier
releases of GlassFish, and some will be new.


 Remember that convenience and security are often at odds with each
other. We have tried hard to keep the out-of-the-box experience with
GlassFish Server 3.1 convenient while not creating security problems.
And we have tried to make it very easy to run in a very secure
administrative environment. 


See also:


Part 1: Admin Authentication


Part 2: Remote Administration


Part 3: Admin Message Encryption


When you install GlassFish Server or create a new domain, GlassFish does not encrypt admin message traffic. This means that admin messages could be intercepted, spoofed, or falsified.  To guard against this, when you enable secure admin using


asadmin enable-secure-admin


GlassFish encrypts all admin traffic using SSL/TLS. (As described in earlier parts of this blog series, before you run enable-secure-admin shut down all instances, leaving only the DAS running.  Then after you enable secure admin you can restart the DAS and start up any instances you want.)  Remember also that after you run the enable-secure-admin command GlassFish allows remote administration.


Turn off encryption using


asadmin disable-secure-admin


which also disables remote administration.


Several things work differently when you encrypt admin traffic.


Default Self-signed Certificates


GlassFish Server creates two self-signed certificates when you install it or create a new domain. 


One, as in past releases of GlassFish, is used in the DAS for SSL server authentication to clients. You will notice this the first time you use asadmin or some other admin client to contact the DAS.  Because the cert is self-signed, there is no trusted certificate chain to a certificate authority.  Because of this, admin clients such as asadmin will display the cert and ask you if you want to trust it. Once you accept the DAS cert from a given domain asadmin remembers it and will not ask you about that cert again. 


This cert is also used in GlassServer 3.1 to authenticate the DAS to remote instances so those instances can trust that the admin messages actually come from the DAS.


The second self-signed cert which GlassFish generates is used by the instances for SSL server authentication to clients. You will rarely see evidence of this cert yourself, although if you use the monitoring add-on (which allows you to connect directly to instances to retrieve monitoring information) asadmin will ask you whether you trust that cert. 


Although it's more than we want to talk about here, you can use your own certificates and private keys instead of the generated, self-signed ones. If your certs are from a certificate authority then they will be trusted automatically and admin clients will not ask you if you want to trust them.


Certificates and Authentication


How does this help with security?  With secure admin disabled, the DAS insists that you log in using valid administrator credentials. But you do not really know that it's the DAS you are talking to. Someone could have started a different process at the admin port (by default, 4848) to capture your admin username and password for illicit use later. But when the server uses SSL certificate authentication, you can be confident that you are actually talking to the DAS - if you trust the cert, that is. This is why clients prompt users about self-signed certs.  Now, you know you are talking to the DAS - thanks to the SSL server authentication - and the DAS knows you are a valid administrator - thanks to the admin user and password you provide.


Beyond that, the DAS and the instances use SSL mutual authentication. 
That is, the DAS uses its certificate to authenticate itself to the
instances, and the instances use their certificate to authenticate
themselves to the DAS.   This way the DAS and the instances have
confidence that the messages have not been compromised in flight -
either intercepted or spoofed - and that the sender and receiver of the
admin messages are who they claim to be.


SSL and Encryption


When you enable secure admin, GlassFish Server encrypts all admin messages among admin clients, the DAS, and the instances. Notably, it encrypts the username and password along with the rest of the message so no one who might be monitoring network traffic can intercept your credentials and use them to sign in illicitly to the DAS.


In fact, once you enable secure admin, even if you try to connect to the DAS over an open (HTTP) connection the DAS redirects the request to HTTPS.  Plus, the asadmin utility will not send the username and password over an insecure connection if secure admin is enabled.  This way the sensitive credentials are never sent in the clear - with secure admin enabled, that is.


If you use a browser to send REST requests to the DAS, for example,
the browser automatically follows the redirection request.  The DAS will
then prompt the browser for authorization, which means the browser will
automatically prompt you for the admin username and password and then
resubmit the request. Even in this case, with secure admin enabled the
sensitive information is sent only over a secure connection.

With secure admin enabled, it's invisible to you but GlassFish also encrypts its admin messages between the DAS and remote instances.

Configuring Secure Admin


Java stores keys and certificates as entries in keystores. The keystore identifies each entry using an alias.  GlassFish has its own keystore and truststore. When you enable secure admin, GlassFish will by default use the alias s1as to look up the DAS key and certificate and the alias glassfish-instance to look up the instance key and certificate.  You can optionally tell GlassFish Server to use different aliases instead.


enable-secure-admin
[--adminalias=alias (default s1as)]
[ --instancealias=alias (default glassfish-instance)]


The enable-secure-admin and disable-secure-admin commands change quite a few parts of the domain's configuration for you.  You do not need to worry about the details, but if you are curious you can take a look at your domain's domain.xml file in the config directory before and after you enable secure admin to get a feel for what is happening behind the scenes.

Comments:

I tried this procedure w/Glassfish 3.1 on Redhat Linux running java 1.6.0_25. I actually also set the "security enabled" checkbox in the admin-listener via the the glassfish web admin console. I restarted glassfish successfully, but then I was unable to re-connect via https://localhost:4848. I received the following error: [#|2011-06-03T21:46:37.366-0400|SEVERE|oracle-glassfish3.1|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=25;_ThreadName=Thread-1;|PWC3989: An exception or error occurred in the container during the request processing com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Unexpected end of file from server at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:131) at com.sun.jersey.api.client.Client.handle(Client.java:629) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:601) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:499) at org.glassfish.admingui.common.security.AdminConsoleAuthModule.validateRequest(AdminConsoleAuthModule.java:225) : :

Posted by guest on June 03, 2011 at 04:29 PM CDT #

There is more to enabling secure admin than just turning on SSL for the admin listener. The admin console check-box in 3.1 is confusing. There is a change in 3.1.1 (not yet released but there are nightly and promoted builds available from the GlassFish download site) so checking the box will make all the required changes in configuration. Until then, please use the enable-secure-admin asadmin command.

Posted by Tim Quinn on June 06, 2011 at 01:58 AM CDT #

Thanks for your response. yes, I realize a number of things would need to be set up to support the the secure admin - I just assumed that the UI "checkbox" was synonymous with enable-secure-admin command. Good to hear that it is being added in 3.1.1. Thanks again for your guidance...

Posted by guest on June 06, 2011 at 04:27 AM CDT #

Hello,

I am running GF 3.1 and I have enabled secure admin using the "out of the box" default self-signed certs. With those, I am able to use the admin web gui over ssl. However, when I generate my own certs (signed by our own internal CA), I am unable to use the admin gui. The admin gui just spins in initialization and the GF server log indicates an SSLHandshake exception (see below). I believe I have generated the certs correctly and I add our internal CA to the cacerts file. The cert I add under "s1as" does appear correctly in my browser as a valid certificate when I access a secured page of my app running on GF. The cert I enter under "glassfish-instance" seems to be the problem. If I replace only the 's1as' key/cert, I can use the secure admin, but once I replace the 'glassfish-instance' key/cert which includes replacing the key/cert in keystore.jks and adding that cert to 'cacerts.jks', I have the problem. I noticed the default glassfish-instance cert has a "-instance" suffix on the CN of its subjectDN. I was wondering if there were any particulars/conventions that I am missing when trying to generate this cert. Thanks -

Relevant portion of stack trace when simply trying to initially access the admin gui follows:

Caused by: com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:131)
at com.sun.jersey.api.client.Client.handle(Client.java:629)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:601)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:459)
at org.glassfish.admingui.common.util.RestUtil.get(RestUtil.java:659)
at org.glassfish.admingui.common.util.RestUtil.restRequest(RestUtil.java:184)
at org.glassfish.admingui.common.handlers.RestApiHandlers.restRequest(RestApiHandlers.java:210)
... 50 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

Posted by guest on September 07, 2012 at 01:17 PM CDT #

Hi.

First, I'll assume that you are using the same aliases for the self-signed certs you are generating: s1as and glassfish-instance. You do not have to, but if you use different aliases then you need to change some other configuration in order to use the other aliases.

Second, I'll assume you are also restarting any running servers (the DAS and any instances) once you enable (or disable) secure admin.

One question: Do you see the same problem if you use the asadmin tool instead of the admin console? For example, try

asadmin uptime

after you have enabled secure admin and have restarted.

I'm not sure what the problem is. Here is some background that might help.

The credentials associated with the "s1as" alias are used to identify the domain admin server (the DAS) while the "glassfish-instance" credentials are used to identify any instances. Normally the instances only need to identify themselves to the DAS itself, and the DAS uses its key to identify itself to the instances (it does send admin messages to them) and to clients of the DAS, such as asadmin and the admin console.

So, given that you are trying to connect from a browser to the DAS I do not see why changing the credentials associated with "glassfish-instance" would make any difference.

I do not know why this would affect things, but you mentioned that you have added your CA's cert to cacerts.jks. So in theory there should be no need to also add the self-signed certs you are creating for the domain to cacerts.jks also. If I understand what you are doing, the self-signed certs come from your internal CA and if its cert is in cacerts.jks that should be sufficient.

One other detail: I expect you are generating your own DNs for the two key pairs you are generating and not duplicating the values from the default ones. If so, you should also run "asadmin enable-secure-admin-principal" for each of your generated DNs.

- Tim

Posted by Tim Quinn on September 12, 2012 at 04:00 PM CDT #

Tim,

Thanks for your response and info. First, your assumptions are correct -

- I am using the s1as and glassfish-instance aliases - just replacing with my own certs. Mainly, to minimizer config changes required.

- Second, I am restarting DAS (just running a simple, single domain config) after enabling secure admin

- When I enable secure admin, the first time I execute the command, I am prompted "Do you want to trust this certificate..." With the command displaying details of the cert I loaded under 's1as' (but not 'glassfish-instance'). If I say yes, it continues with command - eg., "uptime" display time system has been running.

- I agree with your statement about adding my certs (technically not self-sighed, but signed by my CA) to cacert.jks. I added them as act act of "desperation", seeing if somehow glassfish required them in cacerts - I thought maybe that's how Glassfish knew which "principals" to trust... I do have my CA cert in cacerts. Also, I have since removed my (non-CA) certs from cacerts.jks, but nothing changed. Note: I did still have the original (install gen'd, self-signed) 's1as' and 'glassfish-instance' certs in cacerts.jks file. I also removed them and restarted - however, this made no difference.

- The last point about executing "asadmin enable-secure-admin-principal" seemed promising as I (somehow) completely missed that when combing thru the "Oracle GlassFish Server 3.1 Security Guide" and other docs, which do describe. However, when I try to execute this I get the following error (I am running GF 3.1 build 43):

asadmin> enable-secure-admin-principal
Command enable-secure-admin-principal not found.
Check the entry of command name. This command may be provided by a package that is not installed.
Command enable-secure-admin-principal executed successfully.

The admin guide seesm to imply that by default, the aliases 's1as' and 'glassfish-instance' will be used to identify the certs to trust - and I assume that means those in cacerts.jks. Since I can't run secure admin principal, I can't get any further trying to evaluate this issue...

Thanks in advance for any further info - esp. if you can tell me why enable secure admin principal (and list-secure-admin-principals + probably others) is not available in asadmin command.

Posted by guest on September 17, 2012 at 12:32 PM CDT #

Oh, sorry. The enable-, disable-, and list-secure-admin-principal commands were added in 3.1.1 so were not there in 3.1.

Things do seem to work correctly using the command line: the prompt to trust the cert, the correct completion of the command if you say to trust the cert, and the completion of the command. What I don't see is why the glassfish-instance cert would cause trouble with accessing the server using the console. That server will use the s1as information - and that's what you see when the browser asks you if you want to trust that cert.

Have you tried this with any later version of GlassFish, just to see what would happen? I don't know if using a more recent version is an option for you.

I will ask some members of the team more familiar with the console if they have any suggestions.

If you want to pursue this could you open a GlassFish console issue here:

http://java.net/jira/browse/GLASSFISH

You might need to log in in order to see the Create an Issue option.

Thanks.

- Tim

Posted by Tim Quinn on September 17, 2012 at 04:02 PM CDT #

I should have asked this earlier - what version and build of Java are you using on each end?

Posted by Tim Quinn on September 17, 2012 at 04:07 PM CDT #

Not quite sure what you mean by "both sides", but my glassfish installation is running on jdk1.6.0_25. I have not tried with a later version of GF, but I can probably test that. Right now we are baselined on 3.1, but we may be able to update to 3.1.1 if necessary... I'd like to try and find a way to make it work w/ 3.1 for now - and then update after more testing/etc. of newer version.

Based on some digging I did in jira entries for GF and other sources, my understanding/assumption is that when using the admin GUI, DAS talks to admin functions via a REST I/F - really in my case I assume it's talking to itself, but still over a REST I/F. I assume that the DAS, as client, uses "glassfish-instance" as it's client identity. I assume this is why I somehow need to identify the DN of the glassfish-instance as "allowed"? Is this correct? and if so, is there any way to do this (even if manually) in GF 3.1? Maybe that's why enable-secure-admin-principal as added to 3.1.1?

I also know that the domain.xml has a restAuthURL setting - I know several people have indicated they had to manually override to change "localhost" to their hostname. However, this was after getting specific "hostanme validation" errors from SSL. I am not getting that far.

Posted by guest on September 17, 2012 at 09:19 PM CDT #

I guess I should've said upgrade to GF 3.1.2 vs. 3.1.1,whichever makes sense...

Posted by guest on September 17, 2012 at 09:21 PM CDT #

I guess I should've said upgrade to GF 3.1.2 vs. 3.1.1,whichever makes sense...

Posted by guest on September 17, 2012 at 10:54 PM CDT #

By "both sides" I meant the client system (where the browser is) and the server system (where the DAS is). These can be the same system, of course; I just wanted to be clear about your particular use case.

The DAS always identifies itself using s1as, not glassfish-instance.

Given that the CLI seems to work but not the console, we're outside the area where I can help effectively, especially in blog comments. Please open an issue so this can be worked more effectively. Sorry we couldn't figure it out quickly.

Posted by Tim Quinn on September 18, 2012 at 11:31 AM CDT #

Tim,

As it turns out, I think I've got it to work! I was fiddling around with it some more, and perusing some of the xml config files and for some reason, after reading your last response, I decided to try one more time to add the cert I generated and added as keypair in "keystore.jks" to "cacerts.jks". As I mentioned in previous post, I had added this cert before, and it didn't help. Previously, I just added it under some random alias, assuming it didn't matter (for cacerts.jks that is - I know it matters for keystore.jks). But this time, I decided to add it into cacerts.jks under the "s1as" alias - replacing the self-signed cert that was in there from initial install. As you mentioned before and I agreed, since I was using a CA to sign my certs, it seems all I should need in cacerts.jks is my CA cert, not the end-entity cert. (Note that I do still have my CA cert in cacerts.jks as well.) However, when I added the end-entity cert into cacerts.jks unser "s1as", enabled-secure-admin and restarted glassfish, I was now able to successfully get the login screen on the admin web gui and log in - using https://<my hostname>:4848.

This seems strange that I have to add the cert into cacerts as well and also under 's1as', but it does seem to work. Note also that I didn't make any changes to the 'glassfish-instance' entry - as you indicated, it doesn't seem to enter into the picture when talking to the DAS.

I'm not sure if this is just a GF 3.1 thing, or what. Would you suggest that I open a JIRA entry for this? At the least, if this is a required step, it should be included in the documentation I believe...

Thanks again for all your info/suggestions/help!

Posted by guest on September 18, 2012 at 05:34 PM CDT #

I owe you an apology. I should have realized this earlier, especially when we were talking about the secure admin principal-related commands.

In 3.1 the cert itself does need to be in cacerts. In that version that's how we tracked what DNs should be accepted as valid administrators. In later versions for various reasons we used entries in the domain.xml configuration, and that's where the secure admin principal-related commands entered the picture.

Despite the fact that users view the console in a web browser, the console actually "runs" in the server, and although I'm not familiar with all its details there must be some part that triggers some initialization that was failing because of the missing cert.

You are welcome to open a JIRA issue if you want to, but honestly it's probably not worth it. The mechanism is different in later releases of GlassFish and I think it's very unlikely that at this point the 3.1 documentation will be updated to reflect this situation.

I'm glad you found the solution, and again I'm sorry for not realizing the fix earlier.

- Tim

Posted by Tim Quinn on September 19, 2012 at 09:02 AM CDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

News and musings on the technology I work on at Oracle.

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today