Many software configurations use Java Application Servers (JAS) like WebLogic for serving dynamic applications. In addition to using a JAS, many companies will implement load balancing, proxying, or a number of other use cases by configuring an HTTP server front-end, such as Oracle HTTP Server (OHS) or Apache. By using an HTTP front-end, you can also improve performance by serving static content from the front-end HTTP server, and dynamic content from the JAS. I've been playing a bit with Oracle WebLogic 11g with an Apache front-end, and this post will document some of my findings and configuration practices. I'll be updating this post as a get further along, so feel free to come back periodically and see what's changed.
This environment is using Oracle Enterprise Linux 6, Oracle DB 12c, ODEE 12.4, WebLogic 11g (10.3.6), and Apache 2.2.15. The installations are fairly generic and sandbox, with only some minor changes as all of the elements are installed on the same machine. The biggest change is that the various Documaker applications have been consolidated into the AdminServer managed server to reduce the JVM overhead. NOTE: This document assumes you're using the latest WebLogic Plugin 11g, which is as of this writing 18.104.22.168.0. This is not the same version that comes with the WebLogic 10.3.6 installer (which is 10g). There are differences in the two plugins and the 10g Plugin will not support newer SSL implementation needs. The first step is to install and configure the WebLogic plugin for Apache. Obtain the 11g plugin from edelivery.oracle.com (search for WebLogic Plugin and download the 22.214.171.124.0 version). Once you've downloaded and unzipped the file, you will be able to find and extract the appropriate plugin version for your Apache server version and OS. Note the README.txt file contains explicit instructions for deploying the plugin to your environment. I will paraphrase these instructions here, but reference them for your installation.
LoadModule weblogic_module plugin/lib/mod_wl.so
By using the Include directive here, we have externalized the WebLogic plugin configuration to a separate file. Create the /etc/httpd/conf/weblogic.conf file and add the following directives:
This directive tells Apache that all URLs ("/*") will use the weblogic-handler, which further indicates that traffic should be redirected to the WebLogic server on host testbed, listening on port 7001. Start Apache using apachectl start, or service start httpd and then access http://testbed/console. (Remember: testbed is the hostname, and Apache and WebLogic are on the same host). You should see the WebLogic console login screen (the application may have to deploy first). It's possible that you'll see an error like this:
"Error:Failure of server APACHE bridge:No backend server available for connection: timed out after 10 seconds or idempotent set to OFF."
This is a rather generic error unfortunately. You can review the Apache logs (/etc/httpd/logs/error_log) and you might see:
[Thu Jan 12 08:48:35 2017] [error] [client xxx.xxx.xxx.xxx] ap_proxy: trying GET /console/console.portal at backend host 'xxx.xxx.xxx.xxx/7001; got exception 'CONNECTION_REFUSED [os error=13, line 1735 of ../nsapi/URL.cpp]: Error connecting to host xxx.xxx.xxx.xxx:7001 errno = 13'
One possible solution for this (which worked in my environment) is modifying the SELinux parameter to Permissive:
# setenforce 0
After setting the Enforce parameter to Permissive, you can restart Apache (apachectl stop then apachectl start) and run the test again. If everything works, soldier on to set up SSL.
Before we dig in, I advise a review of the official Oracle documentation on SSL and WebLogic. It will give you some primer information that is helpful. After that, you can continue here.
The first step in securing your applications running on WebLogic is to setup SSL. By default, Documaker applications will use a test SSL certificate, which is why when you access the applications for the first time you might get various errors about the certificate being untrusted. The reason for this is because digital certificates are issued by Certificate Authorities (CAs). Most browsers come configured with a trust store, which is a storage area for root CA certificates for each of the CAs that the browser is configured to trust. Trust is chained, meaning, if you or your company requests an SSL certificate from a CA, you'll issue a CSR (certificate signing request) from your server(s), which you'll submit to your CA. This establishes the chain of trust: the CA verifies that your company is who it says it is, and signs the certificate using their root certificate, establishing the digital chain. When your browser connects to your company's secured web server using secure protocols (e.g. HTTPS) one of the activities that happens is an SSL handshake, where the client (browser) requests an identity certificate from the server. The server returns the certificate, which the client compares against the root certificates in the trust store. If the certificate is properly signed, valid, and not expired, and meets other security parameters (such as the server name matching the hostname in the certificate, or the certificate is encrypted using an appropriately secure encryption scheme that the browser will accept, to name a few) the browser will allow the connection to proceed. Otherwise, the handshake will fail and the connection will be deemed insecure. Side note: certificates are stored in keystores, which I'll discuss below.
In short, test certificates are not signed by CAs, and therefore most browsers will not trust them, which is why you'll be presented with a warning when you try to access Documaker applications secured with test certificates. To resolve this you'll need to configure your identity store with appropriately-signed certificates, or if your company acts as a CA on its own behalf and issues its own certificates for internal use, you'll need to import the root CA used to sign these certificates into every client browser trust store.
First, a bit There are two types of keystores: identity and trust. The Identity keystore contains the private certificates that identify the server - these should be kept safe. The trust keystore contains the public certificates of CAs and other servers which this server will trust. The default installation includes demo keystores for identity and trust. You will want to keep these as-is in case you should need to revert to them. Create a new keystore using the instructions found here. Once you have created the keystores you can configure them on the Configuration > Keystores tab. Additionally, you will need to access the Configuration > SSL tab to add the private key passphrase for the identity store, and optionally enable Use JSSE SSL.
NOTE: When creating the private and public keys you will have the option to specify the bit length for the encryption key and the hash method. Do not use a length less than 1024 bits, and do not use SHA-1 as the hash method. I recommend using SHA-2 hashing and 2048-bit key length. Note that using SHA-2 means you will need to enable Use JSSE SSL. JSSE requires WebLogic 10.3.6 or higher. If you wish to use TLS v1.2, you need to use Java 7, which also requires WebLogic 10.3.6. Since WebLogic 10.3.6 is supported with ODEE 12.3 and higher, if you require TLS v1.2 or SHA-2 support you must use ODEE 12.3 or higher.
SPECIAL NOTE: Some earlier releases of Documaker Enterprise Edition shipped with keystores that had keys of 1024-bit length. Some browsers have been updated to require longer bit-length in keys, and if you attempt an SSL connection with key bit-length that is shorter than the browser requires, you'll be unable to connect. Some browsers such as Firefox will give you a reasonable error message (e.g. "SSL_ERROR_WEAK_SERVER_CERT_KEY") but not all browsers will. You can regenerate the keys to use a longer bit-length by logging into a terminal session on the WebLogic server and executing the following commands. After executing the -genkey command you'll be prompted to enter some information about the key; enter whatever is appropriate for your use.
keytool -delete -keystore DemoIdentity.jks -storepass DemoIdentityPassPhrase -alias DemoIdentity
keytool -genkey -keystore DemoIdentity.jks -storepass DemoIdentityPassPhrase -alias DemoIdentity -keypass DemoIdentityPassPhrase -keyalg RSA -keysize 2048 -validity 3650
Other problems you may encounter are "An error occurred during a connection to 192.168.1.72:7002. SSL received a weak ephemeral Diffie-Hellman key in Server Key Exchange handshake message. Error code: SSL_ERROR_WEAK_SERVER_EPHEMERAL_DH_KEY" or "SSL_ERROR_NO_CYPHER_OVERLAP", which means that the WebLogic server is configured to allow a version of SSL that the browser doesn't like, likely SSLv3. To disable this, you need to tell WebLogic to use a specific version of SSL, and that is done via a startup parameter:
You can apply this parameter is several places, depending on your configuration (e.g. startManagedWebLogic.sh, WebLogic console Server Start page, etc). It may also be beneficial to add the following SSL debugging flags here as well:
You can establish all of these parameters by adding a line to your startWebLogic.sh/cmd scripts, which should be located in [MIDDLEWARE_HOME]/user_projects/domains/idocumaker_domain:
JAVA_OPTIONS="-Dssl.debug=true -Dweblogic.StdoutDebugEnabled=true -Djavax.net.debug=all"
By adding this line and restarting the appropriate server(s) you will get additional debugging information for SSL connections.
After you've secured the necessary certificates and created keystores, you'll need to attach them to the managed servers in WebLogic which are running your Documaker applications. Login to WebLogic console and navigate to Environment > Servers. Default installations include the following managed servers: soa_server, idm_server, and dmkr_server. Repeat these steps for each managed server:
After restarting the managed servers, you should be able to access your Documaker web applications using the secure port and protocol, e.g. https://xxxx:10002/DocumakerAdministrator. If you receive errors, review your configuration and review the SSL Debugging information here for additional assistance. Once you have validated that all is well you can secure the AdminServer using the same configuration methods. After this is completed, you're ready to configure the WebLogic plugin. Note: if your Listen Address name doesn't match, you have a few options: you can disable the RequireHostNameMatch setting (set to false) for the WebLogic Plugin, or you can use SSLHostMatchOID setting to configure a different portion of the Subject Distinguished Name to use for matching. Refer to WebLogic Plugin configuration details for specifics.
The WebLogic Plugin 11g uses the Oracle Wallet for SSL configuration. The plugin files downloaded earlier contain the orapki tool to assist you in managing the wallet. You need to have the JAVA_HOME environment variable set to a valid JDK. The commands below will create the wallet files in the /etc/httpd/ssl/wallet directory. You will be prompted for a password current the creation process.
/etc/httpd/plugin/bin/orapki wallet create -wallet /etc/httpd/ssl/wallet -auto_login
chmod a+r /etc/httpd/ssl
chmod a+r /etc/httpd/ssl/wallet/*.sso
chmod a+r /etc/httpd/ssl/wallet/*.p12
chmod a+r /etc/httpd/ssl/wallet
Next, you need a copy of the server identity certificate - this is the certificate that resides in your identity keystore which is tied to your WebLogic host. Documaker ships with a demo identity keystore which contains the necessary certificates, and that's what we'll use. If you have a different scenario (e.g. you have a non-demo keystore or certificate) you should review the orapki documentation. Note that you will have to modify the keystore somewhat, because the orapki tool requires that the private key password and the keystore password be the same. You can temporarily change the private key password using the keytool:
keytool -keypasswd -alias demoidentity -keystore DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase
Once the private key and keystore passwords are the same, you can then import the keystore into the wallet:
/etc/httpd/plugin/bin/orapki wallet jks_to_pkcs12 -wallet ./wallet -pwd <wallet password> -keystore <MIDDLEWARE_HOME>/wlserver_10.3/server/lib/DemoIdentity.jks -jkspwd DemoIdentityKeyStorePassPhrase
Note: You can run the keytool command again if you wish to revert the private key passphrase back to its original form. If you choose not to, you should use the WebLogic Console to update the managed servers SSL settings with the new private key passphrase. Also of note - if you are using NodeManager to control your WebLogic domain (which is a good idea for production environments) you may receive a failure to start up if you change the identity key passphrase.
You can update NodeManager with the new passphrase by adding the following lines to the end of the nodemanager.properties file located in the WL_HOME/common/nodemanager directory. I've used the values below to match what I used with keytool and orapki above; obviously you should replace these with what is correct for your environment. Note that NodeManager will encrypt the passphrase values upon startup, so it's a good idea to start up NodeManager after making these changes so they don't exist in unencrypted form too long.
Edit the weblogic.conf file for SSL. Previously we simply added the necessary configuration to support the WebLogic plugin which allows Apache to be a proxy. Now, we'll add the ability for Apache to maintain an SSL connection, which is necessary for secure communications. Recall that in the initial configuration we created an all-encompassing directive <LocationMatch /*> that routes all traffic from Apache to WebLogic. Most implementation practices will avoid this sort of behavior, so we'll explicitly define what needs to be forwarded now. Replace the contents of the weblogic.conf file with the following:
The top level of the file defines some directives which apply to all subsequent <LocationMatch> directives. This will allow you to redirect traffic for specific services and ports. Note that the configuration is slightly different if you are running a clustered WebLogic configuration. If clustering is configured, you'll be using directives like the following:
The WebLogic 11g plugin may fail to load properly if you have not configured the LD_LIBRARY_PATH appropriately. Consult the /etc/httpd/ssl/proxy.log and if you see a message similar to "mod_weblogic(ssl): Cannot Load library: libwlssl.so" then you need to review your system to make sure httpd is able to dynamically load the plugin libraries.
After some mild Sturm und Drang, we have SSL enabled and Apache is proxying our requests to WebLogic using the WebLogic plugin. We can access the Documaker applications by a simple Apache URL with no WebLogic server or port. From here, we can expand further by creating an Apache cluster and a WebLogic cluster to handle more users in the system. Note: If you enabled the SSL logging in the WebLogic Plugin or on the Managed Servers, I highly recommend you remove those settings once everything is working because there is a prodigious amount of debugging information that will be logged. Til next time!