X

Technical info and insight on using Oracle Documaker for customer communication and document automation

  • January 12, 2017

Oracle WebLogic Plugin with Apache Configuration

Andy Little
Technical Director

Many software configurations use Java Application Servers (JAS) like WebLogic for serving dynamic applications.  In addition to using a JAS, many companies will implement load balancing, proxying, or a number of other use cases by configuring an HTTP server front-end, such as Oracle HTTP Server (OHS) or Apache. By using an HTTP front-end, you can also improve performance by serving static content from the front-end HTTP server, and dynamic content from the JAS. I've been playing a bit with Oracle WebLogic 11g with an Apache front-end, and this post will document some of my findings and configuration practices. I'll be updating this post as a get further along, so feel free to come back periodically and see what's changed.

Initial Plugin Deployment 

This environment is using Oracle Enterprise Linux 6, Oracle DB 12c, ODEE 12.4,  WebLogic 11g (10.3.6), and Apache 2.2.15.  The installations are fairly generic and sandbox, with only some minor changes as all of the elements are installed on the same machine. The biggest change is that the various Documaker applications have been consolidated into the AdminServer managed server to reduce the JVM overhead. NOTE: This document assumes you're using the latest WebLogic Plugin 11g, which is as of this writing 11.1.1.9.0. This is not the same version that comes with the WebLogic 10.3.6 installer (which is 10g). There are differences in the two plugins and the 10g Plugin will not support newer SSL implementation needs.  The first step is to install and configure the WebLogic plugin for Apache. Obtain the 11g plugin from edelivery.oracle.com (search for WebLogic Plugin and download the 11.1.1.9.0 version). Once you've downloaded and unzipped the file, you will be able to find and extract the appropriate plugin version for your Apache server version and OS. Note the README.txt file contains explicit instructions for deploying the plugin to your environment. I will paraphrase these instructions here, but reference them for your installation.

  1. Unzip plugin folder to the Apache configuration directory in a subfolder called plugin (e.g. /etc/httpd/plugin
  2. Add the contents plugin/lib folder to the [APACHE_HOME]/lib folder, or edit the /etc/init.d/httpd script file to add the plugin/lib folder to LD_LIBRARY_PATH
  3. Edit the /etc/httpd/conf/httpd.conf to add the WebLogic module. Add the lines below:

LoadModule weblogic_module plugin/lib/mod_wl.so
<IfModule mod_weblogic.c>

Include conf/weblogic.conf

</IfModule> 

By using the Include directive here, we have externalized the WebLogic plugin configuration to a separate file. Create the /etc/httpd/conf/weblogic.conf file and add the following directives:

<IfModule mod_weblogic.c>
        WebLogicHost testbed
        WebLogicPort 7001
        MatchExpression *.*
</IfModule>
<LocationMatch /*>

SetHandler weblogic-handler

</LocationMatch>

This directive tells Apache that all URLs ("/*") will use the weblogic-handler, which further indicates that traffic should be redirected to the WebLogic server on host testbed, listening on port 7001. Start Apache using apachectl start, or service start httpd and then access http://testbed/console. (Remember: testbed is the hostname, and Apache and WebLogic are on the same host). You should see the WebLogic console login screen (the application may have to deploy first).  It's possible that you'll see an error like this:

"Error:Failure of server APACHE bridge:No backend server available for connection: timed out after 10 seconds or idempotent set to OFF."

This is a rather generic error unfortunately. You can review the Apache logs (/etc/httpd/logs/error_log) and you might see:

[Thu Jan 12 08:48:35 2017] [error] [client xxx.xxx.xxx.xxx] ap_proxy: trying GET /console/console.portal at backend host 'xxx.xxx.xxx.xxx/7001; got exception 'CONNECTION_REFUSED [os error=13, line 1735 of ../nsapi/URL.cpp]: Error connecting to host xxx.xxx.xxx.xxx:7001 errno = 13' 

One possible solution for this (which worked in my environment) is modifying the SELinux parameter to Permissive:

# getenforce
Enforcing
# setenforce 0
# getenforce
Permissive

After setting the Enforce parameter to Permissive, you can restart Apache (apachectl stop then apachectl start) and run the test again. If everything works, soldier on to set up SSL.

Using SSL

Before we dig in, I advise a review of the official Oracle documentation on SSL and WebLogic. It will give you some primer information that is helpful. After that, you can continue here.

The first step in securing your applications running on WebLogic is to setup SSL. By default, Documaker applications will use a test SSL certificate, which is why when you access the applications for the first time you might get various errors about the certificate being untrusted. The reason for this is because digital certificates are issued by Certificate Authorities (CAs). Most browsers come configured with a trust store, which is a storage area for root CA certificates for each of the CAs that the browser is configured to trust. Trust is chained, meaning, if you or your company requests an SSL certificate from a CA, you'll issue a CSR (certificate signing request) from your server(s), which you'll submit to your CA. This establishes the chain of trust: the CA verifies that your company is who it says it is, and signs the certificate using their root certificate, establishing the digital chain. When your browser connects to your company's secured web server using secure protocols (e.g. HTTPS) one of the activities that happens is an SSL handshake, where the client (browser) requests an identity certificate from the server. The server returns the certificate, which the client compares against the root certificates in the trust store. If the certificate is properly signed, valid, and not expired, and meets other security parameters (such as the server name matching the hostname in the certificate, or the certificate is encrypted using an appropriately secure encryption scheme that the browser will accept, to name a few) the browser will allow the connection to proceed. Otherwise, the handshake will fail and the connection will be deemed insecure. Side note: certificates are stored in keystores, which I'll discuss below.

In short, test certificates are not signed by CAs, and therefore most browsers will not trust them, which is why you'll be presented with a warning when you try to access Documaker applications secured with test certificates. To resolve this you'll need to configure your identity store with appropriately-signed certificates, or if your company acts as a CA on its own behalf and issues its own certificates for internal use, you'll need to import the root CA used to sign these certificates into every client browser trust store.

Configuring Keystores

First, a bit There are two types of keystores: identity and trust. The Identity keystore contains the private certificates that identify the server - these should be kept safe. The trust keystore contains the public certificates of CAs and other servers which this server will trust. The default installation includes demo keystores for identity and trust. You will want to keep these as-is in case you should need to revert to them. Create a new keystore using the instructions found here. Once you have created the keystores you can configure them on the Configuration > Keystores tab. Additionally, you will need to access the Configuration > SSL tab to add the private key passphrase for the identity store, and optionally enable Use JSSE SSL.

NOTE:  When creating the private and public keys you will have the option to specify the bit length for the encryption key and the hash method. Do not use a length less than 1024 bits, and do not use SHA-1 as the hash method. I recommend using SHA-2 hashing and 2048-bit key length. Note that using SHA-2 means you will need to enable Use JSSE SSL. JSSE requires WebLogic 10.3.6 or higher. If you wish to use TLS v1.2, you need to use Java 7, which also requires WebLogic 10.3.6. Since WebLogic 10.3.6 is supported with ODEE 12.3 and higher, if you require TLS v1.2 or SHA-2 support you must use ODEE 12.3 or higher.

SPECIAL NOTE: Some earlier releases of Documaker Enterprise Edition shipped with keystores that had keys of 1024-bit length. Some browsers have been updated to require longer bit-length in keys, and if you attempt an SSL connection with key bit-length that is shorter than the browser requires, you'll be unable to connect. Some browsers such as Firefox will give you a reasonable error message (e.g. "SSL_ERROR_WEAK_SERVER_CERT_KEY") but not all browsers will. You can regenerate the keys to use a longer bit-length by logging into a terminal session on the WebLogic server and executing the following commands. After executing the -genkey command you'll be prompted to enter some information about the key; enter whatever is appropriate for your use.

cd [MIDDLEWARE_HOME]/wlserver_10.3/server/lib
keytool -delete -keystore DemoIdentity.jks -storepass DemoIdentityPassPhrase -alias DemoIdentity
keytool -genkey -keystore DemoIdentity.jks -storepass DemoIdentityPassPhrase -alias DemoIdentity -keypass DemoIdentityPassPhrase -keyalg RSA -keysize 2048 -validity 3650

Other problems you may encounter are "An error occurred during a connection to 192.168.1.72:7002. SSL received a weak ephemeral Diffie-Hellman key in Server Key Exchange handshake message. Error code: SSL_ERROR_WEAK_SERVER_EPHEMERAL_DH_KEY" or "SSL_ERROR_NO_CYPHER_OVERLAP", which means that the WebLogic server is configured to allow a version of SSL that the browser doesn't like, likely SSLv3. To disable this, you need to tell WebLogic to use a specific version of SSL, and that is done via a startup parameter:

-Dweblogic.security.SSL.protocolVersion=TLS1

You can apply this parameter is several places, depending on your configuration (e.g. startManagedWebLogic.sh, WebLogic console Server Start page, etc). It may also be beneficial to add the following SSL debugging flags here as well:

-Dssl.debug=true 
-Dweblogic.StdoutDebugEnabled=true
-Djavax.net.debug=all

You can establish all of these parameters by adding a line to your startWebLogic.sh/cmd scripts, which should be located in [MIDDLEWARE_HOME]/user_projects/domains/idocumaker_domain: 

JAVA_OPTIONS="-Dssl.debug=true -Dweblogic.StdoutDebugEnabled=true -Djavax.net.debug=all" 

By adding this line and restarting the appropriate server(s) you will get additional debugging information for SSL connections. 

Configuring Managed Servers 

After you've secured the necessary certificates and created keystores, you'll need to attach them to the managed servers in WebLogic which are running your Documaker applications. Login to WebLogic console and navigate to Environment > Servers. Default installations include the following managed servers: soa_server, idm_server, and dmkr_server. Repeat these steps for each managed server:

  1. Click the server name (e.g. dmkr_server)
  2. Under Configuration > General, tick the box for SSL Listen Port Enabled.
  3. Make sure, on this page, that you have properly configured the Listen Address for the exact DNS name or IP address that matches your SSL certificate.
  4. Enter the desired port number for SSL Listen Port. Usually this is the Listen Port + 1 (e.g. Listen Port = 10001, SSL Listen Port = 10002).
  5. Click Save.
  6. Navigate to Configuration > SSL and change any desired settings. 
  7. Navigate to Configuration > Keystores and change any desired settings. 

After restarting the managed servers, you should be able to access your Documaker web applications using the secure port and protocol, e.g. https://xxxx:10002/DocumakerAdministrator. If you receive errors, review your configuration and review the SSL Debugging information here for additional assistance. Once you have validated that all is well you can secure the AdminServer using the same configuration methods. After this is completed, you're ready to configure the WebLogic plugin. Note: if your Listen Address name doesn't match, you have a few options: you can disable the RequireHostNameMatch setting (set to false) for the WebLogic Plugin, or you can use SSLHostMatchOID setting to configure a different portion of the Subject Distinguished Name to use for matching. Refer to WebLogic Plugin configuration details for specifics.

Configuring the WebLogic Plugin for Apache

The WebLogic Plugin 11g uses the Oracle Wallet for SSL configuration. The plugin files downloaded earlier contain the orapki tool to assist you in managing the wallet.  You need to have the JAVA_HOME environment variable set to a valid JDK. The commands below will create the wallet files in the /etc/httpd/ssl/wallet directory.  You will be prompted for a password current the creation process.

mkdir /etc/httpd/ssl

/etc/httpd/plugin/bin/orapki wallet create -wallet /etc/httpd/ssl/wallet -auto_login

chmod a+r /etc/httpd/ssl

chmod a+r /etc/httpd/ssl/wallet/*.sso

chmod a+r /etc/httpd/ssl/wallet/*.p12

chmod a+r /etc/httpd/ssl/wallet 

Next, you need a copy of the server identity certificate - this is the certificate that resides in your identity keystore which is tied to your WebLogic host. Documaker ships with a demo identity keystore which contains the necessary certificates, and that's what we'll use. If you have a different scenario (e.g. you have a non-demo keystore or certificate) you should review the orapki documentation. Note that you will have to modify the keystore somewhat, because the orapki tool requires that the private key password and the keystore password be the same. You can temporarily change the private key password using the keytool:

keytool -keypasswd -alias demoidentity -keystore DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase

Once the private key and keystore passwords are the same, you can then import the keystore into the wallet:

/etc/httpd/plugin/bin/orapki wallet jks_to_pkcs12 -wallet ./wallet -pwd <wallet password> -keystore <MIDDLEWARE_HOME>/wlserver_10.3/server/lib/DemoIdentity.jks -jkspwd DemoIdentityKeyStorePassPhrase 

Note: You can run the keytool command again if you wish to revert the private key passphrase back to its original form. If you choose not to, you should use the WebLogic Console to update the managed servers SSL settings with the new private key passphrase. Also of note - if you are using NodeManager to control your WebLogic domain (which is a good idea for production environments) you may receive a failure to start up if you change the identity key passphrase. 

<Jan 25, 2017 8:10:11 AM> <SEVERE> <Fatal error in node manager server>
weblogic.nodemanager.common.ConfigException: Incorrect identity private key password
        at weblogic.nodemanager.server.SSLConfig.loadKeyStoreConfig(SSLConfig.java:170)
        at weblogic.nodemanager.server.SSLConfig.<init>(SSLConfig.java:102)
        at weblogic.nodemanager.server.NMServer.init(NMServer.java:186)
        at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:148)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:380)
        at weblogic.NodeManager.main(NodeManager.java:31) 

You can update NodeManager with the new passphrase by adding the following lines to the end of the nodemanager.properties file located in the WL_HOME/common/nodemanager directory. I've used the values below to match what I used with keytool and orapki above; obviously you should replace these with what is correct for your environment. Note that NodeManager will encrypt the passphrase values upon startup, so it's a good idea to start up NodeManager after making these changes so they don't exist in unencrypted form too long. 

KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreFileName=<MIDDLEWARE_HOME>/wlserver_10.3/server/lib/DemoIdentity.jks
CustomIdentityKeyStorePassPhrase=DemoIdentityKeyStorePassPhrase
CustomIdentityAlias=demoidentity
CustomIdentityPrivateKeyPassPhrase=DemoIdentityKeyStorePassPhrase 

Edit the weblogic.conf file for SSL. Previously we simply added the necessary configuration to support the WebLogic plugin which allows Apache to be a proxy. Now, we'll add the ability for Apache to maintain an SSL connection, which is necessary for secure communications. Recall that in the initial configuration we created an all-encompassing directive <LocationMatch /*>  that routes all traffic from Apache to WebLogic. Most implementation practices will avoid this sort of behavior, so we'll explicitly define what needs to be forwarded now. Replace the contents of the weblogic.conf file with the following:

WLLogFile /etc/httpd/ssl/proxy.log
WLSSLWallet /etc/httpd/ssl/wallet
Debug ALL
DebugConfigInfo ON
SecureProxy ON
WLProxySSL ON
FileCaching ON
<LocationMatch "/DocumakerCorrespondence*">
SetHandler weblogic-handler
WebLogicHost testbed
WebLogicPort 9002
</LocationMatch>
<LocationMatch "/plugin*">
SetHandler weblogic-handler
WebLogicHost testbed
WebLogicPort 9002
</LocationMatch>
<LocationMatch "/DWS*">
SetHandler weblogic-handler
WebLogicHost testbed
WebLogicPort 10002
</LocationMatch>

<LocationMatch "/DocumakerAdministrator*">
  SetHandler weblogic-handler
  WebLogicHost testbed
  WebLogicPort 10002
</LocationMatch> 
<LocationMatch "/DocumakerDashboard*">
  SetHandler weblogic-handler
  WebLogicHost testbed
  WebLogicPort 10002
</LocationMatch>
<LocationMatch "/DocumakerDashboard*">
  SetHandler weblogic-handler
  WebLogicHost testbed
  WebLogicPort 10002
</LocationMatch>

The top level of the file defines some directives which apply to all subsequent <LocationMatch> directives. This will allow you to redirect traffic for specific services and ports. Note that the configuration is slightly different if you are running a clustered WebLogic configuration. If clustering is configured, you'll be using directives like the following:

WebLogicCluster testbed1:9002,testbed2:9002

The WebLogic 11g plugin may fail to load properly if you have not configured the LD_LIBRARY_PATH appropriately. Consult the /etc/httpd/ssl/proxy.log and if you see a message similar to "mod_weblogic(ssl): Cannot Load library: libwlssl.so" then you need to review your system to make sure httpd is able to dynamically load the plugin libraries.

Conclusion

After some mild Sturm und Drang, we have SSL enabled and Apache is proxying our requests to WebLogic using the WebLogic plugin. We can access the Documaker applications by a simple Apache URL with no WebLogic server or port. From here, we can expand further by creating an Apache cluster and a WebLogic cluster to handle more users in the system. Note: If you enabled the SSL logging in the WebLogic Plugin or on the Managed Servers, I highly recommend you remove those settings once everything is working because there is a prodigious amount of debugging information that will be logged. Til next time! 

Join the discussion

Comments ( 2 )
  • Ravi Verma Thursday, November 14, 2019
    Thank you for the write-up. I tried to use your instructions on Apache 2.4. The proxy keeps failing with the following message in the Apache log.

    [Thu Nov 14 14:15:28.097151 2019] [weblogic:error] [pid 19212:tid 140618638554880] [client XX.XXX.XX.XX:51461] *******Exception type [HALF_OPEN_SOCKET_RETRY] (attempt to read from disconnected socket:Was unexpected EOF) raised at line 792 of URL.cpp
  • Andy Thursday, November 14, 2019
    Ravi -- check out MOS Document 2307148.1 (https://support.oracle.com/epmos/faces/DocumentDisplay?id=2307148.1), it has the solution for this error condition.
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.