By Vijay Shanmugam-Oracle on Aug 03, 2010
IntroductionThis topic is created to give better understanding of how Oracle Workflow uses SSL in different modules and if in case of an issue how to troubleshoot it.
SSL is a technology that defines the essential functions of mutual authentication, data encryption, and data integrity for secure transactions. Exchange of data between the client and server in such secure transactions is said to use the Secure Sockets Layer (SSL). SSL uses 2 types of Certificates:
- User certificates - These are Certificates issued to servers or users to prove their identity in a public key/private key exchange.
- Trusted certificates - These are Certificates representing entities whom you trust - such as certificate authorities who sign the user certificates they issue.
Read more information in MOS Doc Enabling SSL in Oracle Applications Release 12.
Oracle Workflow modules act as a HTTP/SSL client in different scenarios connecting to the EBS or non-EBS SSL servers. For SSL/TLS connection, the Workflow's client process should have access to the following.
- Necessary SSL libraries (mostly available)
- Trusted certificates (ca.crt) used to validate if the server certificate is valid.
- Client certificates (if client authentication required).
The Key/Trust Store accessible to the Workflow process should have the correct certificates for the client code to participate in SSL handshake with the server. In summary, the SSL client should be able to validate the SSL server certificate's authenticity using it's root certificate and exchange cipher suites with the server.
When troubleshooting SSL issues with Workflow, it is important to understand in detail as to where exactly each Workflow's HTTP client process executes so the necessary setup can be verified.
Workflow Manager UI
Workflow Notification Mailer is configured from Oracle Applications Manager >> Workflow Manager screens. When configuring IMAP and SMTP servers with SSL Enabled option checked, the Workflow Manager code attempts to connect to the IMAP and SMTP servers over SSL to validate connectivity before saving the configuration parameters. Since the OAM UI executes within OACORE OC4J container, it would use
$OA_JRE_TOP/bin/java. The root certificate in the JRE's store should correspond to the Server Certificates on IMAP and SMTP servers in order for the connection to succeed.
Mailer executes within the Concurrent Manager process in the CM tier. The Java run-time used to run Mailer Service is configured as $AF_JRE_TOP/bin/java. If SSL is enabled, Mailer initiates SSL connection for following three reasons.
- SMTP server - Establish SMTP connection to send e-mails.
- IMAP server - Establish IMAP connection to receive e-mails.
- EBS or non-EBS web server -Establish HTTP connection to a Web server to fetch OAF content or if images are to be embedded, connect to a content server.
When Status Monitor page is loaded, there are two separate actions.
- Loading of the OAF page first
- Then loading of the Monitor Applet within that above OAF page that shows the diagram
Status Monitor makes HTTP requests during both actions above.
- OAF controller GraphMonitorCO.java - When status monitor diagram page is loaded, this OAF controller code runs within OC4J? and it acts as HTTP client making a loop back request to Web server to fetch tags to embed the Status Monitor applet. If any exception occurs while loading the status monitor diagram page, it will result in OAF page error. OC4J runs using JRE at $OA_JRE_TOP/bin/java.
- Monitor Applet - The monitor applet code running in Web Browser JVM (JInitiator or Sun JRE plugin) makes HTTP requests to fetch data to display diagram on the applet. The applet loads only after the status monitor page loads successfully above in (a). Any exceptions within the applet can only be tracked through Java console output on the browser.
From R12.1, Business Event System supports invoking web services. This includes following steps.
- Consuming the WSDL - WSDL is consumed in a OAF page to create web service meta-data. The controller makes HTTP(S) request to the WSDL URL. In order for the OAF page to successfully connect to a HTTPS WSDL URL, the OC4J JVM should have access required SSL libraries and root certificate installed.
- Invoking the web service - Invocation of the earlier consumed web service may occur in one of the following two processes.
- OC4J - If the web service is invoked from a OAF page using synchronous subscription, then the OC4J process acts as SSL client. Like any OAF page, the process runs using $OA_JRE_TOP/bin/java
- Concurrent Manager - If the web service is invoked using a
asynchronous subscription, it is executed by Java Deferred Agent
Listener in Agent Listener Service process. Like Workflow Mailer
Service, this runs using $AF_JRE_TOP/bin/java
In summary, Workflow's SSL client code executes in following run-time environments
- $OA_JRE_TOP/jre/bin/java (Web Tier)
- $AF_JRE_TOP/jre/bin/java (Concurrent Tier)
- Sun JRE
For any SSL handshake errors involving Workflow code as client,
- Always verify that the JVM from which Workflow code initiates a SSL connection has the required root certificate installed
- If the server presents a certificate chain to validate, then the complete chain is installed on the client side.
- Most importantly, as part of SSL enablement of EBS, is the trusted certificate/certificate chain installed into internal EBS JVMs that could potentially act as SSL client to our own EBS servers.
SSL connectivity can be verified from the run-time environment where Workflow acts as client to a SSL server to confirm if the setup is correct. This helps troubleshoot general SSL setup without involving Workflow code.
For example, for Status Monitor SSL issues,
- GraphMonitorCO.java - Check connectivity from $OA_JRE_TOP/bin/java by using this JRE's trust-store to the web-server.
- Status Monitor Applet - Check connectivity from client
machine based on appropriate run-time such as Sun JRE or JInitiator. For
JInitiator, the certificates are stored under <JInitiator Home>
\lib\security\certdb.txt. Java run-time is accessible using <JInitiator Home> \bin\java.exe
For connectivity testing following can help.
- openssl utility available in Unix based platforms
- This sample
be used to test a handshaking from the Java run-time
- Download the Java class source code in a directory. There is no package name for this Java class.
- Compile SSLSocketClientTest.java
javac -classpath $CLASSPATH SSLSocketClientTest.java
- Run it as below from required Java run-time
java -classpath . -Dserver=server.com -Dport=443 -Dtruststore=<jre/lib/security /cacerts> SSLSocketClientTest
These steps are mentioned as part of EBS SSL setup MOS Doc Enabling SSL in Oracle Applications Release 12.
- Navigate to the $OA_JRE_TOP/lib/security directory
- Backup the existing cacerts file.
- Copy your ca.crt and server.crt files to this directory. Issue the following command to insure that cacerts has write permissions. chmod u+w cacerts
- Add your Apache ca.crt and server.crt to cacerts keytool -import -alias ApacheRootCA -file ca.crt -trustcacerts -v -keystore cacerts
- When prompted enter the keystore password (default password is changeit).
keytool -import -alias ApacheServer -file server.crt -trustcacerts -v -keystore cacerts
A certificate chain establishes as chain of trust. The certificate issued by a CA is not signed by their own root certificate but is signed by another CA's root certificate. For example, VeriSign is the most common CA whose user certificates that all the web browsers trust. This is because, the web browsers are pre-installed with VeriSign's root certificate. If another CA XyZ issues a certificate signed using VeriSign's root certificate, then the browser can trust the certificate from XyZ simply because the root certificate is issued by CA.
The chain of trust is
VeriSign's Root CA Certificate >> XyZ's Intermediate CA Certificate >> Server Certificate
There must be a chain of trust from the server certificate up through intermediate authorities up to one of the trusted Root Certificate in order for the server to be trusted. If the client is unable to build the chain of trust starting from the server certificate up to a trusted Root Certificate, then the SSL handshake fails with X509CertChainIncompleteErr.
Concatenate all the certificates in the chain into one single file as per the order in which they appear in the chain. Server Certificate should be the first one in the chain and followed by the intermediate certificates and finally the root certificate. You can verify this order and download the certificates using a Web browser. Import the concatenated certificate into the JDK from which the Workflow's code acts SSL client.