First Steps

Background

This post details the steps needed to install and configure Documaker in a clustered WebLogic environment. These steps were captured using Oracle Documaker Enterprise Edition (ODEE) 12.7.1 April 2024 CPU, WebLogic 12.2.1.4.0, and Oracle Database 19c. Ensure you install the software and prerequisites in accordance with your license agreement(s).

Approach

The approach used to install ODEE depends on your WebLogic environment. The ODEE installation guide presents two methods on page 34 of the guide:

  • Method #1 is used if a WebLogic domain exists and has the appropriate managed servers, clusters, machines, and Coherence clusters. ODEE may be deployed directly into the domain. You will need some information about the domain’s configuration to complete the installation route.
  • Method #2 is a basic installation of ODEE into a new WebLogic domain. You can configure the domain for clustering after the installation is completed.

For the purposes of this post, we will use Method #2 and install into an Oracle Cloud Infrastructure environment using Oracle Enterprise Linux virtual machines (Linux 4.14.35-2047.536.5.el7uek.x86_64 #2 SMP Fri Apr 19 13:44:47 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux using csh)

Understand

It is highly recommended to thoroughly review the Cluster Administration documentation before continuing. In particular, review the following:

  1. Understanding WebLogic Server Clustering
  2. Load Balancing in a Cluster
  3. Failover and Replication in a Cluster
  4. Cluster Architectures
  5. Service Migration
You should plan your cluster architecture before continuing. You should be able to identify a server that will function as the administrative server ("AdminServer"), and how many non-admin cluster members are needed. This document assumes the AdminServer is the first node installed, the cluster admin node, and all other nodes are cluster member nodes.

Software Download

  • Oracle Database 19c can be downloaded from here (Linux) or here (Windows) where you can choose the appropriate installer. We won't cover installation and configuration of the database in this post.
  • WebLogic 12.2.1.4.0 can be downloaded from here. Scroll to the section titled Oracle WebLogic Server 12.2.1.4 and locate the subheading Fusion Middleware Infrastructure Installer and download the package. Transfer the file to your target server and unzip.
  • ODEE 12.7.1 April 2024 CPU is homed in MOS support document 3013639.1. Here you will see the patch numbers associated with the CPU for a specific build version of ODEE. For 12.7.1.4, the patch number is 36508049. Copy and paste this value into the search  bar and press enter. The search results will show recommended links, which include the pages for downloading the CPU update for target operating systems. For our purposes we will use the third link to download 12.7.1.0.0 for Linux, so click the link which opens in a new tab where you can click the Download button.

FMW Installation

Before stepping into Fusion Middleware (FMW) installation, make sure your planned system satisfies the prerequisites for FMW. There are many possible options which are detailed in the FMW 12c Certification Matrix, which covers various components of FMW — our focus is WebLogic server and ADF, which are used by ODEE and its applications. You'll need to have an appropriate JDK, such as 1.8_202 (available here) or 1.8_401 (available here). Follow the appropriate installation instructions for your target operating system as detailed here. Make sure that you install the software as the appropriate user (e.g. "oracle" for Linux, or as Administrator on Windows). In this post we will cover the basic installation details — for more details, see the official FMW installation guide.

Run the installer, which will launch an X-Window for GUI-based installation.

$ java -jar fmw_12.2.1.4.0_infrastructure.jar
  1. Welcome Screen – Click Next.
  2. Auto Updates – Click Next.
  3. Installation Location – enter the desired  directory where the software should be installation, then click Next. In this document, we refer to this location as MW_HOME.
  4. Installation Type – select Fusion Middleware Infrastructure, then click Next.
  5. Prerequisite Checks – the installer will determine if the system prerequisites are met. Address any issues here before continuing. You may receive a warning if you are using an operating system that is new or recently patched, as the installer may not be aware of this version. It is generally ok to proceed with these warnings; however you may wish to ensure that your system matches the certification matrix for best compatibility.
  6. Installation Summary –  Click Save Response File to create a silent installation instruction file – this will be referred to as the RSP file.  Click Install to perform the installation.
  7. Installation Progress – once completed, click the Finish button, and the installer will exit.

Repeating Installations on Additional Nodes

WebLogic must be installed on each machine (node) that will participate in the cluster, so it's a good idea to use a repeatable process to shorten the configuration and setup time. Oracle's Universal Installer provides a silent installation mode that you can use. In Step 6 above, you saved the RSP file which you can use to install WebLogic on all additional nodes that will be in the cluster. Keep in mind that the directory structure on all nodes must be the same, and the RSP file helps follow this paradigm. For each node, run the following, where RSP_FILE is the fully-qualified path/filename of the RSP file from Step 6.

$ java -jar fmw_12.2.1.4.0_infrastructure.jar -silent -responseFile RSP_FILE

The installation will proceed without any interaction as shown in the truncated example below:

$ java -jar fmw_12.2.1.4.0_infrastructure.jar -silent -responseFile /install/fmw12214/install_fmw_12214.rsp

Launcher log file is /tmp/OraInstall2024-06-10_09-06-06PM/launcher2024-06-10_09-06-06PM.log.
Extracting the installer . . . . . . . Done
Checking if CPU speed is above 300 MHz.   Actual 2445.404 MHz    Passed
Checking swap space: must be greater than 512 MB.   Actual 18095 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
Checking temp space: must be greater than 300 MB.   Actual 17179 MB    Passed
Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2024-06-10_09-06-06PM
Log: /tmp/OraInstall2024-06-10_09-06-06PM/install2024-06-10_09-06-06PM.log
Copyright (c) 1996, 2019, Oracle and/or its affiliates. All rights reserved.
Reading response file..
Skipping Software Updates
Starting check : CertifiedVersions
Expected result: One of oracle-6, oracle-7, redhat-7, redhat-6, SuSE-11, SuSE-12, SuSE-15
Actual Result: oracle-8.8
Check complete. The overall result of this check is: Passed
CertifiedVersions Check: Success.
Starting check : CheckJDKVersion
Expected result: 1.8.0_191
Actual Result: 1.8.0_411
Check complete. The overall result of this check is: Passed
CheckJDKVersion Check: Success.
Validations are enabled for this session.
Verifying data
Copying Files
Percent Complete : 10
Percent Complete : 20
Percent Complete : 30
Percent Complete : 40
Percent Complete : 50
Percent Complete : 60
Percent Complete : 70
Percent Complete : 80
Percent Complete : 90
Percent Complete : 100
The installation of Oracle Fusion Middleware 12c Infrastructure 12.2.1.4.0 completed successfully.
Logs successfully copied to /opt/oracle/oraInventory/logs.

Patching

The OPatch utility is used to apply patches to FMW. The README distributed with a patch will indicate the version of OPatch that is required, however sometimes this information is dated and the only method to determine the exact version of OPatch required is to actually run attempt to install the patch. OPatch will error out if a prerequisite is not met.  The ODEE System Requirements guide states on page 12 that "WebLogic 12.2.1.4 with ADF Bundle Patch 12.2.1.4.200603 applied" is part of the requirements for ODEE. As of the time of writing, this patch has already been superseded by ADF Bundle Patch 12.2.1.4.240228.

The download for this patch can be obtained from Oracle Support Document 2659387.1, where you should click the link for Patch 36348444 and then download the file (p36348444_122140_Generic.zip). Once the patch file is downloaded, transfer it to the target environment and unzip it to create a folder "3634844".  Inside this folder you will find the README.txt file which contains instructions for installing the patch, as well as prerequisites. You should follow the instructions in this file, which are presented here in condensed fashion for informational purposes.

Patch 36348444 requires OPatch 13.9.4.2.13 or higher, which is not indicated in the Patch README. You can download OPatch 13.9.4.2.15 (latest at the time of this document) from Patch 28186730 by selecting the appropriate release version from the dropdown at this link. Download the file (p28186730_1394215_Generic.zip) and unzip it into a directory on your target environment to create a folder "6880880". Inside this folder you find a README.txt file which you should review. You should follow the instructions in this file, which are presented here in condensed fashion for information purposes.

  1. Set ORACLE_HOME environment variable, which corresponds to the MW_HOME location from the previous section. The exact method depends on your shell.
$ set ORACLE_HOME=/scratch/fmw
$ echo $ORACLE_HOME
/scratch/fmw
  1. Ensure the bin directory of the JDK is added to the PATH.
  2. Check the OPatch version. If your version of OPatch is not 13.9.4.2.15, proceed with Step 4, otherwise skip to Step 5.
$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 13.9.4.2.15
OPatch succeeded.
  1. Execute the OPatch upgrade from the 6880880 directory, referencing the ORACLE_HOME by path as shown.
$ cd 6880880
$ java -jar ./opatch_generic.jar -silent oracle_home=/scratch/fmw
  1. Apply the ADF Bundle Patch by running OPatch from the 3634844 directory where you unzipped the ADF Bundle Patch. When prompted at two instances, press y. Note: some output has been removed for brevity.
$ cd 36348444/
$ $ORACLE_HOME/OPatch/opatch apply
Oracle Interim Patch Installer version 13.9.4.2.15
...
OPatch continues with these patches:   36348444
Do you want to proceed? [y|n]
Is the local system ready for patching? [y|n]
...
OPatch succeeded.

Repeating Patching on Additional Nodes

You will need to repeat the above steps for each additional node in the cluster.

Repository Creation Utility (RCU)

FMW stores information in a collection of schemas that need to reside in a database (see FMW Certificate matrix above for supported databases). As part of the installation process, you'll need to create these schemas using the Repository Creation Utility (RCU). Note: the schemas created by the RCU are referenced by a prefix that you assign during the installation process. Later, when you create a WebLogic domain, the domain will be linked to the schema you will create here. Each schema can be associated to exactly one domain – when you create the domain, there will be some internal processes that generate information specifc to that domain and populate it to the schema. If for some reason you have to delete the domain and start over, you will need to drop the schemas  and recreate them using the RCU.

Run the installer, which will launch an X-Window for GUI-based installation

$ $ORACLE_HOME/oracle_common/bin/rcu
  1. Welcome Screen – Click Next.
  2. Create Repository – select Create Repository and System Load and Product Load then click Next. Note: if you have to drop and recreate the schema, select Drop here, and complete the process, then restart the RCU and select Create.
  3. Database Connection Details – enter the database  for your database, then click Next. An interim dialog box will pop up as the RCU checks prerequisites – click OK to dismiss this window.
  4. Select Components – provide a prefix (e.g. DEV) for your schema. If there are other schemas located within the database, you can  differentiate them with a prefix. Tick the box for Oracle Platform Security Services, which will automatically select other components. Click Next. An interim dialog box will pop up, dismiss when appropriate.
  5. Schema Passwords – provide a password for the schema objects. You may optionally create different passwords for each schema, however this is not required and you may use a single password for all schemas.
  6. Map Tablespaces – click Next. Several interim dialog boxes will be need to be confirmed to proceed.
  7. Summary – Click Create to proceed. An interim dialog box will show progress, which will self-dismiss when completed.
  8. Completion Summary – click Close to exit the utility.

ODEE Installation

We'll cover a condensed version of the Documaker Enterprise Edition Installation Guide instructions, specifically for Oracle Enterprise Linux. Refer to the Installation Guide for complete details. Copy the downloaded zip file (p36508049_127100_Linux-x86-64.zip) to the target environment, and unzip it to a directory (e.g. 36508049). 

Run the installer from the 36508049/Disk1 directory, which will launch an X-Window for GUI-based installation:

$ cd 36508049/Disk1
$ ./runInstaller

For all screens, the default options are assumed unless otherwise noted. Follow along starting on page 18 of the installation Guide.

  1. Welcome – click Next.
  2. Installation Location – enter a directory (e.g. /scratch/odee) where ODEE will be installed. This will be referred to in this document as ODEE_HOME. Click Next.
  3. Administrator – provide a password for the documaker user, which will be provisioned in the WebLogic security realm. Optionally you can rename the Documaker Administrators group, which will be created in the WebLogic security realm. Click Next.
  4. Database Server – provide connection information for your database where ODEE components will be loaded. Do not use localhost; use the actual host name or IP address.  Note: the load step will be completed later. Click Next.
  5. Administration Schema Details – provide a database username and password. This schema contains ODEE administrative artifacts. Click Next.
  6. Assembly Line Schema Details – provide a database username and password. This schema contains ODEE assembly line artifacts. Click Next.
  7. Application Server – recall that we are following Method #2 from the installation guide, so most of this information must be changed from the defaults.
    1. Set administrative user ID and password
    2. Set Host value = the server host name where you installed FMW
    3. Set Oracle Home to the MW_HOME value (e.g. /scratch/fmw)
    4. Set Project Path = domain main directory (e.g. /scratch/domains)
    5. Set Domain Name = domain name (e.g. odee_domain)
    6. Set Admin Server/Port = WebLogic administrative server name and port (e.g. AdminServer and 9071). Note that the defaults here do not match the defaults that the FMW Domain Config tool uses (which are AdminServer and 7001, respectively, so ensure you maintain continuity of these settings.
    7. The remaining values should be cleared out (Default Server Name, Cluster Name, Machine Name, Coherence Cluster). If Method #1 were to be used (deploying into an existing domain with clusters), you would update the values here appropriately. Click Next.
  8. JMS – modify the provider URL to indicate the hostname of the FMW host. Optionally specify credentials to secure access to JMS services if desired. This user must be defined in the WebLogic security realm.
  9. Hot Folder – accept default and click Next.
  10. SMTP Email Server – accept default and click Next.
  11. UCM – accept default and click Next.
  12. UMS – accept default and click Next.
  13. Installation Summary – click Save to save the response file for later use; then click Install. The installation will proceed. Click Next when available, then click Finish.

Post-Installer Steps

These steps will complete the installation. We will create and load database schemas, and create a WebLogic domain and deploy artifacts. Follow along starting on page 26 of the installation guide. In order to complete these steps as shown, you need to have SYSADMIN access to the database. Some informational outputs have been removed for brevity.

Database Load

Start SQL*Plus from ORACLE_HOME/documaker/database/oracle11g directory. Modify the container option shown below to match your database configuration. Alternatively, you can run these scripts using another SQL tool such as SQLDeveloper.

$ sqplus / as sysdba
SQL> alter session set container=orclpdb1;
SQL> alter session set "_ORACLE_SCRIPT"=true;
SQL> @dmkr_admin.sql
SQL> @dmkr_admin_user_examples.sql
SQL> exit

Reference Implementation MRL Load

If you do not have an MRL to configure, you can use the Reference Implementation MRL using the commands below.

$ set ODEE_HOME=/scratch/odee
$ cd $ODEE_HOME/documaker/mstrres/dmres
$ ./deploysamplemrl.sh

Existing MRL Load

If you have an existing MRL that you want to use with ODEE, you'll need to get your MRL loaded into the dmkr_asline schema, and there are a number of ways to do this. You can do this using Documaker Studio or LBYSYNC. In Documaker Studio you can use the Library > Promote function to select and promote library resources into the target library — but you'll need to define the dmkr_asline schema as a target library in your INIs. This is the preferred option as you have granular control over what resources are promoted, and you can collapse revisions to take only the current version/revision. You can also use the Manage > Tools > Deploy option to do a Library Deployment, which will allow you to define the database connection information to the dmkr_asline schema.

If you want to bypass Documaker Studio altogether, you can use LBYSYNC to dump the contents of an existing library into a flat-file deployment, which you can bring into your ODEE environment. To do this, run the tool in an environment that has an existing configuration, supplying your library name as the /fromlby parameter (replacing DMRES with the library defined in your INI):

lbysync /tolby=deflib/master.lby /fromlby=DMRES /ini=fsiuser.ini /c

The DMRES library is defined in the FSISYS.INI (referenced by the FSIUSER.INI):

< Library:DMRES >
    DBTable                  = DMRESD
    CATALOG                  = DMRESC
    Description              = sample resources
    LBYLogFile               = DMRESL
    USERFile                 = DMRES_DMUSER

After you run this in your existing Documaker environment, you can copy the deflib/master.* files to ODEE_HOME/documaker/mstrres/dmres/deflib directory (take a backup of the existing files first). Then, run the deploysamplemrl script as illustrated in the Reference Implementation section above, and it will load the contents of the master.* files into the dmkr_asline schema.

WebLogic Deployment

Starting in ODEE_HOME/documaker/j2ee/weblogic/oracle11g/scripts, modify the following files using a text editor in your target environment.

set_middleware_env.sh – this file should not need modification, but ensure that the MW_HOME environment variable set at the top of the file is correct. If you have run the installer correctly, it should be fine as-is.

weblogic_installation.properties – this file will need modification, specifically these items. It is recommended to make a back up the file before you change it. Replace '<SECURE VALUE>' including the single quotes with the actual value. This is done for security reasons, so you can ensure that passwords are only stored in plaintext for the duration of this installation session (after which you can delete the modified version).

  1. jdbcAdminPassword – the password for the DMKR_ADMIN schema, from Installer Steps 5 above.
  2. jdbcAslinePassword – the password for the DMKR_ASLINE schema, from Installer Steps 6 above.
  3. jmsCredential – the credential for JMS, from Installer Steps 8 above.
  4. adminPasswd – the Documaker administrative user password, from Installer Steps 3 above.
  5. weblogicPassword – the WebLogic administrative password, from Installer Steps 7.1 above.
Note: if it necessary to rerun the domain creation in the target environment, you will need to run the RCU and drop/recreate the schemas. This is because there are unique values that are generated in the security store tables which cannot be reused. You can run RCU and drop the schemas, then run it again and create the same schema using the same prefix; or just run it once and create a new prefix. This script will prompt you twice if you need to run the RCU, in case you want to drop then recreate.

Run the wls_create_domain.sh script. If this is your first execution of these steps, you can answer n, since you have already run the RCU. If you need to run RCU pursuant to the above note, reply accordingly. After RCU operations complete, or if you answered n, the script will launch the FMW Domain Configuration tool. You can refer to Appendix D of the Installation Guide for screen shots of this tool.

$ ./wls_create_domain.sh
Do you need to run the RCU? (y/n)
n
Running FMW Configuration
  1. Create Domain – select Create a new domain, and change the domain location to the value entered in 7.4 and 7.5 above (e.g. /scratch/domains/odee_domain). This value is referred to as the DOMAIN_HOME. Click Next.
  2. Templates – select Oracle JRF. Click Next.
  3. Administrator Account – enter the Weblogic administrative user ID and password set in 7.1 above. Click Next.
  4. Domain Mode – accept defaults and click Next.
  5. Database Configuration Type – select RCU data, and enter the connection information to your database. The Schema Owner should be the prefix + "_STB" (e.g. DEV1_STB) and password from RCU Screens steps 4 and 5. Click Get RCU Configuration when enabled, and when that process completes, click Next.
  6. Component Data Sources – click Next.
  7. JDBC Test – click Next.
  8. Advanced Configuration – tick the boxes for Administration Server, Node Manager, Topology, and Deployments and Services, then click Next.
  9. Administration Server – modify the server name and listen port to match those from 7.6 above (e.g., AdminServer and 9071). Tick the box for SSL, and modify the SSL listen port to match the regular listen port + 1 (e.g. 9072). Click Next.
  10. Node Manager – select Per Domain Default Location, and provide a node manager user ID and password. This is a credential that will be used to connect to and control the node manager process. Click Next.
  11. Managed Servers – click + Add to create a new managed servers and enable SSL, using ports appropriate for the target environment. Create two managed servers (dmkr_app_n and jms_n) for each cluster member node in the architecture plan. The listen ports for each group (dmkr_app and jms) should be different, e.g. 9202 and 9302 respectively, and the SSL port should likewise be 9203 and 9303.
  12. Clusters – click + Add to create two new clusters as shown. Click Next.
  13. Server Templates – click Next.
  14. Dynamic Servers – click Next.
  15. Assign Servers to Clusters – select all dmkr_app_n in the left pane and dmkr_app_cluster in the right pane, and click the > button. Select all jms_n in the left plane and jms_cluster in the right pane, and click the > button. Click Next.
  16. Coherence Clusters – click Next.
  17. Machines – click Unix Machine tab, click + Add, to create a machine for each node in the cluster architecture, and using the node's hostname as the listen address. Click Next.
  18. Assign Servers to Machines – select the AdminServer, dmkr_app_1 and jms_1 in the left pane, and the first node in the right pane, and click the >. Repeat until each dmkr_app and jms managed has been assigned to the corresponding machine. Click Next.
  19. Virtual Targets – click Next.
  20. Partitions – click Next.
  21. Deployments Targeting – click the Library node in the left pane, and dmkr_app_cluster in the right pane, and click the > button. Click Next.
  22. Services Targeting – click the opss-data-source node in the left pane, and dmkr_app_cluster in the right pane, and click the > button. Click Next.
  23. Configuration Summary – click Create.
  24. Configuration Process – click Next when available. Note the domain location and Admin Server URL, and click Finish.
  25. The FMW Configuration tool will exit, and you will receive another prompt in the domain creation script. Press <enter>.  The script will load WLST and perform configuration, then exit.
Please verify that the weblogic_installation.properties settings match what was set in the FMW Configuration
Press Enter to load ODEE into Weblogic
Initializing WebLogic Scripting Tool (WLST) ...
...
Finished.

Before continuing, you will need to start the Admin Server for the first time. To do this, open a new terminal session and navigate to DOMAIN_HOME (e.g. /scratch/domains/odee_domain) and run startWebLogic.sh. Once you see the RUNNING message, you can continue with step 5.

$ cd $DOMAIN_HOME
$ ./startWebLogic.sh
...
<Jun 10, 2024 5:43:18,984 PM GMT> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>
<Jun 10, 2024 5:43:18,994 PM GMT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>

Back in the other terminal session, run the create_users_groups.sh script (reminder this is in the ODEE_HOME/j2ee/weblogic/oracle11/scripts directory).

$ ./create_users_groups.sh
Initializing WebLogic Scripting Tool (WLST) ...
...
documakerAddDocumakerUsers(): INFO: Finished

Finally, run JPSQuery using a browser or cURL. Note that since we are not installing Documaker Interactive, some demo users/groups will not be found.

$ curl http://localhost:9071/jpsquery/
...
<tr><td>updated : Documaker Administrators</td></tr>
<tr><td>updated : documaker</td></tr>

At this point, Documaker is installed and configured. You can open a browser and connect to the AdminServer to start the JMS Server and then use a terminal session to start DocFactory if you want to do some basic validation before proceeding, however these steps are not documented here; see the Installation Guide page 37-38. Return to the AdminServer terminal session and press CTRL-C to stop the AdminServer.

Cluster Configuration

The configuration presented below is guideline to provide a basic cluster setup. Your specific requirements may necessitate different configuration. In this section, you will complete some additional configuration of the WebLogic domain, and then propagate those changes out to the other WebLogic nodes. It's possible to repeat the procedure, but it's a good practice to get the first node running and make sure you have all your changes in place before replicating the domain across the cluster. For example, if you have to configure external LDAP providers (which is outside the scope of this document), configure specific authentication/authorization requirements, deploy other applications, or configure trust stores, certificates and SSL, you should complete all those activities before getting to the domain packing stage later in this document.

The Documaker Administration and Dashboard applications are typically not internal end-user facing applications, that is, they are typically used by system administrators who are responsible for monitoring and configuring the system, therefore clustering for high-availability and failover is not typically a requirement. The Documaker Interactive application is almost always an internal end-user facing application, and as such clustering for failover and high availability is desirable. Similarly, high availability and failover for the Documaker Web Services (DWS) is a desirable configuration. The instructions herein assume that all Documaker applications and services will be in the cluster.

WebLogic's implementation of JMS provides an independent JMS Server for each node, and uses a mechanism called uniform distributed queues that is meant to ensure all messages are shared across all JMS Servers in the cluster. However, this architecture is not compatible with ODEE's internal messaging services so you should not use uniform distributed queues, or create JMS Servers for every node. Instead, WebLogic provides a separate mechanism called migratable services that allows for the definition of a singleton service that WebLogic will automatically migrate if a failure is detected. ODEE must use this configuration.

In the previous section, you performed some initial domain configuration and deployed the ODEE artifacts to the domain — however, as we are following Method #2, we are not deploying to a set of clusters, so we will need to refactor this configuration so the Documaker applications/web services, data sources, and JMS services are appropriately targeted. While this does involve extra steps, it will also familiarize you with the use of the WebLogic console. Begin with all WebLogic services on the cluster admin node shut down.

Applications/Web Services and Data Sources

  1. Start the Node Manager as shown here.
  2. Start the AdminServer as shown here.
  3. With the AdminServer running, open a browser and connect to the console (e.g. https://localhost:9072/console). Note: since this server is running in demonstration mode, you will likely receive warning from the browser about the SSL Certificates. This is expected until you replace the certifications with non-demonstration versions.
  4. In the Domain structure panel, expand Services and select Data Sources. Perform the following steps for dmkr_admin and dmkr_asline.
    1. Click the name.
    2. Click the Targets tab.
    3. Check dmkr_app_cluster_1. The default will target all members of the cluster, which is our desired choice. Click Save.
    4. Click Summary of JDBC Data Sources in the breadcrumb trail.
  5. In the Domain Structure panel, select Deployments.
    1. Perform the following steps for each of these applications: DocumakerAdministrator, DocumakerDashboard, and DWSAL1.
      1. Click the Name.
      2. Click Targets tab.
      3. Tick the check box for type Enterprise Application and click Change Targets.
      4. Check the dmkr_app_cluster box, then click Yes.
      5. Click Summary of Deployments in the breadcrumb trail.
    2. Perform the following steps for each of these applications: DocumakerAdministratorHelp and DocumakerDashboardHelp
      1. Click the Name.
      2. Click Targets tab.
      3. Check the app_cluster_1 box, then click Save.
      4. Click Summary of Deployments in the breadcrumb trail.

JMS Services

Next we need to configure the JMS cluster for migration and set up the migration policy. There are different options available to you, such as setting up database migration basis and other options, which are explored more fully in the documentation. Your requirements and policies may dictate a different configuration than what is presented here.

  1. Open a browser window to the WebLogic console (https://HOSTNAME:9072/console).
  2. Expand and click Domain Structure > Environment > Clusters > jms_cluster
  3. Click the Migration tab
  4. Change Migration Basis to Consensus.
  5. Add all node_n candidate machines to the Chosen section by clicking >>.
  6. Click Save.
  7. Expand and click Domain Structure > Environment > Clusters > Migratable Targets > jms_1.
  8. Click the Migration tab.
  9. Change the Service Migration Policy to Auto-Migrate Exactly-Once Services.
  10. Click the >> under Constrained Candidate Servers to move all jms_n to the Chosen section.
  11. Click Save.

Persistent Storage

Next we need to configure a new persistent store, which we will then configure for our JMS modules. There are different options available to you, such as setting up JDBC persistence, which are explored more fully in the documentation. Your requirements and policies may dictate a different configuration than what is presented here.

  1. Expand and click Domain Structure > Services > Persistent Stores, and click New > Create FileStore.
  2. Name the store jms-filestore and use the directory jms-filestore, then click Next..
  3. Click the Targets tab, and select jms_1 (migratable), then click Finish.
  4. Expand and click Domain Structure > Services > Messaging > JMS Servers > AL1Server.
  5. Change Persistent Store to jms_filestore. Click Save. Note you may see an error; this is expected.
  6. Click the Targets tab.
  7. Change the target to jms_1 (migratable), then click Save. You may see an error. Click Activate Changes in the upper left — not all changes will be active at this time.

JMS Modules

  1. Expand and click Domain Structure > Services > Messaging > JMS Modules > AL1Module.
  2. Click the Targets tab and uncheck jms_server_1_1 and check jms_cluster > All servers in the cluster. Click Save; an error may appear.

At this point, if you still have any unactivated changes shown in the Change Center in the upper left, click Activate Changes, and all errors should go away. You will still need to restart for some changes to take effect. You can use NodeManager or the Console to restart. To use NodeManager, open a terminal window and run FMW_HOME/oracle_common/common/bin/wlst.sh, then use the nmConnect() command. Once connected, issue nmKill('AdminServer'). To restart the AdminServer, use: startServer('AdminServer','DOMAIN_NAME','t3://HOSTNAME:9071','WEBLOGIC_USER','WEBLOGIC_PASS')

Domain Packing/Unpacking

With the primary components of the domain set up and configured, it is now time to pack the domain and propagate it to the members of the cluster. 

  1. In a terminal session, run the following command. Replace upper case elements with values suitable for your environment. Note: the -template argument is the path/filename where the domain output (a JAR file) should be located.
$ MW_HOME/oracle_common/common/bin/pack.sh -domain=DOMAIN_HOME -template=/scratch/odee_domain.jar -template_name="ODEE CLUSTER" -template_author="DOCUMAKER" -managed=true
<< read domain from "/scratch/domains/odee_domain"
>>  succeed: read domain from "/scratch/domains/odee_domain"
<< set config option Managed to "true"
>>  succeed: set config option Managed to "true"
<< write template to "/scratch/odee_domain.jar"
..............................
>>  succeed: write template to "/scratch/odee_domain.jar"
<< close template
>>  succeed: close template
  1. Copy the template (JAR file created by the pack command) to the target server(s) in your cluster.
  2. Unpack the domain. Replace upper case elements with values suitable for your environment. Note: the -template argument is the path/filename of the packed domain that you copied.
$ MW_HOME/oracle_common/common/bin/unpack.sh -template=/scratch/odee_domain.jar -domain=DOMAIN_HOME -overwrite_domain=true
<< read template from "/scratch/odee_domain.jar"
>>  succeed: read template from "/scratch/odee_domain.jar"
<< set config option DomainName to "odee_domain"
>>  succeed: set config option DomainName to "odee_domain"
>>  validateConfig "KeyStorePasswords"
>>  succeed: validateConfig "KeyStorePasswords"
<< write Domain to "/scratch/domains/odee_domain"
...........................................................................
>>  succeed: write Domain to "/scratch/domains/odee_domain"
<< close template
>>  succeed: close template
  1. On each of the member nodes, perform the following to enroll the node's NodeManager with the AdminServer. Note that the AdminServer must be running on the cluster admin node. Replace the upper case elements with values for your system.
$ MW_HOME/oracle_common/common/bin/wlst.sh
...
wls:/offline> connect('WEBLOGIC_ADMIN','WEBLOGIC_ADMIN_PWD','t3://ADMIN_SERVER:9071')
...
wls:/odee_domain/serverConfig/> nmEnroll('DOMAIN_HOME','DOMAIN_HOME/nodemanager')
Enrolling this machine with the domain directory at /scratch/domains/odee_domain ...
Successfully enrolled this machine with the domain directory at /scratch/domains/odee_domain.
wls:/odee_domain/serverConfig/> exit()

Start NodeManager

NodeManager must be running on all nodes within the cluster, so start it using the same command on all nodes. Note that this example uses nohup to allow the terminal session to exit and leave the service running. You may want to configure Node Manager as a service for resiliency and automated startup. Replace the upper case elements with values appropriate for your environment..

$ nohup DOMAIN_HOME/bin/startNodeManager.sh &
...
<time/date> <INFO> <Secure socket listener started on port 5556, host <hostname>/<ip>>

Start AdminServer

Start WLS, and issue the nmConnect() command to connect to the NodeManager on the primary node. Then, start the AdminServer on that node. Replace the elements shown in upper case with appropriate values for your environment.

$ $MW_HOME/oracle_common/common/bin/wlst.sh
...
wls:/offline> nmConnect('NODEMANAGER_USER','NODEMANAGER_PASS','HOSTNAME','5556','DOMAIN_NAME','DOMAIN_HOME')
Connecting to Node Manager ...
...
wls:/nm/odee_domain> startServer('AdminServer','DOMAIN_NAME','t3://HOST_NAME:9071','WEBLOGIC_USER','WEBLOGIC_PASS')
Starting server AdminServer ...
<time/date> <INFO> <odee_domain> <AdminServer> <The server 'AdminServer' is running now.>
Successfully started server AdminServer ...

Start Managed Servers

Using either the WebLogic Console or WLST, start the managed servers on all nodes.

To start up the managed servers on all nodes:

  1. Open a browser to the WebLogic console (e.g., https://<host>:<port>/console),  and login with your WebLogic Admin credentials.
  2. Navigate to Domain Structure > Environment > Servers.
  3. Click the Control tab.
  4. Tick the boxes next to each managed server you wish to start (e.g. dmkr_app_1, dmkr_app_2, dmkr_app_n…), then click Start. The console will send start commands to each node's NodeManager to start the managed servers. You can click the Refresh button and the webpage will periodically update.

Or, to start managed servers using WLST, use the following:

$ $MW_HOME/oracle_common/common/bin/wlst.sh
wls:/offline> nmConnect('NODEMANAGER_USER', 'NODEMANAGER_PASS', 'HOSTNAME', '5556', 'DOMAIN_NAME', 'DOMAIN_HOME')
Connecting to Node Manager ...
wls:/nm/odee_domain> startServer('dmkr_app_1', 'odee_domain', 't3://'HOSTNAME:9071', 'WEBLOGIC_USER', 'WEBLOGIC_PASS')
Starting server dmkr_app_1 ...
Successfully started server dmkr_app_1 ...
wls:/nm/odee_domain> nmDisconnect()

Repeat for each node in the cluster, connecting to its node manager and issuing the startServer command.

Load Balancing

WebLogic supports load balancing in a number of different configurations, which can be reviewed here. In general, you will need an external load balancing component that is either hardware-based (BIG-IP, F5) or software-based with a web server that can host the proxy plugin (Apache, NGINX, etc.). This will allow users to access clustered services and applications with a single URL that is not mapped to a specific node/port. WebLogic also provides a basic Servlet that can be configured as a front-end, however it is generally advised to use an industry-standard solution as mentioned above. For more information on setup and configuration of the proxy plugin with a web server, see here.

ODEE Clustering

The previous instructions went through all the necessary configuration to set up a WebLogic cluster for managed servers and services that host ODEE applications. The remaining effort is to configure the processing tier where ODEE's DocFactory runs — this effort is very simple. 

  1. Identify additional machines where you want to run ODEE, and ensure they are configured with the appropriate prerequisites.
  2. Copy the ODEE_HOME directory from the first node where ODEE was installed to all target nodes. Keep in mind that directory structures must match – for example, if the ODEE_HOME directory is /scratch/odee on server A, then it must also be in /scratch/odee on server B.
If you make changes in the ODEE_HOME on one server that affects files on the file system remember to propagate changes to all other ODEE nodes as there is no automated propagation mechanism!

Also keep in mind that the majority of ODEE's settings reside in the database, so setting changes made therein will affect all the nodes. For example, if you are changing scaling settings, it is important to make sure that all ODEE nodes are relatively homogenous with respect to available memory, CPU, and disk space, or at least ensure you configuring scaling to be 100% automatic and that upper bounds are relative to the specifications of your least-capable node. If you tuned your performance on your best node and the other nodes are not as capable, then your overall system performance will likely be less performant than expected as the lesser nodes will not be as capable. Conversely, if you tune performance to the least capable node then you will have unused capacity on more capable nodes. Therefore, keep your nodes as similar as possible.

Depending on your system configuration and requirements it may be advantageous to consider deploying the Receiver on only one node, especially if using older versions of ODEE. If you are using hot folder directories, it is recommended to use a shared file system only for the hot folder and not for the rest of the ODEE filesystem (otherwise performance suffer to an excruciating degree). Additionally, you may only want to run the Receiver on a single node if hot directories are used to prevent file contention (especially for operating systems like Windows that do not provide a file locking flag), so use performance testing to validate what works for your environment. To enable this configuration you will have to remove ODEE_HOME/documaker/docfactory/deploy/receiver.jar from all but one node. You may also want to do this for the Scheduler as well (scheduler.jar) so multiple instances of this worker are not constantly checking for work in the database.

Wrapping Up

If you've made it this far, congratulations for sticking around! There's a lot to digest here, so I recommend taking your time and reading this thoroughly, and have a look at some of the reference materials I've linked below. If this seems daunting, Oracle Consulting is here to help. Happy clustering!