Log Consolidation in Documaker Enterprise Edition

In an out-of-the-box installation, Oracle Documaker Enteprirse Edition ("ODEE") comes preconfigured with default logging options, which include: 

• Assembly Line Worker logging into the <ODEE_HOME>/documaker/docfactory directory, specifically in the /temp and /logs subdirectories.
• Transactional logging into the LOGS and/or ERRS tables in the Assembly Line database tables.
• Assembly Line Docupresentment instance logging into the <ODEE_HOME>/documaker/docupresentment directory.
• Depending on the installation's operating system, additional logging may be made to the Windows Event Log or nohup logging on Linux systems.

Over the lifetime of a system, the components of the system will undergo many changes at a low level such as form or resource changes, as well as high level changes such as scaling or migration. Additionally, changing environment parameters such as changing user bases, new users, or changed/new system policies may result in changes to the components. Whenever change occurs, testing is critical to ensure continued system performance to its requirements. Log detail is crucial to the ability to detect and diagnose problems or characteristics of a system. Other factors may contribute to the need to change the logging configuration, such as security constraints (e.g. restricted access to the file system or Documaker Dashboard) or a desire for collaborative diagnostics. Owing to the differences between each implementation, as well as the different requirements and best practices from one organization to another, it is difficult to suggest a "one-size-fits-all" logging solution – so, from a product perspective, each component is configured with the aforementioned out of the box logging setup so that you can modify it for your particular needs. What follows is an exploration into log consolidation for enterprise-wise logging scenarios.

You may or may not know that with the ability to configure each component's logging path was introduced in Patch 29340765: ODEE COMPONENT LOG CONSOLIDATION starting with ODEE 12.6.1 onwards. The feature allows full configuration of the log path, including remote file system mounts with dynamic directory naming. This makes diagostic work much easier for multi-node systems (e.g. systems where there are more than one DocFactory machines for a given Assembly Line) and allows for wider access controls when the log file system is separated from the runtime or configuration filesystem. 

As mentioned above, the installation process provides a default logging configuration out of the box. To enable log consolidation, we'll need to take some extra steps during the implementation process to configure the necessary components. A summary of the steps is shown below:

1. Define the Consolidated Log Path.
2. DocFactory 
   1. GENDATA Worker processes
   2. Non-GENDATA Worker processes
   3. Service/Process Configuration
3. Docupresentment
   1. Service/Process Configuration
   2. Logging Configuration in logconf.xml

Consolidated Log Path

Defining the consolidated log path will be different for each implementation, and may even be different for each environment or assembly line. To complete this step, you will need to know the desired location of all log files and make sure that the path is available as a mounted filesystem on each DocFactory node in the ODEE system. Hereafter, we'll call this the "log file path".

DocFactory

GENDATA Worker Processes

The GENDATA Worker processes inlude Assembler, Distributor, and Presenter. Configuration of these workers is done via Documaker Administrator. 

1. Login to Documaker Administrator, expand the desired Assembly Line, then select the Assembler and click Configure.

2. Expand the STARTUP-CFG Context/Category, and select the Assembler Group.

 In the list of Properties that follows, locate the property name env.JVM_OPTIONS. The value of this property will vary based on your system and contains multiple comma-delimited parameters that are passed to the worker process. Specifically we are interested parameter -Dlog4j.file=<some path>. Modify the path to include the log file path defined in the previous step. If you have a multi-node environment, include the ~HOSTNAME variable. This will be replaced at runtime with the hostname of the node where DocFactory is running. Examples:

• -Dlog4j.file=/mount/logfiles/documaker/assembler/logs/log4j.log
• -Dlog4j.file=/mount/logfiles/~HOSTNAME/documaker/assembler/logs/log4j.log

Take care only to modify the -Dlog4j.file parameter and its value. Click the Save icon to record the change. 

Expand the LOG4J-Logger context/category and select the ErrorLogger group. Click the plus icon n the Properties panel, and add a property named appender-ref with the value process-roll. Select the LogLogger group, and repeat the addition of the new property.

Finally, select the LOG4J-Appender context/category, and select the process-roll group. Create a property called File with a value  ~FILE and save the change. This ensures that the JVM option we modified earlier (-Dlog4j.file) is passed to the logging configuration.

Modify the <ODEE_HOME>/documaker/mstrres/dmres/FSIUSER_n.INI files. Open each file and locate the following line

< DATA >
TraceFile = ~MRLDIR ../../docfactory/logs/docfactory_<worker_name>~PROCESSID .log

Change this to reflect the log file path and optionally hostname, e.g.

< DATA > 
TraceFile = /mount/path/to/logs/~HOSTNAME /docfactory_<worker_name>~PROCESSID .log

Repeat the entirety of the configuration steps above for the Distributor and Presenter workers.

Non-GENDATA Worker processes

The Non-GENDATA Worker process include all the workers except Assembler, Distributer, and Presenter.  That is, the Supervisor, Scheduler, Receiver, Identifier, Batcher, Archver, Publisher, PubNotifier, and Historian. Configuration of these workers is done via Documaker Administrator. 

1. Login to Documaker Administrator, expand the desired Assembly Line, then select a non-GENDATA worker and click Configure.

2. Expand the LOG4J-Appender context/category and click the group process-roll. Modify the File property to include the log file path defined earlier. If you have a multi-node environment, include the ~HOSTNAME variable. Click Save. Repeat for the group roll. An example is shown below:

Repeat the above configuration for all non-GENDATA workers.

Service/Process Configuration

The service or process configuration will vary depending on your operating system.

Linux

When running DocFactory on the Linux operating system, you have several options available to you as to how the processes are started and stopped. This is largely dictated by the Linux distribution, your Linux system administrators, and your organizations common practices. For Oracle Enterprise Linux, the typical method is to employ systemd and systemctl. There are many primers on these systems, so you'll want to do some reading if you're not familiar with them. Under this scheme, a unit file for a DocFactory Assembly Line is created in the service directory, e.g. /usr/lib/systemd/system. The unit file might look like this:

[Unit]
Description=Assembly Line 1 - Docfactory
After=wls_nodemanager.service

[Service]
Type=forking
WorkingDirectory=/opt/oracle/odee/documaker/docfactory/bin
ExecStart=/opt/oracle/odee/documaker/docfactory/bin/docfactory.sh start
ExecStop=/opt/oracle/odee/documaker/docfactory/bin/docfactory.sh stop
User=oracle
Group=dba
KillMode=process
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Note the ExecStart and ExecStop options should point to the Assembly Line's Oracle Home, and the associated docfactory.sh file. Also note that the After setting indicates that this service should be started after the wls_nodemanager.service. Since the JMS services are required for DocFactory to run, we've configured WebLogic NodeManager to start up via systemd as well. The unit file should be saved with a useful name, such as docfactory_asline1, so that you can issue commands using the name e.g. $ sudo systemctl start docfactory_asline1

Next, we need to ensure that any logging generated by the service is output into our consolidated logfile path. To do this, we should modify the docfactory.sh script mentioned in our unit file. Using an appropriate text editor for Linux such as vi or nano, open the docfactory.sh file for editing.

Search for this section in the file:

     LD_PRELOAD=${LD_PRELOAD_SAVE:-""}
     export LD_PRELOAD
     if [ -z ${*:-""} ] ; then
        mkdir ../logs 2>/dev/null
        nohup ${JAVA_EXE} ${JAVA_ARGS} | tee &
        #${JAVA_EXE} ${JAVA_ARGS} &
        #${JAVA_EXE} ${JAVA_ARGS}  >>../logs/docfactory_console.log 2>&1 &
        #${JAVA_EXE} ${JAVA_ARGS}  2>&1 >>../logs/docfactory_console.log 2>&1 | tee -a ../logs/docfactory_console.log &
     else
        mkdir ../logs 2>/dev/null
        nohup ${JAVA_EXE} ${JAVA_ARGS} ${*:-""} | tee &
        #${JAVA_EXE} ${JAVA_ARGS} ${*:-""}  &
        #${JAVA_EXE} ${JAVA_ARGS} ${*:-""}  >>../logs/docfactory_console.log 2>&1 &
        #${JAVA_EXE} ${JAVA_ARGS} ${*:-""} 2>&1 >>../logs/docfactory_console.log 2>&1 | tee -a ../logs/docfactory_console.log &
     fi

You'll notice there are several options already provided for you to redirect log output; the appropriate setting depends on your specific system. In most cases, a simple redirect to a new file is sufficient – consult with your Linux sysadmin for advice. You'll need to make the following edits:

1. Take a backup of the file.
2. Comment out or remove the nohup command and unused ${JAVA_EXE} lines in this block.
3. Create a variable with the log file path.
4. Uncomment the appropriate ${JAVA_EVE} line
5. Modify the uncommented line with the log file path.
6. If using a multi-node system, create a variable called NODEHOST and set it, then use that variable in the log file path. 

An example is shown below with relevant changes highlighted:

     LD_PRELOAD=${LD_PRELOAD_SAVE:-""}
     export LD_PRELOAD
     LOGFILEPATH=/mount/path/to/logs
     NODEHOST=`hostname`
     if [ -z ${*:-""} ] ; then
        mkdir $LOGFILEPATH 2>/dev/null
        #nohup ${JAVA_EXE} ${JAVA_ARGS} | tee &
        #${JAVA_EXE} ${JAVA_ARGS} &
        ${JAVA_EXE} ${JAVA_ARGS}  >> $LOGFILEPATH/$NODEHOST/docfactory_console.log 2>&1 &
        #${JAVA_EXE} ${JAVA_ARGS}  2>&1 >>../logs/docfactory_console.log 2>&1 | tee -a ../logs/docfactory_console.log &
     else
        mkdir $LOGFILEPATH 2>/dev/null
        nohup ${JAVA_EXE} ${JAVA_ARGS} ${*:-""} | tee &
        #${JAVA_EXE} ${JAVA_ARGS} ${*:-""}  &
        ${JAVA_EXE} ${JAVA_ARGS} ${*:-""}  >> $LOGFILEPATH/$NODEHOST/docfactory_console.log 2>&1 &
        #${JAVA_EXE} ${JAVA_ARGS} ${*:-""} 2>&1 >>../logs/docfactory_console.log 2>&1 | tee -a ../logs/docfactory_console.log &
     fi

The end result looks like this:

Now, when starting DocFactory, your logging output will be written to the appropriate location and directory.

Also make sure that we are adding the new log file path -Djdbc.log.dir property in JAVA_ARGS

This will redirect startup.log tothe new folder instead of <odee_home>/docfactory/logs folder

Windows

To create a Windows Service for the DocFactory, you can run the <ODEE_HOME>/documaker/docfactory/bin/docfactory_supervisor_install.bat file, which will create the service using the properties defined in the docfactory_supervisor.properties file in the same directory. You'll need to modify the properties file before running the BAT file as shown below:

1. Locate the value service.jvm.args.length=X and increment that value by 1. Note the resulting number, which we'll call Y.
2. Add a new value,  service.jvm.args.Y=-Djdbc.log.dir=<log file path>
3. If you're using a multi-node environment, include the ~HOSTNAME variable in your path, e.g. service.jvm.args.Y=-Djdbc.log.dir=y:\\logs\\~HOSTNAME
4. Save the file, then run the BAT file to create the Windows Service.

Docupresentment

Service/Process Configuration

The service or process configuration will vary depending on your operating system.

Linux

The options for running Docupresentment as a service are similar to DocFactory. You can create a unit file for Docupresentment using the same instructions for DocFactory, noting that the ExecStart and ExecStop settings will be different:

ExecStart=/opt/oracle/odee/documaker/docupresentment/docserver.sh start
ExecStop=/opt/oracle/odee/documaker/docupresentment/docserver.sh stop

Next, we need to ensure that any logging generated by the service is output into our consolidated logfile path. To do this, we should modify the docfactory.sh script mentioned in our unit file. Using an appropriate text editor for Linux such as vi or nano, open the docserver.sh file for editing. Take a backup first!

Search for this section in the file:

if [ -z ${*:-""} ] ; then
  nohup ${JAVA_EXE} ${JAVA_ARGS} | tee &
else
  nohup ${JAVA_EXE} ${JAVA_ARGS} ${*:-""} | tee &
fi

Then modify as shown below:

Windows

To create a Windows Service for the Docupresentment, you can run the <ODEE_HOME>/documaker/docupresentment/bin/docfactory_ids_service.bat file, which will create the service using the properties defined in the idswdservice.properties file in the same directory. You'll need to modify the properties file before running the BAT file as shown below:

1. Locate the value service.jvm.args.length=X and increment that value by 1. Note the resulting number, which we'll call Y.
2. Add a new value,  service.jvm.args.Y=-Djdbc.log.dir=<log file path>
3. If you're using a multi-node environment, include the ~HOSTNAME variable in your path, e.g. service.jvm.args.Y=-Djdbc.log.dir=y:\\logs\\~HOSTNAME
4. Save the file, then run the BAT file to create the Windows Service.

Logging Configuration in logconf.xml

Finally, modify the <ODEE_HOME>/documaker/docupresentment/logconf.xml file.  Locate the each occurrence of text like <param name="File" value="[path/to/a/file]" /> there will be multiple occurences. Modify each to reflect the log file path, and if necessary, the ~HOSTNAME variable. This will require some attention as the attributes may not be presented in order of name then value, and some are repeated. An example is shown below of only the affected lines after modification, using the ~HOSTNAME option for multi-node systems.

<param name="File" value="/mount/paht/to/logs/~HOSTNAME/docupresentment/~PREFIXwatchdog.log"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/~INSTANCE-error_tracker.log" name="File"/>
<param name="File" value="/mount/paht/to/logs/~HOSTNAME/docupresentment/~PREFIX~THREADID.log"/>
<param name="File" value="/mount/paht/to/logs/~HOSTNAME/docupresentment/~INSTANCE-diagnostic.log"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/~INSTANCE.log" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/~INSTANCE-trantime.log" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/receive.msg" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/receive-dsimessage.msg" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/send-dsimessage.msg" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/send.msg" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/receive-http.msg" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/send-http.msg" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/receive-http-dsimessage.msg" name="File"/>
<param value="/mount/paht/to/logs/~HOSTNAME/docupresentment/send-http-dsimessage.msg" name="File"/>

Save the configuration XML file with your changes and, if necessary, restart the Docupresentment service.

A Word About Database Logging

The out of the box configuration of ODEE is such most logging information is written into the LOGS and ERRS tables. The Dashboard application is similarly configured, as it can present log/error information in the context of jobs and transactions processed by ODEE. However, in some situations you may not wish to have logging information written to the database. For example, in a performance test environment, the increased loads typically associated with performance testing may be negatively impacted by significant logging output. Your organizational standards for logging may be such that consolidation to centralized files is all that is needed, so database logging is redundant. Or, access to the database to perform queries on the LOGS and ERRS tables may not be granted. In these cases, it may be desirable to turn off database logging — with the caveat that the Dashboard will no longer be able to display logging information. 

To turn off database logging:

1. Login to Documaker Administrator and configure the Assembly Line.
2. Expand LOG4J – Appenders and select Appender.  Deactivate the LogAppender and ErrorAppender. Click Save.

Optional Step

In ODEE_HOME/documaker/docfactory/config, modify the log4j.xml for the Supervisor by commenting-out all references to LogAppender and ErrorAppender — generally there are two references to each appender. Save the changes.

<!--        <appender-ref ref="LogAppender"/>
        <appender-ref ref="ErrorAppender"/>
-->

In ODEE_HOME/documaker/docfactory/deploy, modify the log4j.xml in each worker JAR and perform the same configuration.  Take a backup of JAR files in deploy directory first!

For Linux users, you can run the following one-liner in the deploy directory in a Bash shell:

For Windows you can just use WinZip or 7-zip to modify the log4j.xml in place.

Enjoy, and happy logging!

Nile and Andy