If you’re deploying Oracle Documaker Enterprise Edition (ODEE), you’re likely managing a complex set of batch and interactive workloads. Observability—being able to monitor, trace, and diagnose system behavior—is essential. In this post, we’ll explore how to instrument both the WebLogic tier and DocFactory workers for full telemetry coverage using open-source tools. We’ll also walk through a hands-on setup using OpenTelemetry, Jaeger, and Prometheus to collect and analyze metrics and traces.

System Architecture Overview

ODEE is a multi-tiered platform for document automation. Each layer plays a unique role:

The WebLogic Tier hosts user-facing applications and integration endpoints:

• Dashboard, Administrator, Interactive: For managing jobs and reviewing output.
• DWS (Documaker Web Services): SOAP/XML endpoints for job submission.
• JMS (Java Message Service): Buffers and routes work items to the processing tier.

WebLogic handles resource pooling, thread management, and JDBC connections, making it a critical observability surface.

The Processing Tier contains the DocFactory Workers (e.g., Assembler, Presenter, Archiver) that run as standalone JVMs outside WebLogic. They:

• Pick up jobs from JMS
• Retrieve config/data via JNDI
• Perform document composition, data mapping, batching, distribution, and presentation.

Since they’re outside the app server container, these JVMs must be instrumented independently for logging, metrics, and tracing.

The Database Tier hosts the database and acts as the system of record. It stores:

• Administrative tables
• Transactional tables
• LOGS: Informational and diagnostic events
• ERRS: Structured error records

These tables are essential for job auditing and error triage.

 Logging

The WebLogic Tier handles logging via the WebLogic Diagnostic Framework (WLDF):

• Plain text, XML, or JSON logs
• Accessible via filesystem, JMX, or REST
• Compatible with: ELK Stack, Azure Log Analytics, Splunk, Fluentd

In the Processing Tier, DocFactory workers use Reload4J (log4j-compatible) to write logs to:

• Oracle DB (LOGS, ERRS)
• Optional local disk
• Crash trace dumps (rare)

This centralization makes postmortem debugging far easier across distributed workers.

Metrics

The WebLogic Tier provides metrics that can be gathered using:

• JMX
• WLDF Harvesters
• SNMP

Supported exporters and tools include:

• Prometheus (via JMX Exporter)
• Grafana
• Dynatrace OneAgent
• Azure Monitor
• Oracle Enterprise Manager

In the Processing Tier, each worker exposes JMX metrics including information such as:

• Heap usage
• GC activity
• Job lifecycle events
• Restart thresholds (e.g., JMXMemoryChecks)

These metrics can be collected using Prometheus, visualized in Grafana, or sent to tools like Dynatrace/New Relic.

Tracing

In the WebLogic Tier, tracing can be achieved via:

• WLDF Execution Context IDs
• OpenTelemetry Java Agent
• Dynatrace OneAgent

Workers in the Processing Tier do not have native tracing factility, however you can use:

• Job IDs in logs and ERRS
• JMS correlation IDs
• OpenTelemetry agent injection at the JVM level

This bridges gaps across tiers, giving you full request lineage.

Alerting and Proactive Monitoring

• WebLogic: Use WLDF policies for alerting on memory, stuck threads, etc. Hooks available for JMX and SNMP.
• DocFactory: Supports restart-on-threshold via JMX, plus structured logs for external parsers.
• Easily integrated with PagerDuty, Jira, Opsgenie, and ServiceNow.

Conclusion

For complete observability, instrument both WebLogic and DocFactory workers and enable structured logs, JMX metrics, and OpenTelemetry agents. Then you can generate actionable insights using tools like Prometheus and Jaeger. Done right, this allows faster troubleshooting, proactive alerting, and a smoother support experience across environments.

Tutorial: Emitting and Capturing Telemetry

Let’s walk through setting up telemetry collection for ODEE using OpenTelemetry + Prometheus + Jaeger. Note that the latter two items are available as binaries or containers. For the purposes of this tutorial, we'll use binaries on Linux.

• OpenTelemetry Agent: Injected into each JVM to emit traces and metrics. Download the Agent JAR file from here.
• Jaeger: Collects and displays trace data via browser UI. Grab the Jaeger All-In-One distribution for version 2.x from here.
• Prometheus: Scrapes JVM metrics and supports PromQL-based dashboards.  Download the distribution from here.

Jaeger setup is easy – just unzip it and run it. Modify the command line shown below for your hostname/IP, and desired ports (however if you do change these ports, you will have to change them elsewhere too, so I recommend using the defaults for now.  Adjust paths where necessary.

/opt/jaeger/jaeger-all-in-one \
--collector.grpc-server.host-port <hostname_or_ip>:14250 \
--query.grpc-server.host-port <hostname_or_ip>:16686 \
--collector.zipkin.host-port=:9411

Prometheus setup is similar. Unzip it to a directory, modify the configuration file, and run it. Modify the configuration file (prometheus.yml) as shown below. Note that addresses here (e.g. localhost:9465) refer to the DocFactory worker instances and ports. If they are running on another machine, modify localhost to reflect the hostname or IP for that machine. If the DocFactory workers are running on a remote machine, make sure the ports are available through any firewalls.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: []  # You can fill this in if using alertmanager

rule_files: []

scrape_configs:
  - job_name: 'docfactory-workers'
    static_configs:
      - targets: ['localhost:9465']
        labels:
          worker: 'Scheduler'
      - targets: ['localhost:9466']
        labels:
          worker: 'Receiver'
      - targets: ['localhost:9467']
        labels:
          worker: 'Identifier'
      - targets: ['localhost:9473']
        labels:
          worker: 'Assembler'
      - targets: ['localhost:9474']
        labels:
          worker: 'Distributer'
      - targets: ['localhost:9475']
        labels:
          worker: 'Presenter'
      - targets: ['localhost:9468']
        labels:
          worker: 'Batcher'
      - targets: ['localhost:9469']
        labels:
          worker: 'Archiver'
      - targets: ['localhost:9470']
        labels:
          worker: 'Publisher'
      - targets: ['localhost:9471']
        labels:
          worker: 'PubNotifier'
      - targets: ['localhost:9472']
        labels:
          worker: 'Historian'

Run Prometheus via command line (adjust paths where necessary):

/opt/prometheus/prometheus \
  --config.file=/opt/prometheus/prometheus.yml \
   --storage.tsdb.path=/opt/prometheus/data \
   --web.listen-address=:9090

Next, we'll inject the OpenTelemetry Agent into each DocFactory worker. Copy the Agent JAR file into the <ODEE_HOME>/documaker/docfactory/lib directory. Then, using Documaker Administrator, modify each worker's JVM Options to add the Agent configuration flags after any existing options. The configuration flags used are the same for all workers, however some values will differ. The format of the flags differs based on worker type: Assembler/Distributer/Presenter use comma-delimited flags and the remaining workers use space-delimited flags. In Administrator, select each Worker, and click Configure. Then expand the STARTUP – CFG context/category, and click the worker name. 

For Assembler, Distributer, and Presenter, locate env.JVM_Options property, and add the flags after existing options. Note that these flags should be comma-delimited with no spaces (here they are shown on individual lines for clarity/comments; do not include the comments in the property value).  For all other workers, locate the JVMOptions property, and add the flags after existing flags. These flags are space-delimited. 

-javaagent:/scratch/odee13/documaker/docfactory/lib/opentelemetry-javaagent.jar, # the fully path to the Agent jar file
-Dotel.service.name=docfactory-assembler,                                        # the name of this Agent; should match the worker (e.g. docfactory-receiver, docfactory-identifier, etc.)
-Dotel.resource.attributes=worker_id=Assembler                                   # set worker name, e.g. Assembler, Receiver, Identifier, etc.
-Dotel.traces.exporter=zipkin,                                                   # enable the zipkin exporter in the Agent
-Dotel.exporter.zipkin.endpoint=http://localhost:9411/api/v2/spans,              # configure the Jaeger zipkin collector. If Jaeger runs on a different host, indicate here. 
-Dotel.metrics.exporter=prometheus,                                              # enable the Prometheus exporter in the Agent
-Dotel.exporter.prometheus.port=9473,                                            # port for Promtheus exporter; unique to each Worker; this must match the prometheus.yml 
-Dotel.exporter.prometheus.canonical=true,                                       # additional exporter configuration.
-Dotel.logs.exporter=none                                                        # additional exporter configuration.

Launch and Observe

Start your DocFactory workers, then visit the dashboards provided by Jaeger and Prometheus.

For Jaeger UI, open a browser to http://<host>:16686, then select a worker from the Service dropdown (you'll notice these are the names you provided in the Agent configuration flags via JVM options). You can select any of the operations from the dropwn (the screenshot below shows JOBS table inserts). Modify the lookback timeframe if needed, depending on how busy your system is or is not. Click Find Traces. Any results will be graphed and tabled for comparison or further drill down.

If you click a trace, you can view additional details, such as the query that was run (note that queries are parameterized for safety and privacy, and no parameters are shown). You can also view some database connection details, however secure values are not shown, only hostname/service names and server address/port.

For Prometheus UI, open a browser to http://<host>:9090, then enter a PromQL statement, such as sum by (worker) (jvm_thread_count) and click Execute, then click the Graph tab. This will show the graphed results of metrics captured that match the query. In the screenshot below, I've dragged to select a window of time that shows activity (you can also enter a window of time such as 5m1s before 2025-05-20 10:22:01 to grab a specific slice of time for your results. The data is graphed, and you can hover over data points to review what's going on. Here we see that at 10:19, the Receiver had 71 threads running to process incoming jobs and transactions. You can also mouse-over worker feeds in the legend below to highlight the graphed data.

With this setup in place, you now have end-to-end observability for your Oracle Documaker environment—metrics from every JVM, traceability across service boundaries, and real-time dashboards to monitor performance and health. Whether you’re debugging a bottleneck, analyzing throughput, or proactively watching for system failures, the combination of OpenTelemetry, Jaeger, and Prometheus gives you the visibility you need to keep DocFactory running smoothly. Observability isn’t just about troubleshooting—it’s about insight, confidence, and control.

As always, feel free to follow up in the Documaker forums!

-Andy