Capture and analyze Oracle WebLogic logs using Oracle Cloud

In this blog I will explain how to configure OCI Logging for a WebLogic domain and ingest its logs into OCI Logging Analytics for analytics purposes.

This blog will just cover instances inside OCI. If you need to ingest logs from a host on-premises or from another cloud provider you need to use the management agent .

High-level architecture diagram for WebLogic logging

In this architecture, the OCI Unified Monitoring Agent collects and parses (if needed) custom logs in compute instances and sends them to the OCI Logging service. You can do a simple search inside log entries in OCI Logging, but if you need analytics or enterprise-wide Log Analytics solution, send the log files to OCI Log Analytics using the OCI Service Connector Hub.

In the next part, I will explain in detail how to implement this solution inside OCI.

Configuring OCI Logging for a WebLogic domain and ingesting logs into OCI Logging Analytics for analytics purposes.

Here are the steps that you need to follow to implement this solution inside your OCI tenancy (I assume you already provision networking resources and install and deploy your WebLogic cluster on OCI compute instances).

1. Install fluentd-based agent

To ingest events from your applications into your custom log, you can install the Oracle fluentd-based agent. This agent allows you to control exactly which logs you want to collect, how to parse them, and more.

Oracle Cloud Infrastructure Logging provides an easy mechanism (Agent Configurations) to enable and manage the agent for a set of supported operating systems. Agent Configurations give you a central experience to easily configure what custom logs you want to ingest across your fleet of hosts.

Depending on the type of your instance (new OCI instance, existing OCI instance, or instance created from custom image) you should follow different steps to install and configure the monitoring agent (please follow the agent management documentation)

After installation and configuration of the agent on the compute instance, you can check the status of an instance using the following command.

sudo systemctl status unified-monitoring-agent

You can enable the monitoring agent when you provision a new Compute instance. Simply select “Show Advanced Options” in the “Create Compute Instance” screen in the OCI console and select the “Oracle Cloud Agent” tab. Then you can enable the “Custom Logs Monitoring” plugin.

You can also enable the “Custom Logs Monitoring” plugin for existing compute instances by navigating to the instance details page (in the OCI console) and select the “Oracle Cloud Agent” tab (this plugin is enabled by default for Compute instances).

Keep in mind, the monitoring agent is more than just a log collector agent. It collects performance metrics, installs OS updates, and performs other instance management tasks.

2. Configure agent and logging service

After agent installation, create the agent configuration and select the following information:

• Which hosts do you want to collect logs from
• Exactly which logs you want to ingest from those hosts
• A Log group/log destination

Log groups are logical containers for organizing logs. Logs must always be inside Log groups. You must create a Log group to enable a log.

Before you perform the agent configuration, you should create an OCI Dynamic Group that includes all the OCI instances from which you want to collect logs from.

To create a Dynamic Group, In the OCI console, from the upper left navigation icon, navigate to Identity & Security -> Dynamic Groups inside OCI console and then create a new Dynamic Group with following matching rule:

ANY {instance.id = 'ocid1.instance.<region>.<location>.<unique_ID>',
instance.compartment.id = 'ocid1.compartment.<region>..<unique_ID>'}

You can specify instance OCID and compartment OCID of all WebLogic instances in the matching rule.

All {instance.compartment.id = 'ocid1.compartment.<region>..<unique_ID>'}

Or simply specify the compartment OCID that includes all WebLogic instances (if this compartment only includes WebLogic instances).

You should also create a policy to grant permission to the Dynamic Group that you created earlier to use log-content in the tenancy.

To create the policy, in the OCI cloud console, navigate to Identity & Security -> Policies and create a new policy with the following statement:

allow dynamic-group <dynamic_group_name> to use log-content in tenancy

After creating the policy and Dynamic Group, you should create the Log group and the custom log. Then in the OCI console, navigate to Observability & Management -> Log groups and create a new Log group and populate the compartment, name, and description for the Log group fields.

Next, select the Logs menu item and create a new custom log and populate the compartment, custom log name, and Log group fields.

Now we can create the agent configuration. In the OCI cloud console, navigate to Observability & Management -> Agent Configuration and create the new Agent Configuration in the compartment and populate the agent name, description, compartment, dynamic group fields that you created in the previous step and WebLogic log location (you can use a wildcard in the path):

WebLogic Server logs by default write inside {Domain_Home}/servers/{Server_Name}/logs/{Server_Name}.log file. But you can change this path in the WebLogic console. We are using the default file path in this blog.

Finally, you should log in to all WebLogic Compute instances and restart the agent using the following command:

sudo systemctl stop unified-monitoring-agent
sudo systemctl start unified-monitoring-agent

After a few minutes, open the custom log you have created in the above step, and you can see WebLogic logs in the explore log screen.

3. Configure Service Connector Hub

The Service Connector Hub orchestrates data movement between services in Oracle Cloud Infrastructure and with 3rd-party products and services. A Service Connector specifies the source that contains the data to be moved, optional tasks, and the target for delivery of data when tasks are complete. An optional task might be a function task to process data from the source or a log filter task to filter log data from the source.

In this step, we will create a Service Connector to send data from the OCI Logging service to the OCI Logging Analytics service.

To create a service connector, in the OCI console, navigate to Observability & Management -> logging -> Service Connectors and create a new Service Connector and populate the name, description, compartment, source, and target fields (in this example the source is the Logging service, and the target is Logging Analytics)

You should also provide a Log group and custom log for the Logging service and a Log group for the Logging Analytics service (we should create this Log group first and I will explain how to do that in the next section).

We also can add a log filter task (to filter log files from the source) and a configuration function task to process data before it is sent to the target (I will not cover those configurations in our example).

4. Configure the Logging Analytics service

The Oracle Cloud Infrastructure (OCI) Logging Analytics service automates the collection of historic and real-time logs from any on-premises or cloud resource. It is a highly scalable, reliable, and real-time log analysis solution.

In the last section, we will configure Logging Analytics and create a simple report for WebLogic logs using the Log Explorer.

First, to enable Log Analytics service in your region, in the OCI console, navigate to Observability & Management -> Logging Analytics and select “Start Using Logging Analytics”.

If you are not the OCI Administrator, you will need some policies and permissions set up before you use the Logging Analytics service (please refer to Logging Analytics documentation).

Now you can navigate to the Logging Analytics, Log Explorer page. First, we need to create a Log group for the Logging Analytics service. Select the Administration menu and then select Log groups from the Administration page and create a new Log group for Logging Analytics in the specific compartment you have access (you used this Log group when you create the Service Connector in step 3).

Navigate to the Log Explorer and select Log Source under the Field’s tab and column select “OCI Unified Schema Logs”.

In the Visualizations tab select the “Records” drop-down selection. Note, you can also add and remove display fields.

Now you can see WebLogic Server logs in the Logging Analytics Log Explorer page.

You can search the log content or create cluster visualization (Clustering uses machine learning to identify patterns from the log records, and then groups the logs that have a similar pattern) eliminating similar log entries. “noise”

For example,  search for the word “Start” in all log contents

Then view a Cluster visualization. (icon looks like a paw print)

In Cluster mode, the result in groups of similar records will be displayed.

5. Logging Analytics and Oracle Enterprise Manager

For those who have Oracle Enterprise Manager in their business, and use it to manage applications like Oracle WebLogic Server, you can obtain other visualizations and insights using log data.

Using the EM Bridge you can build out and explore your Weblogic Server Cluster logs and different components and further enrich the log files to do more in your business. Come back for blogs on that topic or join our February 10th webcast "Improve Troubleshooting with Oracle Enterprise Manager Data & OCI Logging Analytics" to learn more about it (will be available on demand in case you read this after Feb 10th).

OCI Logging Analytics is a comprehensive product with many capabilities not covered (out of scope for this blog) See other Observability blogs on Logging Analytics and the product documentation for more information. https://docs.oracle.com/en-us/iaas/logging-analytics/index.html

I hope this blog helps you get control of all your different log files and helps you get additional insight from them and solve problems faster.

References to learn more

• OCI Logging Service
• OCI Logging Analytics
• OCI Service Connector Hub
• EM Bridge