Alert and analyze Oracle GoldenGate logs to locate issues faster with OCI Logging, Notifications and Logging Analytics

March 28, 2022 | 18 minute read
Vishak Chittuvalapil
Senior Cloud Engineer
Text Size 100%:

We’ll show how you can source Oracle GoldenGate (OGG) server logs with OCI Logging to proactively notify through OCI Notification when an error is detected in the logs. Then use Service Connector Hub to route the logs into Logging Analytics for detailed analysi and root-cause identification.

You can use the OGG error log to view:

  • See a history of GGSCI commands used
  • View the Oracle GoldenGate processes that started and stopped
  • Understand what processing was performed
  • Check if any errors occurred or if there were any informational and warning messages

Since the OGG error log details events as they occurred in sequence, it is a good method to use for detecting the cause of an error. For example, you might discover that:

  • Someone stopped a process
  • A process failed to make a TCP/IP or database connection
  • A process could not open a file

The are several ways the error log can be natively viewed:

Directing GoldenGate Error Logs From OCI Logging to OCI Logging Analytics

Oracle Cloud Infrastructure Logging Analytics service enables you to index, enrich, aggregate, explore, search, analyze, correlate, visualize and monitor all log data from your applications and system infrastructure.

In this example, where the OCI Logging service is already being used, we will cover how you can use it to also source logs from OGG. OCI Logging service is a fully managed, scalable log management and search platform that simplifies collecting, managing and exploring logs.

1. To get started, you will use the OCI Logging custom log capability, since OGG logs are not supported Out of Box with this service. The Oracle Unified Monitoring agent is included by default on the latest Oracle Linux images. View the instructions in the OCI Logging Documentation on how to install and verify the agent. We can check if the agent is installed and running by executing this via ssh on the server where OGG logs exist.

systemctl status unified-monitoring-agent

The underlying OGG error logs in one of the below OCI compute instance where there is Unified Monitoring Agent running and enabled.          

Figure 1: Checking that unified monitoring agent is running
Figure 1: Checking that unified monitoring agent is running

 

Also, validate that Custom Logs Monitoring is enabled for the same OCI compute instance from the OCI Console Compute Instances page.

 

Figure 2: Ensure custom log monitoring is enabled
Figure 2: Ensure custom log monitoring is enabled

 

2. After the agent is installed, a dynamic group needs to be created. Dynamic groups allow you to assign permissions to Virtual Machines and allow these systems to autonomously perform specific actions against Oracle Cloud Infrastructure services. In this manner you allow applications and systems running on your Virtual Machines to use the Logging Service, for example, to dynamically push logs to the centralized service in near-real-time. See Dynamic Group for more information.

In the OCI Console, from the left side, select the navigation icon (looks like 3 bars), drop down the menu and click Identity & Security and then navigate to Dynamic Groups and select Create Dynamic Group. 

   

Figure 3: Navigate to the Identity page to create a dynamic group
Figure 3: Navigate to the Identity page to create a dynamic group

 

For this example, we will create a rule to include all Virtual Machines within one of the compartments. Fill in and configure as shown and then click the Create button.

Figure 4: Configure and create the dynamic group
Figure 4: Configure and create the dynamic group

 

The next step is to add policies for this dynamic group to grant Virtual Machines to allow the push of logs so the Logging Service can source them. From the OCI console, using the navigation icon, drop down the menu and select Identity & Security then navigate to the Policies page and then select Compartments on the left to verify what Compartment you have access. Then select Policies from the left, and in the Compartment field, drop down the menu selection and choose the Compartment you have access.  Then click the Create Policy button.

On the Create Policy page fill and configure as shown placing this command into the Policy Builder manual editor box

Allow dynamic-group logging dynamic_group to use log-content in compartment XYZ

Note:  Replace XYZ with the compartment name you have access. In this example it was CloudGuardTest

Figure 5: Create the policy
Figure 5: Create the policy

 

3. Next you need to create a Log Group. Log groups are logical containers that allow you to manage and organize your logs. In the OCI Console, from the navigation icon, drop down the menu and select  Observability & Management then under Logging click Log Groups and you will then choose Create Log Group after you first confirm you have chosen a compartment you have permissions to work in. Provide a name and description for your Log Group and click on the Create button to submit.

Figure 6: Creating the Log Group
Figure 6: Creating the Log Group

 

4. From the left side of the Logging page,  click on Logs and then click the Create custom log button

Figure 7: Creating a custom log
Figure 7: Creating a custom log

 

On the Create Custom log page, enter a custom log name, verify you have the same compartment and Log Group that you used earlier (figure 6), then click on the Create custom log button.

Figure 8: Creating a custom log
Figure 8: Creating a custom log

 

5. Next is to create the Agent Config. That will allow you to configure what custom logs you want to ingest across your hosts. Agent configuration is a centralized endpoint for all your instances’ agents to connect to. In each agent configuration, you define a set of dynamic groups that can connect and a set of custom logs from your instance for OCI Logging to ingest. You’ll specify which hosts you want to collect logs from, log inputs, and log destination settings.

On the Create custom log,  Create agent configuration page you will provide the following details:

  • Enter a name in the Configuration field and a description, and select the Compartment you have permissions to work in
  • In the Host Groups section, Dynamic group should be selected and under Group you select the dynamic group you created previously (figure 4). Logging_dynamic_group
  • For the Input Type select Log Path
  • Input Name as goldengateInput (any name you like)
  • Set File paths as /home/opc/ogg/ggserr.log (the path where the ogg error logs exist)
  • Log Group as “OGG_log_group” (created previously)
  • Log name as “Goldengate_log” (created previously)
Figure 9: Creating a custom log, agent configuration
Figure 9: Creating a custom log, agent configuration 

 

Click on the Advanced parser options link to further add the parser and expression for indexing the GoldenGate error logs to an appropriate format in OCI Logging.

As displayed below, set Parser as

REGEXP  

Expression as

/^(?<timestamp>[^ ]*)\s+(?<Severity>[^ ]*)\s+(?<Error ID>[^ ]*)\s+(?<Message>.*)$/

And Time Format as

%Y-%m-%dT%H:%M:%S.%L%z

 

Figure 10: Creating a custom log, agent configuration
Figure 10: Creating a custom log, agent configuration

 

And then click the Save Changes button in the Advanced parser options dialog.

This parser will sort each log entry into 4 fields: timestamp, Severity, Error, and Message.

For example: The sample log below will be indexed as

2021-06-16T13:50:29.953+0000  WARNING OGG-02905  Oracle GoldenGate Capture for Oracle, EXT_E.prm:  Replication of OID column in object tables may diverge

  • Timestamp  2021-06-16T13:50:29.953+0000
  • Severity   WARNING
  • Error ID  OGG-02905
  • Message  Oracle GoldenGate Capture for Oracle, EXT_E.prm:  Replication of OID column in object tables may diverge 

 

6. The GoldenGate logs would then be available by navigating to Logging then Logs then to Goldengate_log. You can click on the Explore with Log Search link to see more