Logging Analytics is the Oracle Cloud Infrastructure (OCI) solution for analyzing and visualizing all your log data from applications and other infrastructure. To continuously collect log data from your entities into the OCI Logging Analytics, you install Management Agents on hosts. Management Gateways can be set up in addition to agents to serve as a single point of communication between Management Agents residing on various hosts and OCI. More efficient uploads can be performed to OCI Logging Analytics using the Management Gateway batching capabilities.  This blog focuses on this more advanced option for setting up log collection.

Using the Management Gateway batching feature to get greater efficiency and more resilient logging uploads to OCI Logging Analytics:

  • Efficient uploads: Multiple data requests to OCI Logging Analytics are batched into a single request and uploaded to OCI Logging Analytics.
  • More resiliency: Ability to retry multiple times during any cloud connectivity issues from customers on-premies collection enviornment.
  • Intelligent batching with optimized upload bandwidth: Network traffic is reduced when batching multiple requests from connected Management Agents into a single request.
  • Highly efficient for small uploads: Provides the ability to batch multiple requests, up to 50 MB, and post it as a single request to OCI Logging Analytics.

How to enable Management Gateway batching

The figure below shows multiple upload requests made by Management Agents to OCI Logging Analytics via Management Gateway. These upload requests are batched into a single/multiple request on the Management Gateway based on log sources/size before it is forwarded to OCI Logging Analytics.

Gateway Batching
Figure 1:  Batching in Management Gateway

 

Configuring Management Gateway Batching

This configuration requires the following:

  1. Install and Configure the Management Gateway with the Batching option enabled.
  2. Install and Configure the Management Agent with Log Collection and Batching options enabled.
  3. Verify that the Agents are communicating with OCI.
  4. Review the Management Gateway batching metrics.

Step 1: Install and Configure Management Gateway with the Batching option enabled

Review the Management Gateway response file parameters and create the Management Gateway response file, including this additional parameter: 
   BatchingEnabled=true

Then, follow the steps outlined in the Install Management Gateway section. 

If you already have a Gateway installed and configured, you only need to perform the following:

  1. Set the BatchingEnabled parameter to “true” in the /opt/oracle/mgmt_agent/plugins/GatewayProxy/stateDir/config/gateway.properties file.
  2. Restart the gateway using this command:

     systemctl restart mgmt_gateway

Step 2: Install and configure Management Agent(s) with Log Collection and Buffering options enabled

Review the Management Agent response file parameters and create the Management Agent(s) response files, including this additional parameter: 
  BufferingEnabled=true

Then, follow the steps outlined in the Install Management Agents section. 

In addition, configure your Management Agents with Logging Analytics. Select the option to “Configure Management Agent service for log collection in this region” but skip the actual installation of agents since you already have performed this task. The rest of the steps will complete the OCI Logging Analytics setup and ensure that logs can be uploaded using Management Agents.

If you already have Agents installed and configured for Logging Analytics, you only need to perform the following:

  1. Set the BufferingEnabled parameter to “true” in the /opt/oracle/mgmt_agent/agent_inst/bin/config/emd.properties file.
  2. Restart the agent(s) using this command:

     systemctl restart mgmt_agent

Step 3: Verify that the Agents are communicating with OCI

  • From the OCI console menu, navigate to “Observability & Management” and select “Management Agents”.
  • Ensure you are in the compartment where the Management Agents are installed using the selector on the left side of the screen. 
  • Click on the Agents menu and review the recently installed agent(s), the host(s) they are installed on, and ensure the Logging Analytics Plug-in is installed.

 

Oracle Cloud Console
Figure 2:  Management Agents appear in the UI with the Logging Analytics Plug-in

 

Step 4: Review the Management Gateway batching metrics

To view the Management Gateway batching metrics navigate to its dashboard. Here are the Management Gateway incoming and outgoing requests.

Management Gateway
Figure 3:  Batching Metrics Representation of Management Gateway

 

Summary

This blog summarizes how to configure and use the Management Agent batching feature to reduce network traffic and obtain more resilient and efficient uploads to OCI.  

Sign up for an Oracle Cloud Infrastructure free trial account today to try out the Management Agent batching feature!

References