Many organizations are faced with challenges when it comes to managing log data in cloud environments.  Effective monitoring and analysis of log data are critical for ensuring system performance, security, and reliability. Oracle Cloud Infrastructure Logging Analytics (LA) provides powerful tools for collecting, analyzing, and visualizing log data across various services. Especially in large-scale environments, this simplifies the process of deploying and configuring the Logging Analytics plugin manually.

Automating the deployment of OCI Logging Analytics plugins offers a streamlined and consistent approach to enable comprehensive observability across all instances.

This blog is a step-by-step guide automating the LA plugin deployment on OCI Compute instances while performing autoscaling or existing compute instances where management agents are already active.

Oracle Cloud Agent manages plugins in OCI

Oracle Cloud Agent is a lightweight process that manages plugins running on compute instances, also known as hosts or VMs, residing in the Oracle Cloud Infrastructure. When you are using compute instances, you can deploy Management Agents by using the Oracle Cloud Agent. For information about deploying Management Agents with Oracle Cloud Agent on Oracle Cloud Infrastructure compute instances, see Deploy Management Agents on Compute Instances.

Note: Enabling Management Agent from the OCI Compute Instance is supported only on Linux images. To get more details about OCI Autoscaling please refer to this documentation.

Create an OCI Instance configuration to simplify and automate Compute instances

OCI Instance configuration helps to simplify and automate the deployment and management of Compute instances in a standardized and consistent manner. When you are creating the instance, make sure the management agent is enabled as part of the instance configuration. This will allow the agent to be enabled when the new instance gets launched as part of the autoscaling.

 

Figure-1: Enable Management Agent Plugin
Figure 1: Select the Management Agent Plugin

 

Add the cloud-init script to allow read access for the user oracle-cloud-agent on the log files. For example, the /var/log/messages logs. If the files already have read permission for the agent user, this step is not required.

 

Figure:2 Cloud Init Script
Figure 2: Include the cloud-init script

 

Enable auto-association for a Linux source

Enablng the auto-association for the Linux Syslog Logs source will auto-associate Linux hosts with this log source pointing to /var/log/messages.

Navigate to the Observability and Management main menu, select Logging Analytics Administration, and then Sources.

 

Linux Syslogs Log Source
Figure 3: Linux Syslogs Source

 

Define an OCI function to deploy the LA plugin

We will use the OCI function to deploy the LA plugin as part of the Management Agent when new instances are getting launched. Please refer to the function documentation on how to deploy a function in OCI.

OCI Function code 
import oci
import io
import time

def list_active_agents(plugin, compartment_ocid, management_agent_client):
    response = management_agent_client.list_management_agents(
        compartment_id=compartment_ocid,
        lifecycle_state="ACTIVE",
        availability_status="ACTIVE",
        platform_type=["LINUX"],
        plugin_name=plugin,
        is_customer_deployed=False,
        install_type="AGENT",
        compartment_id_in_subtree=False,
        access_level="ACCESSIBLE")
    return response.data

def handler(ctx, data: io.BytesIO = None):
    try:
        cfg = dict(ctx.Config())
        compartment_id = cfg['compartment_id']
        enable_logging = cfg['enable_logging']
        time.sleep(60)
        print(f"Sleeping for 60 seconds to give time for agents to become active",flush=True)
        print(f"Function config for enable logging: {enable_logging}",flush=True)
signer = oci.auth.signers.get_resource_principals_signer()
        management_agent_client = oci.management_agent.ManagementAgentClient({}, signer=signer)
        la_agent_list = []

        for la_agent in list_active_agents(["logan"], compartment_id, management_agent_client):
            la_agent_list.append(la_agent.id)

        all_agent_list = []
        if enable_logging.upper() == "ALL":
            for all_agent in list_active_agents(["dbaas", "jm", "jms", "appmgmt", "opsiHost", "osmh", "logan", "None"],
                                                compartment_id, management_agent_client):
                all_agent_list.append(all_agent.id)
        else:
            for all_agent in list_active_agents(["None"],
                                                compartment_id, management_agent_client):
                all_agent_list.append(all_agent.id)

        deploy_list = set(all_agent_list) - set(la_agent_list)
        print(f"LA plugin Deploy list of agents:{deploy_list}",flush=True)
        list_management_agent_plugins_response = management_agent_client.list_management_agent_plugins(
            compartment_id=compartment_id,
            lifecycle_state="ACTIVE",
            display_name="Logging Analytics",
            platform_type=["LINUX"])
        la_plugin_id = list_management_agent_plugins_response.data[0].id
        if len(deploy_list) > 0:
            deploy_plugins_response = management_agent_client.deploy_plugins(
                deploy_plugins_details=oci.management_agent.models.DeployPluginsDetails(
                    plugin_ids=[la_plugin_id],
                    agent_compartment_id=compartment_id,
                    agent_ids=list(deploy_list)))
            print(deploy_plugins_response.status)
    except Exception as ex:
        print(str(ex), flush=True)

 

Once the function is deployed, configure the function in the OCI console.

Navigate to the OCI menu, select Developer Service, select Functions and then Applications. Under Applications, click the function name and select the configuration. Add the below two properties to the configuration.

The compartment_id is the agents compartment id.

The enable_logging when set to All will enable Logging Analytics for all instances in the compartment where the management agent is active. If set to any other value, it will only enable Logging Analytics for the active agents where there is no plugin deployed.

 

Functions
Figure 4: Create and cofigure OCI functions

 

Create an Event to trigger the function and launch a new instance

 

Create an Event rule to trigger the function when a new instance is launched. Navigate to the main menu, then choose Observability and Management, then Event Service, and click on Rules.

Event Details
Figure 5: Event Details

 

Select the action type as Functions and add the function created previously.

Create an instance using the instance configuration

Now, create the instance pool with the instance configuration created above including the number of instances needed as the starting point. It will launch the compute instances, and you will see the event getting triggered in Events Service metrics.

 

Autoscaling Proerties
Figure 6: The Autoscale Configuration

 

Events Delivered
Figure 7: Events Delivered Chart

 

Validate the LA plugin for active management agents

The OCI LA plugin will be deployed for the active management agents. Validate this by navigating to the Observability and Management and select the Agents and Gateways menu.

 

LA Plugin Details on Agent
Figure 8:  Agent & LA Plugin Details

 

Navigate to Loging Analytics, then Log Explorer and you will see the logs for the new host launched. For further testing, we will create autoscaling based on a schedule at a defined time.

The newly launched instance will have Logging Analytics enabled, the LA plugin deployed, and the logs will automatically be collected in Logging Analytics.

Syslog Visualization
Figure 9: Linux Syslogs Visualization

This feature helps enable the LA plugin in bulk and automates future instance creation.

Note, management agent will take some time to go into a SILENT state from ACTIVE state. Don’t invoke the script or function immediately after instances were terminated.  To invoke the script immediately, manually delete the agents associated with the terminated instances.

Start testing today by starting your free OCI trial.

Resources