Kubernetes application pod logs contain critical event, state, and diagnostic information for your containerized and serverless applications. Traditionally, collecting and centrally managing application pod log data from a busy Kubernetes cluster has posed a challenge. The Kubernetes project documentation recommends that your containers export log messages through native stdout or stderr buffers. Worker node processes natively collect and manage these message streams.

The Oracle Cloud Infrastructure (OCI) Logging service enables you to easily ingest and consolidate log data from your applications running in Kubernetes clusters, including the Oracle Container Engine for Kubernetes (OKE). OCI virtual machines (VMs) automatically deploy with a lightweight agent for collecting log information from host processes. You can easily configure this agent to collect custom application log data, as shown in the following diagram:

A graphic depicting the architecture for the worker node connected to OCI Logging service through the Unified Monitoring Agent.

In this blog, I show how you can easily enable centralized container log ingestion and consolidation for applications running on OKE. This solution also applies to nonmanaged Kubernetes clusters running on virtual machines within OCI.

Implement OKE application log management

  1. Deploy your OKE cluster in a compartment that has a dynamic group defined. Dynamic groups allow you to delegate specific permissions to processes running on OCI virtual machines. Create a dynamic group configured with a matching rule for Compute instances deployed within this compartment. For example, add the following statement to your dynamic group in the matching rules section, substituting your compartment OCID:

    instance.compartment.id = '<compartment_ocid>'

    A screenshot of the Matching Rules page in the Console with the rule filled in.

  2. Create a policy to allow the dynamic group to interact with the Logging service. The Unified Monitoring Agent is a lightweight agent running on all worker nodes that uses assigned permissions to download configuration updates and upload log content. Ensure that your dynamic group has been assigned the following permission policy, substituting your group and compartment names:

    Allow dynamic-group <my-Group> to use log-content in compartment <target_compartment_name>

    A screenshot of the Policy Builder section in the Console with the permission filled in.

  3. Create your log group. Log groups are logical containers for organizing and managing logs. To enable or create logs, first create a log group, which is fast and easy. Navigate to Observability and Management in the Console, and under Logging, select Log Groups. Ensure that the compartment selected matches the location of your OKE cluster.

    A screenshot of the Create Log Group window with the name and description filled in.

  4. Create a custom log and agent configuration. An agent configuration provides instructions for both the Logging service and a specific group of deployed agents to work together. Custom logs require an agent configuration. Follow these steps to create an agent configuration that includes importing app container log messages from the OKE cluster

    Navigate to Observability and Management, and under Logging, select Logs. Under List Scope and Compartment, choose your target compartment. Then select Create Custom Log to open the Create panel. Again, ensure that your target compartment is specified then select the Create custom log button.

    The Create Agent Configuration panel is displayed, where you can create a configuration. In Group Type, select Dynamic Group, and in the group box, select the group that you identified in step 1.

    Your screen might give you the option to create a policy to allow resources to use the Logging service. This process was already performed in step 2, but click Create to dismiss the prompt.

    A screenshot of the configuration page with the caution window in yellow at the bottom with the Create button outlined in yellow.

    In the Configure Log Inputs section, choose Log Path for the input type, okeapplogs for the input name, and for file paths, use the following path:

    /var/log/pods/default_*/*/*.log

    This file path is the default for application containers in the current versions of OKE. If you want to include all container logs, including system and service containers, remove “default_” from the path and use the following path:

    /var/log/pods/*/*/*.log

    A successful agent configuration matches the following image:

    A screenshot of the Agent Configuration page with the details filled in.

    To complete the task, click Create Custom Log. An initial ingestion delay of a few minutes can occur.

  5. Locate and review your log. Navigate to Logging Service, Logs, and select your custom log that you created in step 4. This selection opens the Explore Log page where you can review log entries, search log content using keywords or expressions, and set up alerts to trigger notifications.

    A screenshot of the Explore Log page with the Filter by Time menu highlighting the past 5 minutes.

Summary

OCI Logging provides a centralized log management solution for your dynamic Kubernetes applications. Logging is one of the Always Free cloud services offered on Oracle Cloud Infrastructure. Try OCI Logging service today!

Oracle Container Engine for Kubernetes is an Oracle-managed container orchestration service that can reduce the time and cost of running Kubernetes applications. This orchestration service is free, and you can qualify for 4,500 free hours of compute infrastructure with free trial credits.