Introduction

Verrazzano Enterprise Container Platform is a hybrid multicloud Kubernetes-based Enterprise Container Platform for running both cloud-native and traditional applications.

The Oracle Cloud Infrastructure (OCI) Logging service is a highly scalable and fully managed single pane of glass for all the logs in a tenancy.

OCI customers running Verrazzano likely will want to ship their logs to OCI Logging so they have a single place to view and analyze all of their log data across all of their applications and services in OCI.

This blog describes how to set up and configure the OCI Logging service to collect logs from application and system containers running in a Verrazzano cluster in OCI.

Prerequisites

These instructions assume you have the appropriate permissions to create resources in an OCI compartment (compute instances, log groups, etc.) as well as Identity resources, specifically a dynamic group for your compute instances and an IAM policy that allows the dynamic group hosts to use logs in the compartment.

Also you will need a running Kubernetes cluster in OCI. For this blog post, I used an Oracle Container Engine for Kubernetes (OKE) cluster. For instructions on how to create an OKE cluster for use with Verrazzano, see the Verrazzano platform setup instructions for OKE.

While this article shows how to configure and view logs using the OCI Console, it is also possible to use the OCI Command Line Interface (CLI) or the OCI Software Development Kit (SDK) to achieve the same results.

Create Dynamic Group and IAM Policy

First create a Dynamic Group for the compute instances running the Kubernetes cluster. For example, to include all compute instances in a compartment, create a Dynamic Group with a matching rule for the compartment. Substitute your compartment OCID.

Next create an IAM Policy that allows the dynamic group to send logs to OCI. It should look something like this:

Note that the dynamic group name in the policy statement must match the name of the dynamic group you created previously. Also, the named compartment must be the same compartment that your compute instances run in.

Create OCI Logging Resources

To enable logging on the hosts you will need to create three OCI resources: a Log Group, a Custom Log, and an Agent Configuration. In the OCI Console, you will find the logging resources by selecting the Logging section from the Observability & Management menu.

Log Group

Log groups provide a way to organize logs as well as limit who has access to logs. Log instances must be part of a Log Group. Create a Log Group like so:

Custom Log

All log records are associated with an OCI Log resource. Create a Log that will collect your Verrazzano log records:

When you click “Create custom log” a screen (or dialog) appears in which you can create the Agent Configuration.

Agent Configuration

The Agent Configuration tells the monitoring agent where to find the logs on the compute instances and how to parse the records. OKE writes container logs to /var/log/containers/ with a string timestamp in each record. Create the following Agent Configuration. If you are using a different type of Kubernetes cluster, then you will need to adjust some of these values accordingly.

Select the dynamic group you created earlier. Your OCI user needs an IAM policy to allow you to read dynamic groups so that you can select the dynamic group. If the group does not appear in the drop-down menu, then you may be missing a required IAM policy.

Select “Log path” for “Input type”. For the “File paths” field, enter “/var/log/containers/*.log”. Do not click “Create custom log” just yet. You still need to set the parser options. Your Agent Configuration should look similar to this:

Select the “Advanced parser options” link. Select “JSON” for the parser and “String” for the time type. Enter “%Y-%m-%dT%H:%M:%S.%NZ” in the time format field.

Click “Save Changes” and finally click “Create custom log”.

Install Verrazzano

If you have not already done so, then create a Kubernetes cluster in OCI. Install Verrazzano following the instructions here. Optionally deploy one or more of the Verrazzano example applications.

Search Verrazzano Logs

It generally takes a few minutes from the time a log record is written to disk on the host until it shows up in OCI Logging. View the logs in the OCI Console by navigating to the Log you created earlier. Under the “Explore Log” section you will see log records. 

Select the “Explore with Log Search” link to open the editor and then click “Show Advanced Mode”. Use the OCI Logging Query Language to filter and sort log records.

Example Log Search Queries

To filter log records to the kube-system and verrazzano-install namespaces, add a subject filter to the query:

To see all log records that have the OAM component name equal to “robert-coh”, add a filter on “logContent”:

If you installed the Bob’s Books example application, then you will see matching log records.

Conclusion

The approach described here provides a relatively simple way to collect Verrazzano logs in OCI Logging without having to make any changes to the Verrazzano cluster. However, there are limitations. For example, if you want to separate system and application logs so different users or roles have access to different types of logs, then you need to explicitly set the log source file paths per namespace (or per pod). In dynamic Kubernetes environments where application namespaces come and go, that can be challenging.

Another limitation is that all of the Verrazzano clusters must be running in OCI. For hybrid environments (multicloud, on premises, etc.), this approach will not work for clusters residing outside of OCI.

In a future blog post, we will show you how to overcome these obstacles.