Logging Analytics,

How to Ingest OCI VCN Flow Logs into OCI Logging Analytics

January 22, 2021 | 6 minute read

Mahesh Sharma

Consulting Member of Technical Staff

Text Size 100%:

－＋

On October 6^th 2020, Clay Magouyrk (EVP) along with the Enterprise and Cloud Manageability team announced six new services in the area of Observability and Manageability within Oracle Management Cloud (OCI). The new offerings included Logging, Logging Analytics, Operations Insights, Application Performance Monitoring, Database Management and Service Connector Hub.

In this post, I will show how to ingest Virtual Cloud Network (VCN) flow logs into Logging Analytics Service, by firstly enabling the logs in Logging Service and then using Service Connectors to orchestrate the movement of the logs to the Logging Analytics Service for deeper analytics.

If you are unfamiliar with VCN flow logs, how they are structured, how to interpret them – then have a quick read of this and this.

Log Ingestion flow

The diagram below represents the flow we will be following, from enabling VCN logs in the Logging Service to our end game – analysing in Logging Analytics Service.

Initially, logs will be enabled in OCI Logging Service, then a Service Connector will be used to send logs to Logging Analytics where they will be ingested and analysed.

As a side note, I’ve been asked by many customers on the difference between Logging Service and Logging Analytics. Let me clarify; The Logging Service makes it easy to collect logs emitted from OCI, like Object Storage logs, Function logs, VCN logs and others. Queries can be written to search the logs by narrowing down the times, dates and points of interests very quickly.

Logging Analytics provides the analytics, making it simpler to explore the data, analyse patterns and out-liners, provide machine learning in the form of clustering and linking, create dashboards, provide topology drill-downs and much more. This post highlights some of these capabilities.

To understand how to send VCN flow logs to Logging Analytics, let’s focus on a use case example, which is loosely based on a customer’s requirement.

Use Case Example:

A user (UserA) has restricted access to one compartment (AppA) in a tenancy. The users’ task is to send public subnet VCN logs to Logging Analytics for analysis. The Tenant super admin is responsible for creating compartment AppA, UserA and assigning OCI policies.

Assigning policies will depend on organisational requirements. Below is the policy I used for reference only. The Quick Start Guide for Logging Analytics, Getting Started with Policies and How Policies Work are good references and should be used to understand the policy requirements needed for your organisation.

Compartment level Policy – (group LAadmin contains UserA)
allow group LAadmin to MANAGE virtual-network-family in compartment AppA allow group LAadmin to MANAGE dedicated-vm-hosts in compartment AppA allow group LAadmin to MANAGE metrics in compartment AppA allow group LAadmin to READ audit-events in compartment AppA allow group LAadmin to MANAGE instance-family in compartment AppA allow group LAadmin to MANAGE object-family in compartment AppA allow group LAadmin to MANAGE log-groups in compartment AppA allow group LAadmin to MANAGE log-content in compartment AppA allow group LAadmin to MANAGE loganalytics-resources-family in compartment AppA allow group LAadmin to MANAGE management-dashboard-family in compartment AppA allow group LAadmin to READ users in compartment AppA

Tenancy level Policy
allow group LAadmin to USE cloud-shell in tenancy allow group LAadmin to MANAGE logging-family in tenancy allow group LAadmin to MANAGE serviceconnectors in tenancy allow group LAadmin to READ compartments in tenancy allow group LAadmin to MANAGE loganalytics-features-family in tenancy allow service loganalytics to {LOG_ANALYTICS_LIFECYCLE_INSPECT, LOG_ANALYTICS_LIFECYCLE_READ} in tenancy allow service loganalytics to INSPECT compartments in tenancy allow service loganalytics to USE tag-namespaces in tenancy allow service loganalytics to READ buckets compartment AppA allow service loganalytics to MANAGE cloudevents-rules in compartment AppA

Let’s have a look at tasks UserA needs to perform to get public subnet VCN logs ingested into Logging Analytics.

Create a Log Group in Logging Service,
Enable Logging Service to capture VCN Flow logs.
Create a Log Group in Logging Analytics.
Configure a Service Connector to send logs to Logging Analytics.

Task1 - Create a Log Group in Logging Service.

Logs are always placed into log groups. Think of log groups as virtual containers or folders that can be used to organise logs and restrict access. For example, one log group could contain Function Logs and another log group could contain Audit Logs in the same compartment. A policy can then be used to restrict access to those logs. For example, only users in the auditors' group are permitted to read or delete Audit Logs.
OCI console menu  Solutions and Platform  Logging Select Log Groups.

Select Log Groups
Choose Compartment AppA
Provide a Name (VCN-Logs-AppAA) for the Log Group and an optional description
Select Create Log Group at the bottom.

Task2 - Enable Logging Service to capture VCN Flow logs

In this task, we will turn-on the collection for VCN logs.
OCI console menu  Solutions and Platform  Logging.

Select Logs
Choose Compartment AppA and Log Group VCN-Logs-AppAA.
Click on Enable Service Log.

From the Enable Resource Log panel.

Ensure the Resource Compartment is correct (AppA)
Choose Virtual Cloud Networks under Service
Select subnet under Resource (Public Subnet-AppAA)
Under Log Category select Flow Logs (All records)
Provide a Log Name (PublicVCN-FlowLogsAppAA)
Click Enable Log at the bottom.

Task3 – Create a Log Group in Logging Analytics

As in task 1, log groups are also required in Logging Analytics
OCI console menu Logging Analytics  Administration.

Select the compartment (AppA)
Select Log Groups
Select Create Log Group

On the Create Log Group page

Enter a Name (VCN-LogGroup-AppAA) and description.
Select Create at the bottom.

Task4 – Create Service Connector

The Service Connector is used to orchestrate the movement of logs to Logging Analytics.
OCI console menu  Logging  Service Connectors.

Select the Compartment (AppA)
Click Create Connector

From the Service Connector page

Enter a Connector Name (AppAA-SC) and optional Description
Under Configure Service Connector, select Source as Logging
Target as Logging Analytics
Select Compartment Name (AppA), Log Group (VCN-Logs-AppAA) and Logs (PublicVCN-FlowLogsAppAA)
Under Configuration target connection, select Compartment (AppA) and Log Group (VCN-LogGroup-AppAA)
Select Create to allow a policy to be created for the Service Connector to write to Logging Analytics service
Select Create at the bottom to finish.

View and Analyse logs

To view and analyse your logs, go to:

OCI console menu Logging Analytics Log Explorer

Select Log Group Compartment (AppA)
Set desired Time Range
Select Log Source as OCI VCN Flow Unified Schema Logs

Summary

As shown, in just four simple steps we can start ingesting VCN flow logs into Logging Analytics. From here it's possible to gain real insights into your VCN’s by using features like Linking, Clustering, and Visualisation to identify things like; accepted or rejected connections which may be due to security rules, tracking traffic for abnormal behaviour, identifying trends based on ports and source/destination addresses, and much more.

Additional Resources

Mahesh Sharma

Consulting Member of Technical Staff

Announcing integration of Service Connector Hub with Logging Analytics

Vimal Kocherla | 2 min read

SCAP Support: Secure Linux Configuration with Enterprise Manager

Harish Niddagatta | 2 min read

Resources for

Why Oracle

Learn

What's New