X

The latest cloud infrastructure announcements, technical solutions, and enterprise cloud insights.

How to ingest OCI VCN flow logs into OCI Logging Analytics

Mahesh Sharma
Security & Cloud Manageability
This is a syndicated post, view the original post here
On October 6th 2020, Clay Magouyrk (EVP) along with the Enterprise and Cloud Manageability team announced six new services in the area of Observability and Manageability within Oracle Management Cloud (OCI).   The new offerings included Logging, Logging Analytics, Operations Insights, Application Performance Monitoring, Database Management and Service Connector Hub.

 

In this post, I will show how to ingest Virtual Cloud Network (VCN) flow logs into Logging Analytics Service, by firstly enabling the logs in Logging Service and then using Service Connectors to orchestrate the movement of the logs to the Logging Analytics Service for deeper analytics.

If you are unfamiliar with VCN flow logs, how they are structured, how to interpret them – then have a quick read of this and this.

Log Ingestion flow

The diagram below represents the flow we will be following, from enabling VCN logs in the Logging Service to our end game – analysing in Logging Analytics Service.

Initially, logs will be enabled in OCI Logging Service, then a Service Connector will be used to send logs to Logging Analytics where they will be ingested and analysed.

As a side note, I’ve been asked by many customers on the difference between Logging Service and Logging Analytics. Let me clarify; The Logging Service makes it easy to collect logs emitted from OCI, like Object Storage logs, Function logs, VCN logs and others.  Queries can be written to search the logs by narrowing down the times, dates and points of interests very quickly.

Logging Analytics provides the analytics, making it simpler to explore the data, analyse patterns and out-liners, provide machine learning in the form of clustering and linking, create dashboards, provide topology drill-downs and much more.  This post highlights some of these capabilities.

To understand how to send VCN flow logs to Logging Analytics, let’s focus on a use case example, which is loosely based on a customer’s requirement.

Use Case Example:

A user (UserA) has restricted access to one compartment (AppA) in a tenancy. The users’ task is to send public subnet VCN logs to Logging Analytics for analysis. The Tenant super admin is responsible for creating compartment AppA, UserA and assigning OCI policies. 

Assigning policies will depend on organisational requirements.  Below is the policy I used for reference only.  The Quick Start Guide for Logging Analytics, Getting Started with Policies and How Policies Work are good references and should be used to understand the policy requirements needed for your organisation.

Compartment level Policy – (group LAadmin contains UserA)

allow group LAadmin to MANAGE virtual-network-family in compartment AppA
allow group LAadmin to MANAGE dedicated-vm-hosts in compartment AppA
allow group LAadmin to MANAGE metrics in compartment AppA
allow group LAadmin to READ audit-events in compartment AppA
allow group LAadmin to MANAGE instance-family in compartment AppA
allow group LAadmin to MANAGE object-family in compartment AppA
allow group LAadmin to MANAGE log-groups in compartment AppA
allow group LAadmin to MANAGE log-content in compartment AppA
allow group LAadmin to MANAGE loganalytics-resources-family in compartment AppA
allow group LAadmin to MANAGE management-dashboard-family in compartment AppA
allow group LAadmin to READ users in compartment AppA

Tenancy level Policy

allow group LAadmin to USE cloud-shell in tenancy
allow group LAadmin to MANAGE logging-family in tenancy
allow group LAadmin to MANAGE serviceconnectors in tenancy
allow group LAadmin to READ compartments in tenancy
allow group LAadmin to MANAGE loganalytics-features-family in tenancy
allow service loganalytics to {LOG_ANALYTICS_LIFECYCLE_INSPECT, LOG_ANALYTICS_LIFECYCLE_READ} in tenancy
allow service loganalytics to INSPECT compartments in tenancy
allow service loganalytics to USE tag-namespaces in tenancy
allow service loganalytics to READ buckets compartment AppA
allow service loganalytics to MANAGE cloudevents-rules in compartment AppA

 

Let’s have a look at tasks UserA needs to perform to get public subnet VCN logs ingested into Logging Analytics.

  1. Create a Log Group in Logging Service,
  2. Enable Logging Service to capture VCN Flow logs.
  3. Create a Log Group in Logging Analytics.
  4. Configure a Service Connector to send logs to Logging Analytics.

 

Task1 - Create a Log Group in Logging Service.

Logs are always placed into log groupsThink of log groups as virtual containers or folders that can be used to organise logs and restrict access. For example, one log group could contain Function Logs and another log group could contain Audit Logs in the same compartment. A policy can then be used to restrict access to those logs. For example, only users in the auditors' group are permitted to read or delete Audit Logs.
OCI console menu Solutions and Platform Logging Select Log Groups.

  • Select Log Groups
  • Choose Compartment AppA
  • Provide a Name (VCN-Logs-AppAA) for the Log Group and an optional description
  • Select Create Log Group at the bottom.

 

Task2 - Enable Logging Service to capture VCN Flow logs

In this task, we will turn-on the collection for VCN logs.
OCI console menu Solutions and Platform Logging.

  • Select Logs
  • Choose Compartment AppA and Log Group VCN-Logs-AppAA.
  • Click on Enable Service Log.

From the Enable Resource Log panel.

  • Ensure the Resource Compartment is correct (AppA)
  • Choose Virtual Cloud Networks under Service
  • Select subnet under Resource (Public Subnet-AppAA)
  • Under Log Category select Flow Logs (All records)
  • Provide a Log Name (PublicVCN-FlowLogsAppAA)
  • Click Enable Log at the bottom.

 

Task3 – Create a Log Group in Logging Analytics

As in task 1, log groups are also required in Logging Analytics
OCI console menu Logging Analytics Administration.

  • Select the compartment (AppA)
  • Select Log Groups
  • Select Create Log Group

On the Create Log Group page

  • Enter a Name (VCN-LogGroup-AppAA) and description. 
  • Select Create at the bottom.

 

Task4 – Create Service Connector

The Service Connector is used to orchestrate the movement of logs to Logging Analytics.
OCI console menu Logging Service Connectors.

  • Select the Compartment (AppA)
  • Click Create Connector

From the Service Connector page

  • Enter a Connector Name (AppAA-SC) and optional Description
  • Under Configure Service Connector, select Source as Logging
  • Target as Logging Analytics
  • Select Compartment Name (AppA), Log Group (VCN-Logs-AppAA) and Logs (PublicVCN-FlowLogsAppAA)
  • Under Configuration target connection, select Compartment (AppA) and Log Group (VCN-LogGroup-AppAA)
  • Select Create to allow a policy to be created for the Service Connector to write to Logging Analytics service
  • Select Create at the bottom to finish.

 

View and Analyse logs

To view and analyse your logs, go to:

OCI console menu Logging Analytics Log Explorer

  • Select Log Group Compartment (AppA)
  • Set desired Time Range
  • Select Log Source as OCI VCN Flow Unified Schema Logs

Summary

As shown, in just four simple steps we can start ingesting VCN flow logs into Logging Analytics. From here it's possible to gain real insights into your VCN’s by using features like Linking, Clustering, and Visualisation to identify things like; accepted or rejected connections which may be due to security rules, tracking traffic for abnormal behaviour, identifying trends based on ports and source/destination addresses, and much more.

Additional Resources

 

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha