In this post, I will show how to ingest Virtual Cloud Network (VCN) flow logs into Logging Analytics Service, by firstly enabling the logs in Logging Service and then using Service Connectors to orchestrate the movement of the logs to the Logging Analytics Service for deeper analytics.
The diagram below represents the flow we will be following, from enabling VCN logs in the Logging Service to our end game – analysing in Logging Analytics Service.
Initially, logs will be enabled in OCI Logging Service, then a Service Connector will be used to send logs to Logging Analytics where they will be ingested and analysed.
As a side note, I’ve been asked by many customers on the difference between Logging Service and Logging Analytics. Let me clarify; The Logging Service makes it easy to collect logs emitted from OCI, like Object Storage logs, Function logs, VCN logs and others. Queries can be written to search the logs by narrowing down the times, dates and points of interests very quickly.
Logging Analytics provides the analytics, making it simpler to explore the data, analyse patterns and out-liners, provide machine learning in the form of clustering and linking, create dashboards, provide topology drill-downs and much more. This post highlights some of these capabilities.
To understand how to send VCN flow logs to Logging Analytics, let’s focus on a use case example, which is loosely based on a customer’s requirement.
A user (UserA) has restricted access to one compartment (AppA) in a tenancy. The users’ task is to send public subnet VCN logs to Logging Analytics for analysis. The Tenant super admin is responsible for creating compartment AppA, UserA and assigning OCI policies.
Assigning policies will depend on organisational requirements. Below is the policy I used for reference only. The Quick Start Guide for Logging Analytics, Getting Started with Policies and How Policies Work are good references and should be used to understand the policy requirements needed for your organisation.
allow group LAadmin to MANAGE virtual-network-family in compartment AppA
allow group LAadmin to USE cloud-shell in tenancy
Let’s have a look at tasks UserA needs to perform to get public subnet VCN logs ingested into Logging Analytics.
Logs are always placed into log groups. Think of log groups as virtual containers or folders that can be used to organise logs and restrict access. For example, one log group could contain Function Logs and another log group could contain Audit Logs in the same compartment. A policy can then be used to restrict access to those logs. For example, only users in the auditors' group are permitted to read or delete Audit Logs.
OCI console menu Solutions and Platform Logging Select Log Groups.
In this task, we will turn-on the collection for VCN logs.
OCI console menu Solutions and Platform Logging.
From the Enable Resource Log panel.
As in task 1, log groups are also required in Logging Analytics
OCI console menu Logging Analytics Administration.
On the Create Log Group page
The Service Connector is used to orchestrate the movement of logs to Logging Analytics.
OCI console menu Logging Service Connectors.
From the Service Connector page
To view and analyse your logs, go to:
OCI console menu Logging Analytics Log Explorer
As shown, in just four simple steps we can start ingesting VCN flow logs into Logging Analytics. From here it's possible to gain real insights into your VCN’s by using features like Linking, Clustering, and Visualisation to identify things like; accepted or rejected connections which may be due to security rules, tracking traffic for abnormal behaviour, identifying trends based on ports and source/destination addresses, and much more.