In February 2020 Oracle Management Cloud (OMC) release, we announced it is possible to ingest and analyse logs from Oracle Cloud Infrastructure (OCI) Virtual Cloud Infrastructure (VCN) Flow Logs, OCI Audit Logs and other OCI logs stored in buckets. In this blog post, I’ll show how to set up OMC to poll for VCN Flow Logs, explore why the logs are useful and look at some meaningful insights we can get from OMC Logs Analytics (LA) services.

What’s interesting about VCN Flow Logs?

Flow logs capture information related to traffic in your VCN. The logs can inform on which traffic has been Accepted (data.action) or Rejected, where the traffic originated from (data.sourceAddress) and it's port (data.sourcePort), the destination (data.destinationAddress) and it's port (data.destinationPort), the time this happened (data.startTime), when it ended (endTime), the communication method (data.protocolName), how many packets (data.packets) and the number of bytes in the period (data.bytesOut). 

The logs when enabled can be viewed in OCI Logging Service (Menu > Logging > Log Search) which has a search capability.  However, once the logs are ingested into OMC, there are many interesting ways in which we can use Log Analytics to analyse the data. For example:

• Troubleshooting connections made from your private to public or on-premise to OCI networks, checking whether packets are being accepted or rejected. This may highlight areas in which your Security Lists need tightening up.
• Help you understand the flow of traffic to and from your VCN's, like which ports/protocols are being used most often.
• Insights into traffic growth.
• Traffic to/from Countries/Cities.
• IP to IP communication.
• Spikes in packets/traffic.
• Sudden propagation of new IP’s.

Below is a screenshot of an OMC Dashboard I have built based on just VCN Flow logs. This post has other great examples.

Let’s look at how we can ingest the logs to get these insights.

Assumptions

I will assume the following actions have already been completed:

1. Creating an OCI tenancy, a compartment, a VCN, an Instance, a user with RSA key pair (in PEM format) loaded for the API calls . A simple setup could look like this:

3. User and Service policies as per your organisation requirements should be in place. For information on some of the policies required, have a read of:

1. 1. 1. Managing Logs and Log Groups
      2. Logging Analytics Quick Start Guide
      3. Common Polices

2. Administration level access to an OMC tenancy.

Log Ingestion Flow

The diagram below represents the flow we will be following, from enabling VCN logs in Logging Service to OMC Logs Analytics.

Initially, logs will be enabled in OCI Logging Service, then a Service Connector will be used to move the logs to Object Storage Buckets, where OMC Log Analytics (LA) will ingest and analyse.

Gathering Information

For OMC to ingest VCN Flow logs, we need to establish a secure connection between OMC and OCI Object Stores and then point Log Analytics  Service to the buckets to ingest the logs. To do this we need to gather information from both OMC and OCI.  The information we gather in this section will be needed later.

The required information can be gathered in a few ways like using the OCI Console and/or OCI Command Line Interface (CLI) and/or Cloud Shell.  I’m using a combination in this post.

Tenancy Namespace:
OCI Console: Menu > Administration > Tenancy Information.
Cloud Shell:

Tenancy OCID
OCI Console: Menu > Administration > Tenancy Information. Copy OCID.
Cloud Shell:

User OCID – OCID of the user calling the API.
OCI Console: On the top right corner of the console click on the Profile, then click on the profile name. The User Information page shows the OCID

Cloud Shell:

Fingerprint – Fingerprint for the User calling the API.
OCI Console: On the top right corner of the console click on the Profile, then click on the profile name. The fingerprint is under API Keys on the User Information Page.
OCI CLI: 

Private Key – The Private and Public RSA key pair are part of the pre-requirements.  The location of the Private key will be unique in your environment.

Region – The region in which Flow Logs are created. In my case, it's “eu-frankfurt-1”.
OCI Console: Menu > Administration > Region Management
Cloud Shell:

Compartment-id – The compartment in which the Flow logs are generated. 
OCI Console: Menu > Identity > Compartment.  Click on your Compartment Name and record the compartment OCID.
Cloud Shell:

Buckets – The Object Storage Bucket name that contains the Flow logs.
OCI Console: Menu > Object Storage.  Copy the Name
Cloud Shell:

OMC Tenant Name –
OMC Console: Menu > Administration > Agent > Download Agents. select Agent Type. The tenant_name will be shown at the bottom of the page. 

OMC URL
OMC Console: Menu > Administration > Agent > Download Agents. select Agent Type. The OMC_URL will be shown at the bottom of the page. 

Enable Logging Service to capture VCN Flow logs

1. Create a Log Group in Logging Service

Logs are always put into log groups.  Think of log groups as virtual containers or folders that can be used to organise logs and restrict access. For example, one log group could contain Function logs and another log group could contain Audit logs in the same compartment. A policy can then be used to restrict access to those logs. For example, only users in the Auditors group are permitted to read or delete audit logs.

OCI console menu  Solutions and Platform  Logging. 

• Select Log Groups.
• Choose Compartment (part of the prerequisites) for example AppA
• Provide a Name .eg. VCN-Logs-AppAA for the Log Group and an optional description
• Select Create Log Group at the bottom.

2. Enable log collection

• Select Logs
• Choose Compartment AppA and Log Group VCN-Logs-AppAA.
• Click on Enable Service Log.

From the Enable Resource Log panel.

• Ensure the Resource Compartment is correct (AppA)
• Select Virtual Cloud Network (subnets) under Service.
• Select subnet under Resource (Public Subnet-AppAA)
• Under Log Category select Flow Logs (All records).
• Provide a Log Name (PublicVCN-FlowLogsAppAA) 
• Click Enable Log at the bottom.

Sending Logs to Object Storage Buckets

In this section, we will create a Service Connector to send the logs to buckets.  Tasks required: 1. Create an Object Storage Bucket, 2. Create a Service Connector

1. Create an Object Storage Bucket

The bucket stores logs in compressed format.  Logging Analytics will ingest the logs from the buckets.

OCI console menu  Object Storage  Object Storage.

• Ensure Compartment is correct (AppA).
• Click on Create Bucket

• Enter a Bucket Name (VCN-Flow-AppAA).

2. Create a Service Connector

The Service Connector is used to orchestrate the movement of logs to Object Storage Buckets.

OCI console menu  Logging  Service Connectors.

• Select the Compartment (AppA)
• Click Create Connector

From the Service Connector page

• Enter a Connector Name (AppAA-SC) and optional Description
• Under Configure Service Connector, select Source as Logging
• Target as Object Storage.
• Select Compartment Name (AppA), Log Group (VCN-Logs-AppAA) and Logs (PublicVCN-FlowLogsAppAA).
• Under Configuration target connection, select Compartment (AppA) and Bucket (VCN-Flow-AppA).
• Select Create at the bottom.

Creating Credentials for OCI Authentication

To capture the Flow logs we need to establish a secure connection between OMC and the OCI Object Storage.
The credentials can be created from the OMC Console or REST API. I’ll show the console way.

From the Menu > Administration > Security > Credential Store.  Click on New Credential. A new dialogue window will appear:

We have already gathered most of the information required from the previous steps.  It’s important to set the Credential type to Oracle.OCI.Auth.

This is an example from my environment.

Note: I did not use a pass_phrase

Ingesting Flow Logs into Log Analytics

We now need to register the Flow log buckets to Log Analytics. Once registered the Log Analytics service will poll for new logs.

To register the buckets, we need to create a (1) JSON configuration file to include details of the buckets we wish to poll and use (2) REST API to register the configuration file.  We should already have most of the information we need to create the JSON configuration file. My JSON configuration file (bucket_config2.json) looks like this.

(1) JSON configuration file:                   

{
	"logType": "FLOW",
	"bucketsInfo": [{
		"credential": "MKS-vnclogs",
		"namespace": "oraxxxxxxxcore",
		"region": "eu-frankfurt-1",
		"pollSince": "BEGINNING",
		"buckets": [{
			"name": "VCN-Flow-AppAA"
		}]
	}]
}

curl -X POST -k -u '(username)' '-H 'X-USER-IDENTITY-DOMAIN-NAME:(idcs-ID)' "https://(OMC_URL)/serviceapi/logan.uploads/registerOSSConfig" -H 'Content-Type:application/json' -d "@bucket_config2.json"

curl -X POST -k -u 'mahesh.xxxxxx@oracle.com' -H 'X-USER-IDENTITY-DOMAIN-NAME:idcs-xxxxxxxxxxxxxxxx66185d3831' "https://omc-4b06xxxxxxxxxx.api.omc.ocp.oraclecloud.com/serviceapi/logan.uploads/registerOSSConfig" -H 'Content-Type:application/json' -d "@bucket_config2.json"
Enter host password for user 'mahesh.xxxxxx@oracle.com':

curl -X GET -k -u '(username)' "https://(omc_url)/serviceapi/logan.uploads/getOSSConfig?logType=FLOW"

[ {
  "bucketId" : "xxxxxxxxxxxxxxxxxx78238c",
  "ociCredential" : "MKS-vnclogs",
  "ociNamespace" : "xxxxxxxxxcore",
  "ociRegion" : "eu-frankfurt-1",
  "ociBucket" : "VCN-Flow-AppAA",
  "logType" : "FLOW",
  "createdOn" : "2020-11-30T17:02:21.194Z",
  "pollSince" : "1970-01-01T00:00:00.000Z",
  "logSourceName" : "OCI VCN Flow Logs"
}]

curl -X DELETE -k -u '(username)' "https://(OMC_URL)/serviceapi/logan.uploads/unregisterOSSConfig" -H 'Content-Type:application/json' -d "@Unregister.json"

Where Unregister.json is:

{
   "bucketIds" : ["bucket_IDs"]
}

Summary

As you have seen it is very simple to ingest VCN Flow logs into the OMC's Log Analytics service to get real insights into VCN's. I have shown just a few use cases, however, there are many more especially when combined with other logs sources like Audit logs, and expanding into other OMC services like Infrastructure Monitoring (IM) and Application Performance Monitoring (APM).