X

Oracle Management Cloud Blog covers the latest releases, customer stories, how-to guides and more.

  • March 31, 2020

How To Ingest OCI VCN Flow Logs into OMC

In the February 2020 Oracle Management Cloud (OMC) release, we announced it is possible to ingest and analyse logs from Oracle Cloud Infrastructure (OCI) Virtual Cloud Infrastructure (VCN) Flow Logs, OCI Audit Logs and other OCI logs stored in buckets. In this blog post, I’ll show how to set up OMC to poll for VCN Flow Logs, explore why the logs are useful and look at some meaningful insights we can get from OMC Logs Analytics (LA) services.

What’s interesting about VCN Flow Logs?

Flow logs capture information related to traffic in your VCN. The logs can inform on which traffic has been allowed (Accept) or denied (Reject), where the traffic originated from (srcaddr) and it's port (srcport), the destination (dstaddr) and it's port (dstport), the time this happened (start_time), when it ended (end_time), the communication method (protocol), how many packets (packets) and the communication size (bytes). 

In its raw form a typical log record would look like this:

version srcaddr dstaddr srcport dstport protocol packets bytes start_time end_time action status
2
x.x.x.x
x.x.x.x
50448
443
6
15
10672
1555431265
1583455265
ACCEPT
OK

The logs can be viewed in OCI Logging Service (Menu > Logging > Log Search) which has a search capability.  However, once the logs are ingested into OMC, there are many interesting ways in which we can use Log Analytics to analyse the data. For example:

  • Troubleshooting connections made from your private to public or on-premise to OCI networks, checking whether packets are being accepted or rejected. This may highlight areas in which your Security Lists need tightening up.
  • Help you understand the flow of traffic to and from your VCN's, like which ports/protocols are being used most often.
  • Insights into traffic growth.
  • Traffic to/from Countries/Cities.
  • IP to IP communication.
  • Spikes in packets/traffic.
  • Sudden propagation of new IP’s.

Below is a screenshot of an OMC Dashboard I have built based on just VCN Flow logs.

 

Let’s look at how we can ingest the logs to get these insights.

Assumptions

I will assume the following actions have already been completed:

1. Creating an OCI tenancy, a compartment, a VCN, an Instance, a user in IAM with RSA key pair (in PEM format) loaded for the API calls (OCI Documentation link). A simple setup could look like this:

2. Flow logs are enabled.

3. Administration level access to an OMC tenancy.

Gathering Information

For OMC to ingest VCN Flow logs, we need to establish a secure connection between OMC and OCI Object Stores and then point Log Analytics (LA) Service to the Flow log buckets. To do this we need to gather information from both OMC and OCI.  The information we gather here will be needed later.

The required information can be gathered in a few ways like using the OCI Console and/or OCI Command Line Interface (CLI) and/or Cloud Shell.  I’m using a combination in this post.

Tenancy Namespace:
OCI Console: Menu > Administration > Tenancy Information. Copy Object Storage Namespace value.
OCI CLI:

 

Tenancy OCID
OCI Console: Menu > Administration > Tenancy Information. Copy OCID.
OCI CLI:

 

User OCID – OCID of the user calling the API.
OCI Console: On the top right corner of the console click on the Profile, then click on the profile name. The User Information page shows the OCID

 

Fingerprint – Fingerprint for the User calling the API.
OCI Console: On the top right corner of the console click on the Profile, then click on the profile name. The fingerprint is under API Keys on the User Information Page.
OCI CLI:

 

Private Key – The Private and Public RSA key pair are part of the pre-requirements.  The location of the Private key will be unique in your environment.

RegionThe region in which Flow Logs are created. In my case, it's “eu-frankfurt-1”.
OCI Console: Menu > Administration > Region Management
OCI CLI:

 

Compartment-id – The compartment in which the Flow logs are generated. 
OCI Console: Menu > Identity > Compartment.  Click on your Compartment Name and record the compartment OCID.
OCI CLI:

 

Buckets – The Object Storage Bucket name that contains the Flow logs.
OCI Console: Menu > Object Storage.  Copy the Name
OCI CLI:

 

OMC Tenant Name
OMC Console: Menu > Administration > Agent > Download Agents. select Agent Type. The tenant_name will be shown at the bottom of the page. 

OMC URL
OMC Console: Menu > Administration > Agent > Download Agents. select Agent Type. The OMC_URL will be shown at the bottom of the page. 

Creating Credentials for OCI Authentication

To capture the Flow logs we need to establish a secure connection between OMC and the OCI Object Storage.
The credentials can be created from the OMC Console or REST API. I’ll show the console way.

From the Menu > Administration > Security > Credential Store.  Click on New Credential. A new dialogue window will appear:

We have already gathered most of the information required from the previous steps.  It’s important to set the Credential type to Oracle.OCI.Auth.

This is an example from my environment.

Note: I did not use a pass_phrase

Ingesting Flow Logs into Log Analytics

We now need to register the Flow log buckets to Log Analytics. Once registered the Log Analytics service will poll for new logs.

To register the buckets, we need to create a JSON configuration file to include details of the buckets we wish to poll and use REST API to register the configuration file.  We should already have most of the information we need to create the JSON configuration file. My JSON configuration file (bucket_config2.json) looks like this.

 {
   "logType":"FLOW",
   "bucketsInfo":[
      {
         "credential":"MKS-vnclogs",
         "namespace":"oraxxxxx",
         "region":"eu-frankfurt-1",
         "pollSince":"BEGINNING",
         "buckets":[
            {
               "name":"oci-logs._flowlogs.ocid1.compartment.oc1..aaxxxxxxxxxxxxxxx"
            }
         ]
      }
   ]
}

Where:
“logType” - depends on the logs that you are ingesting.  The options are FLOW, AUDIT or OCI_LOGS_GENERIC.
“credential” - is taken from the name of the credential store created earlier in OMC.
“pollSince” - Indicates when the logs should start to be polled.  There are three options:
BEGINNING – collects logs form the first time they were stored in the buckets.

2020-03-10T01:00:00.000Z – the absolute time when the buckets logs should start from.
CURRENT_TIME – logs are collected from the time the buckets are registered in LA.

REST API is then used to register the bucket_config2.json.

# curl  -X POST  -u <username> -H "X-USER-IDENTITY-DOMAIN-<tenant_name>" https://<tenant_url>/serviceapi/logan.uploads/registerOSSConfig -H 'Content-Type:application/json' -d "@bucket_config2.json"  

Where:
“username” -  is the name used to logon to OMC.  You will be prompted for the password.

To confirm the buckets have been registered with OMC, we can run:
# curl -X GET  -u <username>  https://<tenant_url>/serviceapi/logan.uploads/getOSSConfig?logType=FLOW

The output would be similar to:

[ {
  "bucketId" : "9f2xxxxxxxxxxxxxxxxxxx",
  "ociCredential" : "MKS-vnclogs",
  "ociNamespace" : "orasxxxxxxx",
  "ociRegion" : "eu-frankfurt-1",
  "ociBucket" : "oci-logs._flowlogs.ocid1.compartment.oc1..aaxxxxxxxxxxxxxxxx",
  "logType" : "FLOW",
  "createdOn" : "2020-03-06T18:01:41.701Z",
  "pollSince" : "1970-01-01T00:00:00.000Z",
  "logSourceName" : "OCI VCN Flow Logs"
}]

Summary

As you have seen it is very simple to ingest VCN Flow logs into the OMC's Log Analytics service to get real insights into VCN's. I have shown just a few use cases, however, there are many more especially when combined with other logs sources like Audit logs, and expanding into other OMC services like Infrastructure Monitoring (IM) and Application Performance Monitoring (APM).