Oracle Cloud Infrastructure recommends, when possible, to use our other Streaming approach along side the OCI Logging Addon for Splunk in lieu of the approach in this blog. For more details see this link: https://docs.oracle.com/en/solutions/logs-stream-splunk/index.html
Each instance in an Oracle Cloud Infrastructure virtual cloud network (VCN) has one or more virtual network interface cards (VNICs) for communication within and outside of the VCN. Oracle Cloud Infrastructure Networking uses security lists to determine what traffic is allowed in and out of a given VNIC. A VNIC is subject to all the rules in all the security lists and network security groups associated with the VNIC's subnet.
To help you meet audit and compliance requirements and monitor the traffic in and out of your VNICs, you can now set up VCN flow logs to record details about traffic that has been accepted or rejected based on the security list or network security group rules. This post describes multiple ways that you can ingest this data and visualize your VCN traffic by using Splunk.
A flow log record is a space-separated string that has the following format:
<version><srcaddr> <dstaddr> <srcport> <dstport> <protocol> <packets> <bytes> <start_time> <end_time> <action> <status>
For example:
2 172.16.2.139 172.16.1.107 73 89 11 102 349 1557424462 1557424510 ALLOW OK
2 172.16.2.145 172.16.2.179 82 64 13 112 441 1557424462 1557424486 REJECT OK
Splunk supports numerous ways to ingest data, from monitoring local files by using an agent or streaming wire data, to pulling data from remote third-party APIs, to receiving data over syslog, TCP, UDP, or HTTP. One example of pushing data is through Oracle Functions, which is used to stream events over HTTPS to the Splunk HTTP Event Collector (HEC). An example of the pull method would be using the Oracle Cloud Infrastructure Object Storage plugin for Splunk.
These pull and push models apply to different use cases and have different considerations. This post pertains to the event-driven push model, which provides a more scalable and closer to real-time feed of flow log data. Because there are no dedicated pollers to manage and orchestrate, the push model generally offers the following benefits:
This solution uses the newly released Oracle Cloud Infrastructure Events service and Oracle Functions alongside Splunk's HTTP Event Collector technology to achieve a highly scalable pattern to meet large-scale enterprise ingestion requirements.
By following the steps in this post, you can create super-functional visualizations like the one shown in the following image.
In the following diagram, the Oracle Cloud Infrastructure Logging service writes the flow log data to an Object Storage bucket. Object Storage then emits a Create Object event to the Events service. The Events service in turn triggers a function. This function then splits the data into chunks allowed by your Splunk HTTP Event Collector and sends those chunks over HTTP or HTTPS to your Event Collector endpoint.
The rest of this post walks you through the following steps to achieve this solution:
Cloud Shell appears at the bottom of the Console. Keep it open because you’ll use it for several steps.
- Create a repository in Oracle Cloud Infrastructure Registry to store the Oracle Functions images.
- Ensure that your user has the correct permissions. Use the following example policy statements, which assume that your user is a member of the group called group flow-log-enablers. Replace this group with any existing group that you want to use. You can also adjust the policy to be less broad to meet your governance requirements.
Allow group flow-log-enablers to manage repos in tenancy where ANY {request.permission = 'REPOSITORY_CREATE', request.permission = 'REPOSITORY_UPDATE'}
Allow service FaaS to read repos in tenancy
Allow service FaaS to use virtual-network-family in compartment
You can think of this operation as attaching the flow log configuration to the subnet.
For a detailed guide on enabling VCN Flow Logs - see this documentation: https://docs.oracle.com/en-us/iaas/Content/Logging/Concepts/service_logs.htm
Now that flow logs are being recorded, start setting up the data pipeline at Splunk.
Follow the steps in the Splunk documentation to create a source type.
On the Advanced tab, set the following parameters:
This step of the data pipeline tells Splunk how to transform the raw flow log data into meaningful columns. Follow the instructions in the Splunk documentation to create a field transform.
In this step, you create a delimiter-based transform on the _raw source key. The delimiter in this case is a space, and you specify that as follows: " "
The full field list for version 2 of VCN flow logs is as follows:
"version","SourceAddress","DestinationAddress","SourcePort","DestinationPort","protocol","packets","bytes","startTime","endTime","action","status"
Configure your Splunk HTTP Event Collector (HEC), and create an HEC token that you will need in a later step. Follow the instructions in the Splunk documentation.
Note: For Splunk Cloud deployments, HEC must be enabled by Splunk Support.
When you configure the input settings, specify any index that you want HEC to forward your flow log data to, and specify flowlogs as the source type. Also note your new HEC token value; you’ll need it in the following step.
The following screenshot shows and example of the data input settings.
Create an Oracle function called splunk-flow-log-python. Request access to a prebuilt function sample. The sample function implements the necessary logic to process the VCN flow logs data, including decoding it and decompressing it, and parsing the events before sending to it Splunk HEC.
1. After you receive the function sample, upload the zip file to an Object Storage bucket.
2. On the Bucket Details page, click Edit Visibility and set the file to public visibility.
3. Get the URL of the file from the Object Details page. For example, https://objectstorage.us-ashburn-1.oraclecloud.com/n/<tenancyname>
4. In Cloud Shell, download the file:
wget https://objectstorage.us-phoenix-1.oraclecloud.com/n/<tenancyname>/b/<bucketname>/o/function.zip</bucketname></tenancyname>
5. In Cloud Shell, extract the function:
unzip function.zip && rm function.zip
6. In Cloud Shell, open the func.yaml file in vi and set the following required parameters:
7. Create an Oracle Functions application called FlowLogs by running the following command in Cloud Shell.
Note: Be sure to attach this application to a VCN and subnet that can communicate with HEC on the required port.
oci fn application create --display-name FlowLogs --subnet-ids '["ocid1.subnet.oc1.phx.aaaaa...."]' --compartment-id ocid1.compartment.oc1..aaaa.....
8. Run the following commands in Cloud Shell to configure and deploy the function code:
fn use context <region> ex: us-phoenix-1</region>
fn update context oracle.compartment-id <compartment-id>
</compartment-id>
fn update context registry phx.ocir.io/<tenancy_name>/[YOUR-OCIR-REPO] <-- Replace phx with the three-digit region code
</tenancy_name>
docker login phx.ocir.io <-- Replace phx with the three-digit region code
You are prompted for the following information:
Note: If you are using Oracle Identity Cloud Service, your username is <tenancyname>/oracleidentitycloudservice/<username>.
Deploy the function to the FlowLogs application that you created in the previous step:
fn deploy --app FlowLogs
After you activate flow logs for a particular subnet, the Logging service generates the first log when an Object Storage bucket is created specifically for the compartment the subnet exists in.
1. Get the name of the bucket. The name follows this format: oci-logs._flowlogs.
2. In the Console main menu, select Application Integration and then Events Service.
3. Create a rule with the following values. For more information about creating rules, see the Events documentation.
To use and retrieve the flow log objects from Object Storage, you must grant access to your function through a dynamic group.
1. Create a dynamic group by following the instructions in the documentation.
2. Add a rule to the group that matches the resource.id field to the function OCID for the function that you created.
resource.id=<fnfunc_ocid></fnfunc_ocid>
3. Create a policy that allows the function to retrieve flow log objects from the flow log bucket:
Allow dynamic-group flowlogs to read buckets in compartment
Allow dynamic-group flowlogs to read objects in compartment
After few minutes, you should start seeing events in Splunk Enterprise.
1. Open the navigation menu. Under Core Infrastructure, click Object Storage.
2. Choose the compartment that contains your buckets.
A list of buckets is displayed.
3. Click the Actions icon (three dots) to the right of the bucket name, and then click View Bucket Details.
4. Next to Emit Object Events, click Edit.
5. In the dialog box, select (to enable) Emit Object Events.
6. Click Save Changes.
Search for events in Splunk by using the following query: source="http:VCNFlowlogs"
To create visualizations like the one shown at the beginning of this post, and to alert on certain activities, use the instructions in the following Splunk documentation topics:
If you don’t see events in Splunk, you can troubleshoot this one pipeline stage at a time by following the data flow direction.
- Ensure that VCN flow logs are captured in the VCN and sent to Object Storage. If you still don’t see any logs, following are some possible causes:
- Ensure that the function is being triggered. If you see request errors, here are some common causes:
This post shows you how to configure a low-overhead and highly scalable data pipeline to stream your valuable VCN flow logs to your existing Splunk Enterprise by using Oracle Functions and Events alongside a Splunk HEC. That data pipeline enables near real-time processing and analysis of data by Splunk Enterprise. For an example of how to use a pull methodology, you can use the Splunk for Object Storage Plugin from SplunkBase.