Ingest OCI service logs to Rapid7 InsightIDR

Serious IT breaches make headlines every other day. Cyberattackers are constantly looking to exploit any gap in IT systems, applications, and hardware. Security teams can prevent and combat attacks by identifying and responding to security events in real time to minimize damage. Security Information and Event Management Software (SIEM) allows security teams to keep on top of security alerts.

Rapid7 InsightIDR

Rapid7 InsightIDR is a software-as-a-service (SaaS) product that collects all audit and security logs from your environment with the help of collectors and agents. It sends them securely to Amazon Web Services (AWS) where the client’s Rapid7 account exists. InsightIDR comes with multiple cloud event sources built in, which allows you to ingest your cloud services audit and security logs from cloud vendors like AWS. InsightIDR currently doesn’t support Oracle Cloud Infrastructure (OCI) as an event source. In this blog, I show you how you can ingest OCI service logs in InsightIDR.

Getting started

This blog assumes that you already have at least one Rapid7 collector configured in your OCI environment.

OCI Service Connector Hub can directly send logs to your OCI Object Storage buckets, but because the logs are processed in batches, the complete batch is stored in gzip format in the bucket. InsightIDR can’t unzip these logs before processing. So, we’re not using the direct Service Connector Hub to Object Storage integration. Instead, we use an OCI function as the target for the service connector, where we parse the logs and stores them in the bucket.

Step 1: Create an Object Storage bucket

Object Storage stores and manages data as objects that are organized into buckets. Each object has a set of metadata that describes the object. We’re sending all our logs to buckets for the Rapid7 collector for access. To set up a bucket, follow the steps in Working with Oracle Cloud Infrastructure Object Storage.

Step 2: Create an OCI serverless function

This function parses the logs sent by the Service Connector Hub and store them in a file in the bucket that we created in the previous step.

To set up an OCI function, follow the steps in Jumpstart your Functions-as-a-Service journey in OCI with Cloud Shell. You can also use the newly released OCI Code Editor to create this function. You also must set up the function as an OCI resource principal, which allows the function to access the OCI Object Storage bucket.

Step 3: Function code

Use the following function code as the template for your function. If you want to process or manipulate some logs, you can do so in this function.

import io
import json
import logging
import oci
import datetime

from fdk import response

out_bucket= '
  
   '
out_namespace = '
   
    '
out_object = 'log_' + str(datetime.datetime.now()) + '.log'

def handler(ctx, data: io.BytesIO=None):
    try:
        signer = oci.auth.signers.get_resource_principals_signer()
        client = oci.object_storage.ObjectStorageClient(config={}, signer=signer) 
        body = json.loads(data.getvalue())
        client.put_object(out_namespace,out_bucket,out_object,data.getvalue())
    except (Exception, ValueError) as ex:
        logging.getLogger().info('error parsing json payload: ' + str(ex))

    logging.getLogger().info("file written  : "+ out_object)
    return response.Response(
        ctx, response_data=json.dumps(
            {"message": "success"}),
        headers={"Content-Type": "application/json"}
    )

   
  

Step 4: Create a service connector in Service Connector Hub

A service connector sends the desired logs to the OCI function that we created in the previous step. On the Create Connector page, choose “logging” as the source and choose the appropriate logs that you want to send to InsightIDR.

Choose the function that we created earlier in this step as the target.

Step 5: Initiate Rapid7 Collector 

The collector agent sits in your environment and collects logs from environments, then it sends it your InsightIDR account for processing.
you need to spin up a small compute machine and install the collector package that you get from InsightIDR on this compute machine, make sure you disable or set permissive to the SE Linux enforment policy.

you can run the command to set the SE linux to permissive mode – setenforce Permissive  on operating systems like Oracle Linux or RHEL

Step 6: Mount the Object Storage bucket to the collector

Now that we have the logs in our bucket, the collector can access these logs and send them to InsightIDR. We mount the bucket where our logs are Mounting an Object Storage Bucket as File System on Oracle Linux.

Ensure that the mount folder on the drive has the correct permissions to be accessed by the collector. The collector was installed as root.

Step 7: Set up a custom collector in Rapid7 InsightIDR

InsightIDR comes with prebuilt connector for cloud services. Unfortunately, OCI isn’t one of them, but it does allow you to ingest and process raw data with the following steps:

1. Select Data Collection, Setup Event Sources, and click Custom Logs.

   

2. Choose your local directory and use the path of our mounted bucket.

3. Click Save.

   

The log collection now begins.

Conclusion

In this blog, we saw how to configure and ingest OCI service logs, audit logs, and metrics to Rapid7 InsightIDR.

Oracle Cloud Infrastructure provides enterprise features for developers to build modern cloud applications. If you want to try out the steps in this blog for free, I recommend the Oracle Cloud Free Tier with US$300 credits for a 30-day free trial. Free Tier also includes several Always Free services that are available for an unlimited time, even after your free credits expire.