Integrating Oracle Cloud Guard with Logging Analytics using Streaming and Functions (Part 2)

December 21, 2022 | 6 minute read
Lasya Vadavalli
Staff Cloud Engineer
Srikanth Reddy
Cloud Engineer
Vishak Chittuvalapil
Senior Cloud Engineer
Text Size 100%:

As a continuation to Part 1, where we ingested logs from Oracle Cloud Guard to the Logging Analytics service through the Streaming service. In part 2, we use Oracle Cloud Infrastructure (OCI) Functions service to ingest Oracle Cloud Guard logs to Logging Analytics.

The following graphic depicts the architecture for the solution:

A graphic depicting the architecture for the solution,moving from Cloud Guard to Events to Functions to Logging Analytics.

When Cloud Guard emits events that match with the defined rule conditions, the OCI Events service triggers the specified function. When invoked, the Functions code reads logs data from Events and sends them to the Logging Analytics upload API endpoint. We can then view the logs from Log Explorer dashboard and process them for any further visualizations and insights.

Overview steps

  1. Enable a cloud event rule in responder recipe.

  2. Create a log group under Logging Analytics to store logs.

  3. Create a log parser and source under Logging Analytics.

  4. Create an OCI function in Python or any supported language to send Cloud Guard logs to Logging Analytics.

  5. Create a dynamic group and provide appropriate policies for the function to access Logging Analytics.

  6. Create an Event rule to trigger the deployed function when the rule conditions for specified event types are met.

For the first three steps, follow the implementation in Part 1. Enable the cloud event rule in responder recipe. Create a log group under Logging Analytics. While provisioning, utilize OCI’s compartment structure to fine tune user access to Logging Analytics components. Create a log parser and source under Logging Analytics.

Step 4

To create the function, use the following steps:

  • In the Oracle Cloud Console menu, navigate to Developer services and select Functions.

  • Select an existing application or click Create Application. Create a function within your application.

  • I’m using OCI Cloud Shell to create my function. Cloud Shell is a recommended option as it does not require any setup prerequisites. If you’re new to OCI Functions, follow sections A, B, and C in the Functions Quick Start on Cloud Shell.

  • We recommend creating a sample Python function first. The following command generates a folder “cgtola” with three files :func.py, func.yaml, and requirements.txt.

    Copied to Clipboard
    Error: Could not Copy
    Copied to Clipboard
    Error: Could not Copy
    fn init --runtime python cgtola
  • Then create a function with the following code:

    func.py:

    Copied to Clipboard
    Error: Could not Copy
    Copied to Clipboard
    Error: Could not Copy
    import io
          import json
          import logging 
          import oci 
          from fdk import response 
    
           #The below method will receive the list of log entries from OCI as input in the form of bytestream and is defined in func.yaml 
           def handler(ctx, data: io.BytesIO = None): 
               funDataStr = data.read().decode('utf-8') 
               funData =  json.loads(funDataStr) 
               logging.getLogger().info(funData) 
          
              #send the log data to a temporary json file. /tmp is the supported writable directory for OCI functions 
               with open('/tmp/test.json', 'w', encoding='utf-8') as f: 
                   json.dump(funData, f, ensure_ascii=False, indent=4) 
               file_details=io.open("/tmp/test.json") 
               signer = oci.auth.signers.get_resource_principals_signer() 
               log_analytics_client = oci.log_analytics.LogAnalyticsClient({},signer = signer)
    
           # Send the request to service, some parameters are not required, see API doc for more info 
    
               log_group_id="ocid1.loganalyticsloggroup.oc1.iad.amaaaaaas4n35viawkqpxtefipgoqemkc2n55b7hylaucb2vck6z4xxxxxxx" 
               upload_log_file_response=log_analytics_client.upload_log_file(namespace_name="orasenatdpltsecitom04", upload_name="CG_logs", log_source_name="Cloud Guard_functions_source", filename="test.json", opc_meta_loggrpid=log_group_id, upload_log_file_body=file_details)

    func.yaml:

    Copied to Clipboard
    Error: Could not Copy
    Copied to Clipboard
    Error: Could not Copy
    schema_version: 20180708 
           name: cgtola 
           version: 0.0.31 
           runtime: python 
           build_image: fnproject/python:3.9-dev 
           run_image: fnproject/python:3.9 
           entrypoint: /python/bin/fdk /function/func.py handler 
           memory: 256

    requirements.txt:

    Copied to Clipboard
    Error: Could not Copy
    Copied to Clipboard
    Error: Could not Copy
    fdk>=0.1.48 
           oci
  • Deploy the function with the following command:

    Copied to Clipboard
    Error: Could not Copy
    Copied to Clipboard
    Error: Could not Copy
    fn -v deploy --app CG-LA

When invoked by the Events service, the Functions code fetches log data from Events service and uploads it to the Logging Analytics API.

Step 5

A screenshot of the Dynamic Group Details page in the Oracle Cloud Console.

We now create policies to grant the created dynamic group access to Logging Analytics resources. For details, refer to Enable Access to Logging Analytics and its Resources.

  • In the menu, select Identity and Security, Identity, and Policies and click Create Policy. Give the following policy statements:

    • allow dynamic-group Lasya-DG to manage loganalytics-features-family in tenancy

    • allow dynamic-group Lasya-DG to manage loganalytics-resources-family in compartment Lasya

Give the first policy only at tenancy (root) level. The second policy can be either given at child compartment level or tenancy (root) level.

Step 6

Create an event to trigger our function when the specified Cloud Guard event types are matched.

A screenshot of the Edit Rule page, showing how to connect to the Events service.

We can verify the uploaded Cloud Guard logs on the Log Explorer dashboard. In the menu, select Observability & Management, Logging Analytics, and Log Explorer.

A screenshot of the Log Explorer in Logging Analytics.

Conclusion

In this blog, we covered how to ingest Cloud Guard events to Logging Analytics using OCI Functions. Logging Analytics makes it easy for you to unlock the value of log data for any source across Oracle Cloud Infrastructure, on-premises, and third-party clouds like Amazon Web Services (AWS) and Azure quickly without involving any data scientists.

For more use cases of Logging Analytics, see the Observability and Management blog.

Lasya Vadavalli

Staff Cloud Engineer

Srikanth Reddy

Cloud Engineer

Srikanth Reddy is a Cloud Engineer with 4 years of IT experience. As a Cloud Engineer in Oracle North America Cloud Engineering - PMO, Srikanth is responsible for administering program management tools for organizational internal usage. Prior to joining NA Cloud Engineering - PMO, he was involved in delivering cloud infrastructure solutions and is a certified professional OCI architect.

Show more

Vishak Chittuvalapil

Senior Cloud Engineer


Previous Post

Monitor OCI Compute Instances and manage VM disk utilization using OCI Stack Monitoring service

Akarsha Itigi | 4 min read

Next Post


Access Jupyter notebooks within OCI Data Flow service

JR Gauthier | 8 min read
Oracle Chatbot
Disconnected