Enabling OCI Notifications for Data Flows in Oracle Analytics Cloud

January 17, 2024 | 6 minute read
Gabrielle Prichard
Analytics Product Manager
Text Size 100%:

Data Flow Failure Notification System

A data flow in Oracle Analytics Cloud (OAC) is a powerful series of data preparation and transformation steps that, when run, produces curated datasets that can be used for analysis in workbooks. Understanding when and how your data flows fail is crucial for several reasons. It’s important to identify any issues that might arise in a timely manner to ensure datasets and workbooks contain accurate and up-to-date information. A data flow notification system that sends emails to certain users in the event of a failure ensures that users are informed and can take immediate action. This article details the steps involved in creating an email notification system for failed data flows using Oracle Cloud Infrastructure (OCI) resources, including Notifications Service and Connector Hub Service.

This article assumes that you already have diagnostic logging configured for your OAC environment and are familiar with OAC diagnostic logs. Read this article for more information on configuring diagnostic logging: Access Oracle Analytics Cloud logs through integration with Oracle Cloud Infrastructure Logging.

Step 1: Create a Notification Topic using OCI Notification Service and Subscribe to the Topic

a) Log into the OCI Console and click the hamburger menu in the top-left corner. Navigate to the Developer Services tab and click Notifications under Application Integration. This directs you to the Notifications Service. 

Notifications service

b) Click Create Topic to open a new dialog that allows you to create a Topic, which is a communication channel for sending messages to its subscriptions. For more information about Topics, visit this resource. In this dialog, enter a name for your topic and an optional description. Click Create to create the topic.

Create a topic

c) Click the new topic to open Topic Details. Scroll down and click Create Subscription to create a subscription, which acts as an endpoint for a topic. There are a few subscription protocols that are supported such as SMS, Slack, and HTTPS. We'll use the Email protocol, which sends an email message when you publish a message to the subscription’s parent topic. Find more information about this process here.

Create a subscription

d) Select Email in the Protocol box and enter the email address that you want to receive emails for data flow failures. When a data flow fails, a message is sent to the topic that you just created, and the message is forwarded to the email addresses subscribed to this topic. Click Create to create the subscription. The subscription shows a status of Pending.

Configure subscription

e) An email is sent to the address entered to confirm the subscription. Click the link in the email to confirm the subscription. The subscription state shows a status of Active after confirmation. 

Step 2: Use OCI Connector Hub to Create a Connector

OCI Connector Hub is a service that allows you to transfer data between services in OCI. 

a) From the OCI Console, click the hamburger menu in the top-left corner. Navigate to Analytics & AI and click Connector Hub under Messaging.

Connector hub

b) Create a connector, which is how information and data is transferred from one service (source) to the other (target). Click Create connector. Give the connector a name and an optional description. For source, select Logging. For target, select Notifications.

Create a connector

c) Scroll down to the Configure Source section. In the Compartment name box, select the compartment in which your log group is located. In the Log group box, select the log group that contains the diagnostic logs for your OAC environment. Finally, select the diagnostic logs associated with your OAC instance, which serve as the source for this connector.

d) The Log filter task section allows you to configure how you’d like to filter the logs (in this particular example, we'll be returning only the logs that indicate a data flow failure). To determine how best to fill out the Log filter task section for your use case, examine the JSON code associated with the OAC diagnostic logs. To follow along with this use case, which is to filter logs based on any failed data flow in the OAC environment, enter data.message in the Property field, select “=” as the Operator, and enter “DATAFLOW reload job ended for * with status: Failed” as the Value.

Configure source

e) Skip the Configure Task section and scroll to the Configure Target section. Select the compartment where you created the topic in Step 1. Then select the topic that you created in Step 1. The logs specified in the source are sent directly to this topic (and eventually to the email addresses that have subscribed to this topic).

f) Specify the format in which you want the message to appear; you can choose to send either formatted messages or raw messages. Formatted messages appear in a user-friendly layout and raw messages contain a raw JSON blob. This example sends formatted messages, which makes it easier for users to read the information presented in the email.

Configure target

f) Select Create to create the connector.

Step 3: Validate that the Notification System Works

a) Create a data flow in your OAC environment that you know will fail and run the data flow. Wait a few minutes and check the inbox of the email that's subscribed to the topic that you created. You should receive an email from OCI Connector Hub containing detailed information regarding the data flow failure.

Additional Information and Important Notes

Although this use case focuses on notifying users of all data flow failures in their environment, you can also notify users when specific data flows fail. To change which logs you’d like to send to your Topic, you can modify the Log filter tasks for the connector source. For example, to send logs for failed runs belonging to the data flow with object Id = 'random.email.add@oracle.com'.'testing' to the Topic created earlier, modify the Value of the log filter task to include this string:

DATAFLOW reload job ended for "random.email.add@oracle.com"."testing" with status: Failed }

Additional filtering

This notification system isn't exclusive to data flows; you can also configure notifications for datasets. Keep in mind that these logs are subject to change, so periodically check that the log filter tasks are working properly.

Call to Action

Create and fine-tune your own notification service for data flows to ensure timely issue resolution, data quality monitoring, and data accuracy. Visit the links mentioned throughout this article for additional guidance and share your progress on our community forum.

Gabrielle Prichard

Analytics Product Manager

Previous Post

Oracle Fusion Analytics - Refresh Materialized Views via Custom Function After Pipeline Completion

Lisa Garczynski | 14 min read

Next Post

Deploy Network Components for Oracle Fusion Analytics Service Endpoints

Dayne Carley | 10 min read