A data flow in Oracle Analytics Cloud (OAC) is a powerful series of data preparation and transformation steps that, when run, produces curated datasets that can be used for analysis in workbooks. Understanding when and how your data flows fail is crucial for several reasons. It’s important to identify any issues that might arise in a timely manner to ensure datasets and workbooks contain accurate and up-to-date information. A data flow notification system that sends emails to certain users in the event of a failure ensures that users are informed and can take immediate action. This article details the steps involved in creating an email notification system for failed data flows using Oracle Cloud Infrastructure (OCI) resources, including Notifications Service and Connector Hub Service.
This article assumes that you already have diagnostic logging configured for your OAC environment and are familiar with OAC diagnostic logs. Read this article for more information on configuring diagnostic logging: Access Oracle Analytics Cloud logs through integration with Oracle Cloud Infrastructure Logging.
a) Log into the OCI Console and click the hamburger menu in the top-left corner. Navigate to the Developer Services tab and click Notifications under Application Integration. This directs you to the Notifications Service.
b) Click Create Topic to open a new dialog that allows you to create a Topic, which is a communication channel for sending messages to its subscriptions. For more information about Topics, visit this resource. In this dialog, enter a name for your topic and an optional description. Click Create to create the topic.
c) Click the new topic to open Topic Details. Scroll down and click Create Subscription to create a subscription, which acts as an endpoint for a topic. There are a few subscription protocols that are supported such as SMS, Slack, and HTTPS. We'll use the Email protocol, which sends an email message when you publish a message to the subscription’s parent topic. Find more information about this process here.
d) Select Email in the Protocol box and enter the email address that you want to receive emails for data flow failures. When a data flow fails, a message is sent to the topic that you just created, and the message is forwarded to the email addresses subscribed to this topic. Click Create to create the subscription. The subscription shows a status of Pending.
e) An email is sent to the address entered to confirm the subscription. Click the link in the email to confirm the subscription. The subscription state shows a status of Active after confirmation.
OCI Connector Hub is a service that allows you to transfer data between services in OCI.
a) From the OCI Console, click the hamburger menu in the top-left corner. Navigate to Analytics & AI and click Connector Hub under Messaging.
b) Create a connector, which is how information and data is transferred from one service (source) to the other (target). Click Create connector. Give the connector a name and an optional description. For source, select Logging. For target, select Notifications.
c) Scroll down to the Configure Source section. In the Compartment name box, select the compartment in which your log group is located. In the Log group box, select the log group that contains the diagnostic logs for your OAC environment. Finally, select the diagnostic logs associated with your OAC instance, which serve as the source for this connector.
d) The Log filter task section allows you to configure how you’d like to filter the logs (in this particular example, we'll be returning only the logs that indicate a data flow failure). To determine how best to fill out the Log filter task section for your use case, examine the JSON code associated with the OAC diagnostic logs. To follow along with this use case, which is to filter logs based on any failed data flow in the OAC environment, enter data.message in the Property field, select “=” as the Operator, and enter “DATAFLOW reload job ended for * with status: Failed” as the Value.
e) Skip the Configure Task section and scroll to the Configure Target section. Select the compartment where you created the topic in Step 1. Then select the topic that you created in Step 1. The logs specified in the source are sent directly to this topic (and eventually to the email addresses that have subscribed to this topic).
f) Specify the format in which you want the message to appear; you can choose to send either formatted messages or raw messages. Formatted messages appear in a user-friendly layout and raw messages contain a raw JSON blob. This example sends formatted messages, which makes it easier for users to read the information presented in the email.
f) Select Create to create the connector.
a) Create a data flow in your OAC environment that you know will fail and run the data flow. Wait a few minutes and check the inbox of the email that's subscribed to the topic that you created. You should receive an email from OCI Connector Hub containing detailed information regarding the data flow failure.
Although this use case focuses on notifying users of all data flow failures in their environment, you can also notify users when specific data flows fail. To change which logs you’d like to send to your Topic, you can modify the Log filter tasks for the connector source. For example, to send logs for failed runs belonging to the data flow with object Id = 'firstname.lastname@example.org'.'testing' to the Topic created earlier, modify the Value of the log filter task to include this string:
DATAFLOW reload job ended for "email@example.com"."testing" with status: Failed }
This notification system isn't exclusive to data flows; you can also configure notifications for datasets. Keep in mind that these logs are subject to change, so periodically check that the log filter tasks are working properly.
Call to Action
Create and fine-tune your own notification service for data flows to ensure timely issue resolution, data quality monitoring, and data accuracy. Visit the links mentioned throughout this article for additional guidance and share your progress on our community forum.