Why Do You Need Anomaly Detection?

To understand the importance of Oracle Cloud Infrastructure’s (OCI) Anomaly Detection service, it’s important to first understand the full scope of OCI. OCI offers a robust and a comprehensive portfolio of infrastructure and cloud platform data and AI services to access, store, and process any type of data, from any source, enabling customers to implement end-to-end, enterprise scale data, and AI architectures in cloud. Customers can deploy a complete, integrated solution, including data management, data integration, and data science, so analytics and data science teams can maximize the value of enterprise data.

A graphic depicting the architecture for Anomaly Detection service.

In this blog, I highlight OCI Anomaly Detection service, which we launched in general availability recently. For more information about the other services in this architecture, see the References at the end of this post.

Anomaly Detection

Anomaly Detection service is a serverless, cloud native service that provides MSET2-based prebuilt anomaly detection algorithms over REST APIs. Anomaly Detection helps you to identify anomalies in a multivariate dataset by taking advantage of the interrelationship among signals. Use cases of anomaly detection exist in many industry verticals, including utilities, oil and gas, transportation, manufacturing, telecommunications, banking, insurance, web businesses, and IT. We published a reference architecture for using Anomaly Detection service for asset management and predictive maintenance.

In this reference architecture, we discuss the end-to-end flow using the Anomaly Detection service along with other OCI Data and AI services. The following diagram shows the main phases: Data collection, analysis, and actions. For data collection, Data Integration and Streaming service are used. OCI Object Storage service store the raw data, training sets, payload data, and predictions. The analysis phase uses Data Science, Oracle Container Engine for Kubernetes (OKE), and Functions service. The actions phase employs Oracle Analytics Cloud (OAC), OKE, Functions, and Notifications services.

A graphic depicting the data flow for Anomaly Detection.

The OCI Console, CLI, and software developer kits (SDKs) make it easy for developers to use Oracle’s Anomaly Detection algorithms in their end-to-end solutions. As shown in the following graphic, an end-to-end solution architecture using Anomaly Detection service has the following categories of functionality:

  • Data ingestion and storage: Asset to cloud connection and data transportation and management. Depending on data sources, you can use Data Integration or Streaming services to ingest the data and store it in Object Storage. Oracle Autonomous Transaction Processing and InfluxDB are also supported as data sources.
  • Data processing and preparation: To build and test the mode, you need to prepare proper training and testing data. The training and testing data must only contain timestamps and numeric attributes from sensor and signal readings.
  • Anomaly Detection models and predictions: Anomaly Detection builds a machine learning model for each signal as a function of interrelationship among signals; the Anomaly Detection algorithms maximize the accuracy of identified anomalies.
  • Applications, integrations, and operations: Use case-dependent interfaces with cloud services, visualization layer, reporting, and others.

A graphic depicting the stages that data travels through from the source to the user.

Anomaly Detection Workflow

Anomaly detection workflow has two phases: Training and detection. During the training phase, data is cleansed and prepared, and the model is trained. A Data Science-based workbench can make it convenient to split your dataset, prepare your training set, invoke Anomaly Detection model training, and test the results. For more information on data requirements, see the documentation.

In the detection phase, anomalies are detected and based on these predictions, Anomaly Detection suggests some actions. You can submit data for anomaly detection in batches, and soon it can also detect anomalies in streaming data. To get hands-on experience with Anomaly Detection service for both the training and detection phases, use this lab.

OCI provides multiple cloud services to handle this end-to-end functionality as part of a data platform. Using these services, the two phases of training and production pipelines using Anomaly Detection service can be built based on requirements. The following graphic provides a reference architecture for end-to-end pipelines.

A graphic depicting the architecture for end-to-end pipelines using Anomaly Detection.

Conclusion: Improve pipelines with OCI Anomaly Detection

Depending on your data sources, data quality, and requirements related to processing, integrations, and operations, your pipeline can look different. To learn more about all your options, related services, and how to use OCI Anomaly Detection service in end-to-end data pipelines, refer to the Anomaly Detection solution playbook.

Reference

In this blog, all the Oracle Cloud Infrastructure data and AI-related services are not shown or discussed. The following services are included in the diagrams at a high level: