We’re announcing the upcoming release of a new managed Kafka service: Oracle Cloud Infrastructure (OCI) Streaming with Apache Kafka. The service is currently in limited availability and will be generally available later this fall. 

Apache Kafka is an open-source distributed event streaming platform that 80% of Fortune 100 companies use for high-performance data integration and mission-critical applications. OCI Streaming with Apache Kafka is a fully managed Kafka service in OCI, allowing you to build real-time, distributed data streaming pipelines, which can collect, process, store and move millions of events per minute in a cost-efficient manner. It’s 100% compatible with open-source Apache Kafka. 

OCI Streaming with Apache Kafka Overview

OCI Streaming with Apache Kafka enables you to run your Kafka clusters easily within your tenancy and offers all the functional capabilities of the latest version of open-source Apache Kafka, plus enhanced performance, security, and high availability, thanks to OCI’s unique architecture. It offers the following features and benefits:

  • Automatic patches, upgrades, and backups
  • Cloud-scale, allowing you to only pay for what you use
  • Automatic cluster healing without any downtime to your workloads, achieving high availability with a 99.9% availability service license agreement (SLA).
  • Fine-grained security access control through Kafka access control lists (ACLs) and using Salted Challenge Response Authentication Mechanism (SASL/SCRAM) or mTLS-based authentication mechanisms.
  • Compatibility with existing open-source Kafka connectors and tools. 

Use cases

Popular use cases of Kafka include the following examples. 

Change data capture

Change data capture (CDC) is a style of application design where state changes are logged as a time-ordered sequence of records. OCI Streaming with Apache Kafka’s support for cloud- scale stored log data makes it an excellent back-end for an application built in this style. You can deploy any open-source Kafka connectors, such as Debezium CDC connectors, in a virtual machine (VM), which polls the source databases for new or changed data based on an update timestamp column and easily stream your data into OCI Streaming with Apache Kafka, as shown in Figure 1. For example: E-commerce companies use CDC with Kafka to track order updates in their database to initiate order processing and other order fulfillment micro services. 

Example of change data capture architecture with a database sending data using OCI Streaming with Apache Kafka to a downstream system
Figure 1: Change data capture architecture

 

Metrics and logs ingestion

OCI Streaming with Apache Kafka can be used as the metrics or log processor from diverse sources. Log ingestion tools like Fluentd, Logstash etc. or Kafka Producer API can collect logs from various applications and place them in Kafka topics for data enrichment and aggregation. OCI Streaming with Apache Kafka can enrich the data by abstracting the details of logs and sending them to  downstream analytics tools for further processing and advance log search capabilities, as shown in Figure 2.

Logs are gathered by OCI Streaming with Apache Kafka to OCI Logging Analytics service.
Figure 2: Metrics and logs ingestion architecture

 

Real-time analytics

OCI Streaming with Apache Kafka can process and analyze continuous streams of data from IOT devices or other upstream applications for real-time insights, anomaly detection, and predictive analytics. For example, financial institutions use the service to process market data feeds, detect trading anomalies, and make real-time trading decisions. Retailers analyze customer behavior and preferences in real-time to offer personalized recommendations and promotions, as shown in Figure 2.

You can analyze all logs from demand forecasting, internet of things (IOT) data, detect trading anomalies, healthcare analytics, user behavior, and fraud detection in real-time with OCI Streaming with Apache Kafka.
Figure 3: Real-time analytics architecture

 

Web and mobile activity data ingestion

You can use OCI Streaming with Apache Kafka to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. These feeds are available for subscription for a range of use cases, including real-time processing, real-time monitoring, and loading into Hadoop or offline data warehousing systems for offline processing and reporting. You can use this solution for the following uses:

  • Clickstream: Clickstream use cases involve collecting website activity data from multiple producers, and analyzing the data in real-time to provide recommendations, such as products to buy, news articles to read, and videos to watch.
  • Game analytics: Gaming companies constantly monitor network lag, user behavior, and in-game activities to offer customers in-game microtransactions, to re-balance network load, change rendering engine parameters, and more. All these actions happen in real- time, in the order of milliseconds to a few seconds.
Various data is ingested into Oracle Autonomous Database using OCI Streaming with Apache Kafka.
Figure 4: Web and mobile activity data ingestion

 

Key takeaways

OCI Streaming with Apache Kafka marks a significant milestone in Oracle’s commitment to delivering high-performance, scalable, and secure data streaming solutions. With OCI’s managed Kafka service, users benefit from the robust capabilities of open source Kafka, enhanced by Oracle’s cutting-edge infrastructure, offering automatic upgrades, fine-grained security, and extreme availability and price-performance.

As we move toward general availability later this fall, we invite you to explore the potential of OCI Streaming with Apache Kafka through our limited availability program.Experience firsthand how it can transform your data streaming needs into a competitive advantage.

If you would like to get more information about the service in Limited Availability, please share your details by completing this form.