The Streaming service provides a fully managed, scalable, and durable storage for ingesting continuous, high-volume streams of data that you can consume and process in real-time. Streaming can be used for messaging, ingesting high volume data such as application log data, operational telemetry data, web click-stream data or other use cases in which data is produced and processed continually and sequentially in a publish-subscribe messaging model.
This blog post show how to provision an instance and the first steps.
First, you need some configurations for use and access OSS.
The administrator needs to follow these steps:
1. Create a user.
2. Create or join a group.
3. Create or join a compartment.
4. Create and assign policies to the compartment.
Create an instance
1. From the navigation menu in the Console, select Analytics and then select Streaming.
2. Click Create Stream.
3. Create a stream called “demo” in the compartment in which your user resides. Choose your compartment. If is your first one you can check the box “Create a stream poll for me”
4. Wait until the stream is created and active
Now let's get some configurations for use the CLI
1. Find the OCID of the compartment that you belong to. From the navigation menu, select Identity, and then select Compartments.
2. Select the compartment in which your user resides.
3. Next to the OCID field, click Copy.
4. Download the “Oracle Cloud Infrastructure CLI”, and add it to your path.
Configuration example file
run the following command
The expected result is a list of all active streams that are in the compartment, displayed in JSON format.
In the Console, select the stream that you want to monitor
In the Resources section, click Produce Monitoring Charts to monitor produce requests. Click Consume Monitoring Charts to monitor consumer requests.
The monitoring in the Streaming service focuses on producers and consumers. For a list of the metrics emitted by the Streaming service, see the documentation.
Each metric available in the Console provides the following statistics:
• Rate, Sum, and Mean
• Min, Max, and Count
• P50, P90, P95, P99, and P99.9
These statistics offer four time intervals:
• 1 minute
• 5 minutes
• 1 hour
The following steps show how to set an alarm, using Put Messages Latency alarm as an example:
1. In the Resources section of the stream details page, click Produce Monitoring Charts.
2. From the Options menu, select Create an Alarm on this Query.
3. Fill out the details of the alarm. For descriptions of the fields, see "To create an alarm (any kind)" in Managing Alarms.
4. Add notifications, as needed.
5. Click Save Alarm
What metrics should I set alarms for?
For producers, consider setting alarms on the following metrics:
• Put Messages Latency: An increase in latency means that the messages are taking longer to publish, which could indicate network issues.
• Put Messages Total Throughput:
◦ An important increase in total throughput could indicate that the 1 MB/s per partition limit will be reached and that event will trigger the throttling mechanism.
◦ An important decrease could mean that the client producer is having an issue or is about to stop.
• Put Messages Throttled Records: It's important to get notified when messages are throttled.
• Put Messages Failure: It's important to get notified if Put messages start failing so that the Ops team can start investigating the reasons.
For consumers, consider setting the same alarms based on the following metrics:
• Get Messages Latency
• Get Messages Total Throughput
• Get Messages Throttled Requests
• Get Messages Failure
A healthy stream is a stream that is active: messages are received successfully, and messages are consumed successfully.
Writes to the service are durable. If you can produce to your stream, and if you get a successful response, then the stream is healthy.
After data is ingested, it is accessible to consumers for the configured retention period.
If Get Messages API calls return elevated levels of internal server errors, the service isn't healthy.
A healthy stream also has healthy metrics:
• Put Messages Latency is low.
• Put Messages Total Throughput is close to 1 MB/s per partition.
• Put Messages Throttled Records is close to 0.
• Put Messages Failure is close to 0.
• Get Messages Latency is low.
• Get Messages Total Throughput is close to 2 MB/s per partition.
• Get Messages Throttled Requests is close to 0.
• Get Messages Failure is close to 0.
To get started working with this stream in code, download the Java SDK and make sure it's on your classpath.
OSS is a Kafka compatible, secure, no lock-in, pay as you use, scalable, cheap streaming solution that allows the purpose mentioned with very low effort and ease to develop and deploy, and this means that you can use any Kafka API as well. Oracle recommends using official confluent Kafka Java API.
Photo by kowalikus on Unsplash