We’re thrilled to announce the general availability of Oracle Cloud Infrastructure (OCI) Queue as a source for OCI Connector Hub. This integration that enables you to move data from OCI Queue to various OCI Connector Hub targets in a scalable and reliable way. This feature allows you to apply the benefits of both the services to build event-driven applications and microservices with ease. 

OCI Connector Hub is a cloud message bus platform for describing, executing, and monitoring data movements between OCI services. OCI Queue is a service for enabling asynchronous, decoupled communication in a serverless manner. OCI Queue handles high-volume transactional data that requires independent processing without loss or duplication. The service uses open standards of STOMP and OpenAPI-defined REST endpoints to communicate with any client.

Integrating OCI Queue and OCI Functions using OCI Connector Hub

To integrate the two services, you must know how to create an OCI queue and how to create an OCI function.

One of the major use cases for our customers is having the ability to trigger functions using queue. A queue-triggered function runs code to process messages in the queue. By creating a Connector Hub connector with OCI Queue as the source and the function as target, you can move the data from the source queue to a function and process millions of messages per day.

Batching

The connector fetches the data from the queue and sends the data to the target function in batches. Batching can be based on either the number of messages or message size and time in seconds. The function is invoked only when either of the batch threshold, such as number of messages, message size, or time, is hit. This feature helps you reduce the number of invocations and optimize the cost.

Function invocation and message handling

A connector invokes Functions in a serial synchronous manner. You can create multiple connectors with the same source queue to enable parallel function invocations. Messages are only deleted from the queue when the connector has successfully delivered the message to the function. If a function fails, all messages in the batch immediately become available again for consumption by the connector. Messages might not be available for consumption by a connector if they’re moved into the dead-letter queue (DLQ). To retry processing these messages, you can set up another connector with the DLQ source and the function target. If you encounter a partial function failure, the connector marks the whole batch as failed, and all the messages become visible again in the queue.

Polling

Connectors use long polling to read messages from queues. Long polling is useful to prevent tight loop retries when the queue is empty and maximizing efficiency of the connection between the services.

Message filtering

You can filter messages present in a queue by specifying an optional channel ID. The connector consumes only the messages that match the channel ID. If the user doesn’t specify any selector, all messages from the queue are consumed.

How to use a queue as a source

Go to Connector Hub page and select Create Connector.

Connectors in an example compartment in the Oracle Cloud Console.

Select Queue as the source and Functions as the target.

The Create connector screen with the source and target configuration section outlined in orange.

Under Configure Source, select the compartment, queue name, and enter optional message filtering details.

configure source

Under Configure Target, select the compartment, function application, and function name. Under Show Additional Options, you can specify the batch size and batch time. Batch size is a menu from which you can select the batch size in KB or number of messages with the following options:

  • Batch size (Messages): You can configure the batch size in number of messages between 1–10,000. The default value is 100.

  • Batch size (KBs): You can configure the batch size in KBs between 1–6,144. Default value is 6,144.

  • Batch time limit: You can configure the batch time limit to specify the amount of time in seconds to gather the requests before invoking the function between 60–600 seconds. The default value is 60 seconds.

The create connector screen with the options for batch type, size limit, and time limit outlined in orange.

You can create default policies with a single selection. You can also enable Connector Logs to easily manage your connector and troubleshoot any issues.

The Create connector screen with the options to create default policies to allow the connector to read and write from the queue to the function and the Enable Logs toggle outlined in orange.

Select the Create button to create the connector, and you’re done! You can go to the source queue or target function following the links under the Source and Target Connection sections to manage them.

The Connector details page, showing the source and target connections.

Pricing

Using Connector Hub incurs no charges. However, source, task, and target service usage charges apply. For a queue source, because Connector hub polls the queue, you might see extra Queue charges. For more information, see Pricing.

Try the feature today!

Connector Hub is accessible through the Oracle Cloud Console under the Analytics and AI section, under the Messaging subsection. It’s also available through software development kit (SDK), CLI, REST API, and Terraform. To learn more about Connector Hub, see the Connector Hub documentation.

To try Connector Hub for yourself, sign up for the Oracle Cloud Free Tier or sign into your account.