X

The Integration blog covers the latest in product updates, best practices, customer stories, and more.

Downstream Throttling in Oracle Integration Cloud via Parking Lot Pattern

Chelsea Zhang
Software Engineer

Background

Parking lot pattern is a mature design to store the data in an intermediate stage before processing the data from the intermediate stage to the end system based on the required rate. The detailed implementation of the parking lot pattern can be done in a variety of  storage technologies but strongly recommended a database table to be used for simplicity. In this blog, we will use the parking lot pattern in the Oracle Integration Cloud (OIC) to explore a solution in the downstream throttling.

 

Problem Definition

In OIC, the downstream throttling was often mentioned as there might be an influx of data that overwhelm the slower downstream systems. Even though, it might be accomplished by the tuning knobs within OIC and WebLogic Server, but when the built-in tuning cannot be adjusted enough capacity to stop flooding the slower system. The parking lot pattern will enable to provide a solution to attend to this scenario.

 

Design Solution

 

 

  1. Process the input data/message based on the order they come in
  2. Each message will be parked in the storage for x minutes (parking time) so the system has a chance to throttle the number of messages processed concurrently
  3. Maximal number of parallel process can be configured to throttle the outgoing calls
  4. A set of integration, connections and database scripts are used the current out of the box OIC features
  5. The solution is generic, which can be used in various typed business integrations without modification to the provided integrations
  6. Error handling of both system/network error and bad requests

 

Database Schema

 

The key piece to the parking lot pattern is the database schema in the design pattern. The Message table is explained here:

Column

Description

ID (NUMBER)

This is the unique ID/key for the row in the table.

STATUS (VARCHAR)

This will be used for state management and logical delete

with the database adapter. There are three values this column will hold:

1.

N - New (Not Processed)

2.

P - Processing (In-flight interaction with slower system)

3.

C - Complete (Slower system responded to interaction)

The database adapter will poll for ‘N’ew rows and will mark the

 row as ‘P’rocessing when it hands it over to an OIC integration.

PAYLOAD (CLOB)

The message that would normally be associated with a

component is stored here as an XML clob.

 

 

Implementation Details

Integration Details

Sample Integration in Par File

Connections

  • Business Front-end Integrations
    • Receive the typed business payload
    • Call producer with opaque interface

 

EmployeeServiceFront(1.0)

 

OrderServiceFront(1.0)

Invoke: EmployeeService

 

OrderService

  • Producer
    • Receive new record
    • Create a new row in group table if not exit
    • Mark the group status as ‘N’

 

RSQProducer(1.0)

Trigger and Invoke: RSQProducer

 

Invoke: RSQ DB

  • Group Consumer
    • Schedule to run every x minutes
    • Invoke a message consumer

 

RSQGroupConsumer(1.0)

Trigger and Invoke:

RQSMessageConsumer

 

Invoke: RSQ DB

  • Message Consumer
    • Mark message status as ‘P’
    • Invoke Dispatcher
    • Delete the message

 

RSQMessageConsumer(1.0)

Trigger and Invoke:

RQSMessageConsumer

 

Invoke: RSQ DB

  • Dispatcher
    • Receive the message
    • Convert opaque payload to the original typed business payload
    • Find the business end point and invoke

 

RSQDispatcher(1.0)

Trigger and Invoke: RSQDispatcher, OrderService, TestService

 

 

Screen Shot of Actual Integration Steps

You can deploy this par file.  It has the following connections that need configuring and activating.

1. Import the par file to the Packages in Oracle Integration:

 

 

2. Invoke and Trigger Connections: initial unconnected status

 

3. Configure and activate Database, the provided sample database called RSQ DB; Oracle Autonomous Transaction Processing (ATP) database is used in this scenario (ATP database information can be found here deploying an ATP instance in Oracle Cloud

 

4. Trigger and Invoke Message Consumer; in the sample it is called RSQMessageConsumer to cause load distribution of calls to message consumer. It requires the connection URL and corresponding admin authentication.

Processes active messages of the given group.

  1. Receives group id/type from the group consumer.
  2. Loads active messages of the group ordered by sequence-id. The messages have to be at least # (parking time) old.
  3. Loops through active messages
  1. Marks message status as 'P'
  2. Invoke the Dispatcher using its opaque interface.
  3. Delete the message
  1. Calls a stored procedure to:
  1. Mark the group status to be 'C' if there are no active messages
  2. Mark the group status to be 'N' if there are new active messages

 

 

5. Trigger the manager interface; in the sample, it connected the RQSManager is used to invoke the parking lot pattern interface in the Oracle Integration. It currently support three operations: Get configs; Update configs; Recover group

 

 

 

6.  Producer Integration connected with database is used to invoke producer interface; in the sample it is called the RSQProducer. Entry point of the parking lot pattern. Receives the resequencing message.

 

  1. Creates a new row in group table if it's not already there.
  2. Marks the status of the group to 'N'.
  3. Creates a message in the message table.

 

7.  Dispatcher is a request/response integration which reconstructs the original payload and send to the real backend integration.

  1. Receives the message.
  2. Converts opaque payload to the original typed business payload.
  3. Uses the group id to find the business end point and invoke it synchronously.

8. Business Integrations are the real integrations that customers used to processes the business messages. They have their own typed interface. I used two test servers to demonstrate some post data

 

9. Error Handling

Recover System/network error If the problem is caused by some system error like networking issues. After you fixing the problem, you can recover by resubmitting the failed message consumer instance.

 

 

10. Connections and Integrations Sample after Triggers and Invokes

 

 

 

 

 

 

Join the discussion

Comments ( 1 )
  • Kelly Monday, September 9, 2019
    Great and clear explanation.
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.