X

Innovative ideas for every utility

Plug-In Batch - A new way of writing flexible batch processes quickly

Anthony Shorten
Senior Principal Product Manager

In Oracle Utilities Application Framework V4.3.0.4.0, we introduced a capability that fundamentally changed the way that bulk (i.e.batch) processing operated. In the past writing code for batch processing meant writing java programs using the Oracle Utilities SDK and building your batch process using a powerful but largely code based approach.

We looked at ways we can improve the experience and reduce implementation costs and also support the Oracle Cloud. We reduced the concepts into fundamental principals associated with processing data to look for improvements.

Fundamentally, batch processing is about selecting the records you want to process, whether that is in a database or in a file, and then processing that information including extracting the information. Therefore we decided to segment the processing into several areas:

  • Determine Set of Records to process. This becomes the method of determining the subset or record set that the batch process needs to process. This can be as simple as an SQL statement to SELECT the records for processing or maybe a file adapter to read the records off a file in various formats.
  • Process Each Record. Once you identify the record to process you should pass each record to something to process it. This can be some some function to process the record within the product or even a file writer to write the information, in various formats, as an extract.
  • Additional Processing Due to Success or Failure. To finish off the segmentation there may be processing that needs to be performed if the process records completes successfully or not at a global level. This would be additional processing for error conditions or even footer processing. It also allowed for jobs to communicate to other jobs (to support job spawning).

For example:

Example Processing

This the Plug-In Batch capability was born. The capability has the following features:

  • Algorithms for each segment. We implemented an algorithm spot for each segment with the following available:
    • Select Records. Determine the subset of records to process. This algorithm primarily uses an SQL statement to determine the processing set.
    • File Upload. This is a specialist algorithm to read files for upload/import processes supporting multiple formats and even automatic decompression (for cloud customers only at this stage).
    • Process Record. This is the algorithm to process the records passed from the Select Records or File Upload algorithm. This algorithm can also be used to write data to a file via the File Adapter (supporting automatic compression for cloud customers).
    • Post Processing. Any processing that needs to be performed after the Process Record algorithm has completed all processing successfully. Use of this algorithm is optional.
    • Error Post Processing. Any processing that needs to be performed after the Process Record algorithm has completed all processing but was not successful. Use of this algorithm is optional.

For example:

  • Most of the processing is handled by the Framework. One of the big advantages of the Plug-In Batch is that the amount of code that actually needs to be built is substantially less than traditional batch coding needed by the Oracle Utilities SDK approach. This is because internally the Oracle Utilities Application Framework takes a bigger role by taking over things like restart logic, interfaces to the architecture, integration to batch instance tables, interfacing to schedulers, etc. In more traditional approaches, that may exist as additional code that is not really necessary in most cases.
  • Quicker Development. By focusing development of batch to critical components of the architecture the development time for Plug In Batch can be substantially lower than using traditional methods.
  • Performant. As the Plug In Batch uses the same technology as other part of the Oracle Utilities Application Framework and the fact that the code is internally expressed as byte code, there are little or no performance issues with using the Plug-In approach.
  • Cloud Ready. The use of Plug-In Batch is the only supported way of building and maintaining batch processes for Oracle Utilities SaaS Cloud Services. The use of this capability is not exclusive to the Oracle Utilities SaaS Cloud Services and can be used with on-premise, IaaS and PaaS implementations of Oracle Utilities products.
  • Maximum Flexibility. By segregating the processing from the record selection, the batch process is easier to maintain and become flexible. For example, if data skew is detected, then the Select Records can be altered to address this without having to introduce or redeploy changes to the processing.
  • No deployment necessary. One of the biggest reasons Plug-In Batch is used on the Oracle Utilities SaaS Cloud is that there is no deployment necessary to implement the code. This removes the need for additional processes to compile and deploy the code saving both cost and time and without the need for any outages.
  • Lots of examples. The products teams are slowly migrating all their existing processes to Plug-In Batch a large number of code can be copied and reused to build extensions. This includes a set of templates for common use cases ready to shortcut Plug-In Batch development.

It is recommended that implementations writing new or updated batch processes use Plug-In Batch to take advantage of the capabilities and flexibility offered. For more information about Plug In Batch refer to the online documentation available with your Oracle Utilities product.

Join the discussion

Comments ( 1 )
  • Abhijit Wednesday, September 9, 2020
    For select statement limitation is you can add only 2000 char length SQL query.
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.