very large messages in a messaging solution may require memory resources many
times greater than the size of the largest message to be handled. Frequently
the architect has no choice but to consume or produce a very large message, a
file containing a batch or related transactions, for example, or a large and
complex XML message generated by, or intended for, an external application. Handling
such messages poses special challenges.
Java CAPS can assist with Batch eWay
support for data streaming when large messages are manifested as files in a file
If it is possible to break large messages up into components and
process components individually, or collect components and assemble them into a
large message. eTL, another of the products in the Java CAPS Suite, can assist
in processing large volumes of data. Whilst ETL (Extract, Transfer and Load) is
typically associated with one off batch extraction and load of data, Java CAPS’
eTL can be used both standalone and in-stream as part of a larger Java CAPS
solution. In this in-stream mode it will be discussed as a possible means of streaming
data between a flat file and database table or between
This extract from the soon–to-be-released book, "Java CAPS Basics
– Implementing Common EAI Patterns", discusses the Java CAPS 5.1 Batch eWay streaming
facilities, and presents and compares a number of data streaming implementations,
including an eTL implementation.
Link to Java
CAPS 5.1.3 Data Streaming section of the manuscript.