Friday May 08, 2009

Right tools for the job? - Interesting discoveries when processing large volume of data

Just now I had an occasion to work with an integration solution intended to process lots of records. By lots I mean over 1 million smallish records. My customary platform to experiment on is Windows XP. Lots of reasons for that, most of them historical – I have tools I know and like and so on. While trying to work with such a volume of data I noticed a number of “interesting” things, which I thought I should share. These things are related to both the platforms (Windows vs. Linux), the tools and the architectural decisions.

I needed lots off data to test the solution I was contemplating, which involved XML processing, to see how constructing and parsing XML affects solution performance. To make it easier to compare timing differences I though I should use lots of records.

The discoveries are discussed in Right tools for the job.pdf.

Thursday May 07, 2009

Getting Hundreds of Files using Batch Local File eWay in Java CAPS 6

Occasionally one needs to pick up and process a large number of files, on the order of hundreds or thousands. With the Batch Inbound eWay/JCA Adapter it is not possible to pick up more then one file per poll. The Batch Local File, if triggered by some event other then an appearance of a file in a directory, perhaps a Scheduler trigger of a manual trigger, with correctly designed logic can process many files in a single invocation.

The document, ProcessingHundredsOfFileWithBatchAdapter.pdf, discusses how Batch Local File-based solution can be constructed to effectively process hundreds of files in a single pass.

Saturday Feb 28, 2009

Java CAPS Quick Note 001 - Split file into multiple files

This Quick Note discusses a solution to the use case provided by Richard Kuiter.

An input file contains the following records:

H100000000000014099120ASN00507  
L1140991200000008261850826185738
L1140991200000008261850826185738
L1140991200000008261850826185738
L1140991200000008261850826185738
L1140991200000008261850826185738
L1140991200000008261850826185738
H100000000000014099126ASN00531  
L1140991260000008262690826269662
L1140991260000008262690826269662
L1140991260000008262690826269662
L1140991260000008262690826269662
L1140991260000008262690826269662

It is required that each block of records starting with the H1 (header) record and containing all the following L1 (line) records, be written to a different file.

The solution involves the use of:
1.    Batch Inbound eWay to locate the input file and provide its name and location to a Java Collaboration Definition
2.    Batch Local File eWay to provide an Input Stream to the Batch Record eWay
3.    Batch Record eWay to break up the input stream into records delimited by carriage return+new line
4.    Batch Local File eWay to write each block of records to a file with a distinct name

Brief steps to implement this solution are given in the full Quick Note at http://mediacast.sun.com/users/Michael.Czapski-Sun/media/QuickNote_001/details. The collaboration code will work in Java CAPS 5 and 6 Repository.

Monday Jul 28, 2008

Java CAPS 6, JCA, Note 7 - Batch Inbound-Triggered JCA (Non)Transacitonality

In Note 6 we explored the transactional behavior of a JCA MDB invoked by a JMS Adapter and orchestrating an Oracle JCA Adapter (a transactional end point) and a Batch Local File JCA Adapter (a non-transactional end point). In this Note let’s explore the issue of transactionality of a JCA Message-Driven Bean invoked by a non-transactional end point.


Let’s take the example from the “Java CAPS Basics: Implementing Common EAI Patterns Companion CD” book, ISBN: 0-13-713071-6, Chapter 11 “Scalability and Resilience”, Section 11.2 “Exception Handling”, subsection 11.2.1 “Exceptions in Java Collaborations”, 11.2.1.2 “Other Java Collaborations”. The book from which this section comes is available on the Companion CD. Let’s re-work this example using Java CAPS 6 JCA Adapters.


The example illustrates exception processing behaviour involving a Batch Inbound Adapter-triggered JCA Message-Driven Bean.
The Batch Inbound Adapter is designed to poll a directory for a file. When it finds a file that matches the name, or the name pattern, it immediately renames the fie by prepending a GUID to the original name then triggers the MDB and passes to it the original name of the file, the current name of the file and the directory in which the file was found. This behavior prevents other possible file pollers from getting hold of the file and gives the first comer exclusive access to the file. The MDB is designed to take the name of the file as given and rename the file to the original name with the suffix “~.in” appended, to indicate the file was read and processed. To do this the MDB will use the Batch Local File JCA Adapter’s capability to post-process the file by renaming or deleting it. To explore the transactional behavior the MDB will explicitly throw an exception after it renames the file.
The complete Note is available in 07BatcyhInboundTriggeredJCANonTransacionality.pdf.

Sunday Jul 27, 2008

Java CAPS 6, JCA, Note 6, JMS-Triggered JCA with Oracle and Batch Local File - Exploration of transactionality

Rather then inventing an example to discuss and illustrate the use of the Oracle JCA Adapter let’s build a solutions that uses the Oracle JCA Adapter and shows additional Java CAPS 6 facilities of interest.


Let’s take the example from the "Java CAPS Basics: Implementing Common EAI Patterns Companion CD" book, ISBN: 0-13-713071-6, Chapter 11 "Scalability and Resilience", Section 11.2 "Exception Handling", subsection 11.2.1 "Exceptions in Java Collaborations", 11.2.1.1 "JMS-Triggered Java Collaborations". The book from which this section comes is available on the Companion CD. Let’s re-work this example using Java CAPS 6 JCA Adapters.


This example illustrates exception handling involving a JMS-triggered JCA Message-Driven Bean.
The MDB is designed to receive a JMS message, update a field in a database table row with the value of the text message, write the text message, together with the timestamp, to a file and finish.


The complete Note is provided in 06JMSTriggeredJCAwithOracle.pdf


Update: Thanks to Dao Tien Tran for pointing out a typo on page 5. I corrected the typo and uploaded the corrected document.  Change is marked in color and with a change bar.




Friday Jul 18, 2008

Java CAPS 6, Using JCA and JBI, Note 3, Batch Inbound, through Batch Local File to BPEL 2.0

Java CAPS 6 has the 5.x compatibility infrastructure which allows one to import 5.x projects right into Java CAPS 6, build, deploy and run without changes. One can also develop repository-based projects in Java CAPS 6 – that’s the 5.x-style projects.  This is the old way of developing Java CAPS solutions – still good and valid.

If one were to decide to not use the old way there is the JBI infrastructure, which allows development of solutions that use BPEL Service Engine, XSLT Service Engine, IEP Service Engine, Java EE Service Engine, etc., and a variety of Binding Components. The implication is that business logic is implemented in BPEL 2.0, which is used to orchestrate other services and resources, including interaction with external systems through Binding Components. This is the new way of developing Java CAPS solutions – 100% compatible with the Open Source OpenESB project since it uses the OpenESB project-developed container and components.

Someone might ask “so what happened to eGate?”. “eGate” meaning Java Collaboration Definition-like logic components, eWays and the JMS messaging backbone.

While the facility seems underadvertised/downplayed, Java CAPS 6 provides a number of 5.1 eWay-based JCA Adapters and a moderately easy means of developing JCA Message-Driven Beans that can use these adapters to implement JCD-like logic components and, effectively, eGate-like solutions that do not use BPEL or the JBI infrastructure.

This Note discusses and illustrates the implementation of a mixed Java CAPS 5.x-like integration solution that retrieves a file from the local file system using JCA Adapters and passes its content to a BPEL 2.0 process executing in the JBI container. This requirement I have seen and heard of being implemented in 5.x many times by many customers.

Most of the material in the first 16 pages of this Note is the same as in Note 2.

The JCA Message-Driven Bean, the piece of JCD-like Java logic, will be triggered by a Batch Inbound Adapter (what one would have called the Batch  Inbound eWay in 5.1), will read the content of the file using the Batch Local File Adapter (eWay) and will send the payload as a string to a BPEL 2.0 Business Process, which will be triggered by this message and will execute in the JBI container. The batch Inbound Adapter will be configured to use a regular expression to match the name of the file. Once it finds the file it will rename the file by prepending the GUID to the name and will pass the new name, the original name and the directory path to the Java code. This is exactly what the 5.1 Batch Inbound does. The JCA MDB will use the new name, the original name and the directory path to dynamically configure the Batch Local File Adapter to retrieve the file content and rename the file (post transfer) to the original name with some string appended to indicate that the file was processed. This, too, is exactly what one would do in a 5.1 JCD in the same circumstance. Once the payload is available the JCA MDB will use the OneWay WSDL interface and the JBI NMR to send it, as a String, to a BPEL 2.0 process.  Both the JCA MDB and the BPEL process will be a part of the same JBI Composite Application and will communicate with one another using the Normalized Message Router (NMR).

The entire Note is available in 03JCA-BInboundThroughBLFToBPEL2.0.pdf

Wednesday Jul 16, 2008

Java CAPS 6, Using JCA, Note 2, Batch Inbound, through Batch Local File to JMS

Java CAPS 6 has the 5.x compatibility infrastructure which allows one to import 5.x projects right into Java CAPS 6, build, deploy and run without changes. One can also develop repository-based projects in Java CAPS 6 – that’s the 5.x-style projects.  This is the old way of developing Java CAPS solutions – still good and valid.


If one were to decide to not use the old way there is the JBI infrastructure, which allows development of solutions that use BPEL Service Engine, XSLT Service Engine, IEP Service Engine, Java EE Service Engine, etc., and a variety of Binding Components. The implication is that business logic is implemented in BPEL 2.0, which is used to orchestrate other services and resources, including interaction with external systems through Binding Components. This is the new way of developing Java CAPS solutions – 100% compatible with the Open Source OpenESB project since it uses the OpenESB project-developed container and components.


Someone might ask “so what happened to eGate?”. “eGate” meaning Java Collaboration Definition-like logic components, eWays and the JMS messaging backbone.


While the facility seems underadvertised/downplayed, Java CAPS 6 provides a number of 5.1 eWay-based JCA Adapters and a moderately easy means of developing JCA Message-Driven Beans that can use these adapters to implement JCD-like logic components and, effectively, eGate-like solutions that do not use BPEL or the JBI infrastructure.


This Note discusses and illustrates the implementation of a JCD-like integration solution that retrieves a file from the local file system and writes its content to a JMS destination. This requirement I have seen and heard of being implemented in 5.x many times by many customers.


The JCA Message-Driven Bean, the piece of JCD-like Java logic, will be triggered by a Batch Inbound Adapter (what one would have called the Batch  Inbound eWay in 5.1), will read the content of the file using the Batch Local File Adapter (eWay) and will write the payload as a string to a JMS destination. The batch Inbound Adapter will be configured to use a regular expression to match the name of the file. Once it finds the file it will rename the file by prepending the GUID to the name and will pass the new name, the original name and the directory path to the Java code. This is exactly what the 5.1 Batch Inbound does. The JCA MDB will use the new name, the original name and the directory path to dynamically configure the Batch Local File Adapter to retrieve the file content and rename the file (post transfer) to the original name with some string appended to indicate that the file was processed. This, too, is exactly what one would do in a 5.1 JCD in the same circumstance. Once the payload is available the JCA MDB will use the JMS OTD to send it, as a TextMessage, to a JMS Queue.  Again, this is something that a 5.x JCD would do.


In effect, this Note describes and illustrates the process of re-creating a 5.x Java Collaboration Definition using Java CAPS 6, but instead of using the repository-based approach it is using JCA MDBs and JCA Adapters.


Complete text of the Note is in 02JCA-BInboundThroughBLFToJMS.pdf

About

In this Blog I post abstracts of articles / writeups / notes on various aspects of Java CAPS and SOA Suite including solutions, discussions and screencasts. The links to the referenced material are included in the bodies of the abstracts.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today