Large File Processing With JCA File Adapter and OSB
By John Graves on Aug 07, 2014
I was asked to prototype a possible use case where a huge file (10+GB) needed to be processed by OSB. Yes, this is a horrible thing, but sometimes we have to architect horrible things! This use case had a couple of important aspects:
1. The file contained many repeating records of the same type. Each record was pretty small.
2. The records in the file were not related and their order didn't matter.
This is important because if there are relationships or order requirements in the file, some more considerations would be needed. I don't go into it here, but it would be possible to attach a single threaded work manager to the OSB Proxy and send the messages into a JMS queue using WebLogic Unit of Order capabilities.
So how can we handle this?
We can't use the file transport because if we touch the payload or loop over the records, we end up bringing the whole message into memory. I don't really want to setup an 11GB JVM!
We could just write some java code to break up the file, but it would be nice to manage this in OSB. We could pass the filename into OSB and use a java callout to break the message apart, but I'd be worried about how long this might take and the result of a proxy hanging around that long on a java callout.
So, let's use the JCA File Adapter that comes with SOA Suite.
Step 1 - Setup the environment and make a schema definition.
Step 2 - Create JCA File Adapter
Step 3 - Bring JCA File Adapter into OSB - Make Proxy
Step 4 - Initial testing to output files.
Step 5 - Testing large files.
Step 6 - Testing 11GB file.