By James Taylor on Jan 13, 2015
I have been working with a customer looking at implementing an enterprise file transfer solution. Very timely as Oracle released Managed File Transfer (MFT) as part of the SOA 12c launch which is a great fit for this use case.
I'm going to put together a series of MFT posts to help customers implement some advance use cases I have implemented in order to demonstrate the MFT capability.
So I don't reinvent the wheel I should make mention of the following tutorials that have been released by product management. I highly recommend doing these tutorials before trying this post as this blog assumes some familiarity.
To complete this example I recommend you complete these tutorials to become familiar with the basics as these blog entries focus on more advance use cases and assumes you have a basic level of MFT functionality.
This first post is going to show how you can transform large batch
files from a comma delimited file to a positional file. I have not
performed any tuning on my environment and have only tested to 200MB without issue. Having said that I have modified my JMV settings to the following:
-Xms2048m -Xmx4096m -XX:PermSize=1024m -XX:MaxPermSize=4096m
If you find you start to hit some memory issues refer to the SOA Developers Guide which provides all the tuning tips for managing large files.
One of the new features of SOA Suite 12C is the introduction of BPEL
templates. I will be utilising these to accelerate the development of
If you didn't know by now, MFT is a file transfer tool. Having said that, it does not possesses transformation capability. Therefore if you want to transform files to a corporate standard or canonical model you need to pass the transformation capability to something else like SOA Suite or ODI. In this example I will transform a simple csv and transform it to a positional file using Oracle BPEL 12.1.3.
I want to have complete visibility of the process so I'm
implementing the the Pass-By-Reference use case where MFT sends the file
to BPEL for transformation as a reference. BPEL then transforms the
file to a temporary location, then MFT sends the file to the correct
To complete this example you will need to complete the following prerequisites:
Download the following file that has all at resources required to complete the steps in this example.
An explanation of each file has been provided in the table below:
|address_book_200meg.zip|| Input example
||The SOA Project Template that will be used to create the composite
||The MFT service that is invoked from the SOA Composite once it has completed its transformation
||This is the MFT process that receives the file from the embedded FTP Server and passes it to the SOA Composite
||This is the custom activity that implement the chunk read so it can transform large files.
- Unzip this file to a temporary location on your development PC.
- Start JDeveloper and navigate to the following location
Tools -> Preferences -> SOA -> Templates
This will provide the location where you need to load the templates ProcessBatchFile.tmpl and MFTBatch.tmpl. If you want you can create your own location or used the SOA MDS.
- Copy the 2 files ProcessBatchFile.tmpl and MFTBatch.tmpl from the resource file to the template location identified above.
Before I start to create the MFT configuration I need to have the BPEL process created as this will be the destination. If you want to build from scratch you can follow the tutorials posted above. In this example I will be utilizing templates to minimize the repetitive tasks.
Create BPEL Process
- Create a SOA Application with the name MFTBatchProcessor.
- Create a SOA Project MFTBatchProcess, and choose the SOA Template MFTBatch downloaded above. If you don't see make sure the templates are loaded into the correct directory. To find the destination, in JDeveloper got to the Tools -> Preferences -> SOA -> Templates. This will specify the location.
- This should create a composite that looks like this
- If you double click the BPEL process you will see most of the code in the tutorial above already completed for you. This template provides more functionality than is required, but I wanted a template that was generic for type of load. In this example I'm MFT is used as the embedded FTP Server therefore the channel that will be used is the FileRefFile.
In the the BPEL process under the AssignFileDirectory drag the custom activity template ProcessBatchFile. Once again if the custom activity doesn't appear check the template location.
- Click Finish to accept the defaults. This will load the ReadChunkFile configuration for the batch load. If the partner link for InvokeReadBatchFile doesn't appear, close the BPEL process and reopen it. Also for some reason the labels don't appear they should read Yes for left hand side, and No for right hand side.
What you have done is basically done all the heavy lifting in 2 simple steps. All that is required now is to implement the transformation and make some small changes in the BPEL process to be unique to this service.
- When I configured the template I did not specify a format as this will be unique for each process. Therefore we must edit the ReadBatchFile file adapter. Go to the composite view and double click the ReadBatchFile reference and click Next to step 7. Ignore the previous steps as these variables are set dynamically by MFT.
Uncheck the Native format translation is not required check box and import the address_csv.xsd and select the element <Root-Element>
- Click Next and Finish
- We have now completed the steps to read the file into BPEL from the directory reference sent by MFT. What we need to do now is transform the file and send the reference back to MFT. To do this we need to use the file adapter. We want to avoid putting large payloads into large XML DOMs to avoid memory issues.
In the Composite add a File Adapter to the Reference swimlane.
- Name the adapter WriteBatchFile
- Click Next to Step 4 and Select the Write radio box.
- In step 5 specify a temporary file location that will store the transformed file. MFT will pick this file up from this location. It does not matter what you name this file as the template provides a unique name. It is also important that you select the Append to existing file checkbox. Remember the file location as you will need to update an Assign activity to reflect the same.
- Import the address-fixedLength.xsd as the target format and select the <Root-element>
- Open the Assign activity AssignMFTVariables that is at the bottom of the BPEL process and edit the file location specified in step 11. This is the location where BPEL saves the transformed file for MFT to pick up. Make sure you only edit the bit highlighted in blue, it must be prefixed with file://
- Wire the new File Adapter to the BPEL Process, your composite should now look like this.
- I have placed an empty DO_TRANSFORMATION activity in the ProcessBatchFile custom activity. Delete this activity and replace it with 2 activities,
XSLT Transform - TransformFixedLength
Invoke - InvokeWrtiteBatch
- Link the InvokeWriteBatch activity with the partner link WriteBatchFile. Also specify a new input variable
Click the Properties tab and enter the 2 properties, the jca.file.Directory should be the same the the file location as specified in the WriteBatchFile configuration.
- Now that the adapters have been configured we can now implement the Transformation. Double click the TransformFixedLength activity.
Set the Source variable to the Output of the ReadBatchFile partnerlink. This is specified as a scope variable as it was included in the custom activity template.
Set The Target to the the Input the the WriteBatch partnerlink.
- This blog entry is not focusing on the Transformation so the mapping is very basic. The key point in the transformation is to specify the For-Each capability. At the end of the transformation it should look like this.
- The last step is to change the location of the MFT service that takes the file reference and sends it to the desired location. I will cover this off in the following section. Until this step is done the deployment wont work.
Create the MFT Transfer Service
- Login to the MFT Console
- Select the Administration tab and select import/export
- Import the MFT file mftTransferBatch.zip from the archive location.
- This should create the transfer mftTransfer with a source and target. Click the Design tab to see if it exists.
- You may notice that the target has a location that does not suit your environment. This can be changed by going to the mftFileTarget Target. This location is the final destination where the completed transform file will reside.
Complete SOA Composite and Deploy
As mentioned above there was a final step in the SOA Composite that needed to be completed once the Transfer MFT had been deployed. This was completed in the previous step.
- Open up JDeveloper and Open the MFTBatchProcess Composite.
- Open the mftBatchReference and click Next to step 3. Here you need to create a connection you your MFT Server. This is the same as any other application server connection. Make sure you specify the correct Domain and use the AdminServer port.
- Choose the MFT Source mftSOASource and click Finish.
- You can now deploy your SOA Composite.
- Once the BPEL process has been deployed you can change the the number of rows read in each chunk read. It is currently set to 100 which is very low for large files. In this step we will raise this value to 50,000. With the file I have provided in this example this is about a 5MB chunk. Other files you create will be different, i.e. this file has 2 Million rows for a 200MB file. If you have larger rows you will reduce the chunk size accordingly. (if your JVM settings are still at the original settings you may get the error - java.lang.OutOfMemoryError: GC overhead limit exceeded)
Login to the SOA EM console and navigate to the MFTBatchProcess composite.
From the SOA Composite drop down button select Service/Reference Properites -> ReadBatchFile
- Change the ChunkSize parameter to 10,000. Then Apply the changes.
Create the Transform MFT Service
This service is the service that receives the file from the embedded FTP server and sends it to the SOA Composite created above.
- Login to the MFT Console
- Select the Administration tab and select import/export
- Import the MFT file mftTransform.zip from the archive location downloaded in the prerequisite.
- This should create the transfer mftTransform with a source and target. Click the Design tab to see if it exists.
- You will notice that the target mftSOATarget is pointing to a wrong location for the SOA Composite. This needs to be corrected to point to your environment.
To get this location login the the SOA EM Console and navigate to the mftBatchProcess and click the Test button
Copy the location minus the ?WSDL on the end.
- Back in the MFT Console open the mftSOATarget and change the location for your environment.
- Make sure you save and Deploy the changes
The process is now complete and is now ready for testing.
Using your FTP client, connect to the MFT Embedded FTP server and change the directory to /mft. (Remember this is a MFT username, not an OS username)
- FTP the address_book_200meg.dat file to that location.
- Once the FTP process has completed in the MFT Console you will see the process in action.
- Once the BPEL service has completed the transformation you will see the service completed in the MFT Console. You will also notice that the the file has grown. This is because the fix length file introduces spaces making the file bigger.
- One of the nice features in MFT is the association MFT makes with the BPEL service. If you click the link for the completed mftFTPEmbeddedSource. This will display the MFT service.
Click the mftSOATarget and navigate down to see the Correlation ID link
Click this link and this will bring up the BPEL service that performed the transformation. As you can see it did a bit of chunking.
- The final step is to check whether the file is in the correct format.
From the Dashboard, if you click the completed mftSOASource link, the the mftFileTarget you will get the information to find the file it created. In my instance it created the file mft_6.dat.
- Login to the OS a look to see if the file exists.
[oracle@homer bin]$ cd /u01/oracle/interface/outbound/mft/mft
[oracle@homer mft]$ ls -l mft_6.dat
-rw-r-----. 1 oracle dba 298504514 Jan 16 11:18 mft_6.dat
- Now view this file.
That is Part 1 completed. I hope you find this useful.