Implementing Fuji integration scenario using Camel SE

I recently tried to implement the scenarios presented in the Fuji screencast integration demos using Open ESB v2 and the new Apache Camel Service Engine. Camel is an integration framework that implements EIPs using a choice of DSL that can be used standalone or in a JBI service engine to build routing and mediations of JBI components. Similarly, Fuji provides a rapid mechanism for implementing enterprise integration patterns and integrating JBI components. It includes a light-weight JBI micro-kernel that can be installed into an OSGi container. I highly recommend you view the integration screencast before proceeding.


So if Fuji already implements these scenarios, then why did I bother writing them using Camel? I did this for a few reasons and mainly myth-busting purposes:

  1. One of the common myths about Open ESB v2 is that you need to use BPEL to mediate and orchestrate services. Using the Camel SE, it is actually possible to write mediations in Open ESB using POJOs for those that have got the 'hump' writing mediations in BPEL. Incidentally, this is a myth in itself since EJBs can also be used with Open ESB and EJB3 is itself a POJO framework.
  2. To compare what we are doing with Project Fuji. I recommend you watch the screencast to get a feel for what's coming in Open ESB v3.
  3. To show that it is possible to use Spring with Open ESB inside GlassFish.
  4. To show that Sun plays nice with other open source projects.

If you haven't done so, I recommend you study Srinivasan's excellent blog entry on Camel SE and screencast of how to get up and running with Camel SE inside Open ESB. You can read more about the Camel SE on the Open ESB wiki.

Fuji integration scenario

I decided to implement the third, combined scenario from the Fuji integration screencast. This integration requires the ESB to broadcast (i.e. multicast) an RSS feed to XMPP and File based on a static content filter. As with Fuji, we will use Camel to mediate between the RSS, File and XMPP binding components available from Open ESB. Here's the diagram borrowed from the screencast that illustrates what we're building:

Camel offers three DSL options for implementing this scenario:

  1. Java-based DSL
  2. XML-based DSL
  3. Scala DSL
I decided to implement this using Camel's Java-based DSL rather than XML or Scala. My reasons were that XML gets overused as a programming language so ruled that option out immediately and the Scala DSL looks interesting but isn't quite there yet. It will be interesting to compare IFL and Scala DSL suitability for more complex integration cases, but that's not covered here but is something I may revisit.

Steps to create the service assembly

Here's what you need to do step-by-step if you want to build the above scenario in Open ESB & Camel SE.

  1. Pre-requisite steps
  2. You will firstly need to download Open ESB, install it and start the bundled NetBeans IDE and GlassFish app server. It also helps if you have Apache Camel installed! NB: The service engine currently requires apache-camel-1.3.0.

  3. Download and install Camel SE and other components
  4. The Camel SE is still in incubator status so it is not part of the Open ESB install. You will need to download the Camel SE JBI component and project plugin for NetBeans. The (simple) installation instructions are provided on the Camel SE wiki page.

    It is also important to download and install the JAR files of the Fuji versions of the XMPP BC (here) and RSS BC (here). Although Open ESB v2 bundles these components already, there are some minor feature enhancements in the Fuji branch that are used in this guide. It is advised that you shutdown and uninstall these components from your existing JBI runtime in GlassFish before attempting to install the updated components.

  5. Create a new Camel JBI module
  6. In NetBeans, create a new SOA->Camel JBI Module project:


    Call it CamelContentFilterJBIModule and point to your Apache Camel installation, as prompted.

    NB: At time of writing, you must use apache-camel-1.3.0.

    The project wizard currently creates a jbi2camel.xsd and jbi2camel.wsdl for you in the default package so that you can invoke your Camel 'routes' from the JBI runtime over the NMR. The default jbi2camel.xsd created defines a single message of type xsd:anyType and therefore will accept any message type so we can reuse this for the camel2jbi case below. In practice, you will likely want to strongly type your message definitions in line with what is expected by the filter.

  7. Create WSDL contract for camel2jbi case
  8. The default artefacts created are for the jbi2camel case. In this scenario, we need to send messages to multiple JBI endpoints and manually need to generate artefacts for camel2jbi. It is likely that this will be generated for you in future versions of the Camel SE NetBeans plug-in.

    In the meantime, here's a camel2jbi.wsdl I prepared earlier (!) which defines the abstract endpoints required for the IM and Archive service. Alternatively, you can create these yourself in NetBeans by following these steps:

    • Create a New-> WSDL Document called camel2jbi.wsdl in the default package.

    • In the abstract configuration, select a one-way operation and assign the input part (in this case called 'body') to be of type AnyMessage from the existing jb2camel.xsd. There is no need to import this in the previous step since it is available in the same location in the file system. Again, you could create your own XSD if you want to use strongly-typed messages.

    • In the concrete definition, leave this as a SOAP binding type for now and use Document Literal. Define the service name as CamelContentFilterJBIModule_service and define just one of your service port endpoints e.g im_endpoint.

    • Now remove the SOAP binding, operation, operation, body and address as highlighted below. We don't need them in there since although we need a binding and service name to configure the Camel SE correctly in jbi.xml, it's fine to leave the binding and service abstract and use the CASA editor to wire them up later.

    • Finally, add a new service port for the archive_endpoint as shown. We need to do this manually again since the New WSDL Document wizard assumes a single service endpoint.

      I'm sure that future updates to Camel SE will render this entire step obsolete and if this all seems a little long-winded, you won't need to repeat these entire steps every time since you can reuse the camel2jbi.wsdl as necessary.

  9. Modify the service unit descriptor
  10. Firstly, make a note of the following values specified in the camel2jbi.wsdl you created (or imported) as highlighted below:

    namespace = http://j2ee.netbeans.org/wsdl/camel2jbi
    interface-name (or port type) = camel2jbiPortType
    service-name = CamelContentFilterJBIModule_service
    endpoint_name (or service port) = im_endpoint and archive_endpoint
    

    You will need to modify the jbi.xml created for you in the META-INF folder of your project. Initially, this only provides a service so you will need to modify it to add consume the services defined above by adding two consumes elements and import the namespace defined above from camel2jbi.wsdl. Call it xmlns:out, for example. When finished, your jbi.xml should look similar to the following (text in bold indicates the insertions you need to make):

    <?xml version='1.0' encoding="UTF-8" standalone="yes" ?>
    <jbi version="1.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://java.sun.com/xml/ns/jbi" 
         xmlns:out="http://j2ee.netbeans.org/wsdl/camel2jbi"
         xmlns:app="http://openesb.org/camel2jbi/CamelContentFilterJBIModule"
    >
    
     <services binding-component="false">
            <provides service-name="app:CamelContentFilterJBIModule_service" 
                      interface-name="app:CamelContentFilterJBIModule_interface" 
                      endpoint-name="jbi2camel_endpoint">
            </provides>
    
            <consumes interface-name="out:camel2jbiPortType"
                      service-name="out:CamelContentFilterJBIModule_service"
                      endpoint-name="im_endpoint">
            </consumes>
            <consumes interface-name="out:camel2jbiPortType"
                      service-name="out:CamelContentFilterJBIModule_service"
                      endpoint-name="archive_endpoint">
            </consumes>
    
        </services>
    </jbi>
    

  11. Define Camel routing rules
  12. We can now define our Camel routing rules in AppRouteBuilder.java. It is possible to use Ruby expressions within Camel but not a procedure (to my knowledge). We could use JRuby directly here but for this example we'll use Java for the filter logic. Replace the configure() method with the following:

    public void configure() {
    
        // Use this route when receiving messages from jbi endpoint
        // Note the jbi uri format is "jbi:<service_namespace>/<service_name>/<endpoint_name>"
    
        // STEP 1: Define the endpoints - these should match the ports in jbi2camel.wsdl and camel2jbi.wsdl
    String cnn = "jbi:http://openesb.org/jbi2camel/CamelContentFilterJBIModule/ CamelContentFilterJBIModule_service/jbi2camel_endpoint"; String im = "jbi:http://j2ee.netbeans.org/wsdl/camel2jbi/ CamelContentFilterJBIModule_service/im_endpoint"; String archive = "jbi:http://j2ee.netbeans.org/wsdl/camel2jbi/ CamelContentFilterJBIModule_service/archive_endpoint"; // STEP 2: Define the filter required in XPath - modify as appropriate String filter = "contains(//EntryList/Entry/description,'buffet')"; // STEP 3: Apply the DSL logic itself from(cnn). choice(). when(xpath(filter)).multicast().to(im, archive); }

  13. Create and configure composite application
  14. Now that our Camel JBI module is built, we need to create a new composite application and add the CamelContentFilterJBIModule into it by dragging and dropping it onto the service assembly canvas. Once this is done, build the service assembly and drag and drop the desired WSDL bindings onto the WSDL ports section. In this scenario, we require RSS, File and XMPP but you can use any protocol bindings (e.g. JMS instead of XMPP) since we have been using abstract service definitions.

    Next connect the endpoints from our Camel service unit to the required port types. Once this is done modify the properties for each port by clicking each of the indicated icons below and setting the required fields for each of the components.

    For the RSS feed, set the URL to http://rss.cnn.com/rss/money_latest.rss

    You may also need to modify other WSDL properties for the binding component which can be found in the WSDL file itself generated in the Process Files folder of your composite application. For example, for the RSS feed itself, modify the binding properties as shown below to only retrieve a single feed article by modifying the archive date.

    An alternative extension to the above is to use the Splitter enterprise integration pattern in Camel to retrieve multiple articles and split them out in your mediation DSL logic. It is also possible to modify these properties directly in the source WSDL by switching to the source view. Search for fileName in the WSDL and rename test.xml to archive.xml. For completion, set the XMPP settings of your IM service or use an alternative binding component by re-wiring in CASA if this is easier. To use XMPP, I recommend you install the Openfire (aka Wildfire) server and Spark client from Ignite Realtime. There is a demo of how to do this on the XMPP BC page on java.net. You can read about this more on http://gallemore.blogspot.com/2007/06/xmpp-to-rss-demonstration.html.

  15. Deploy and test composite application
  16. Build and deploy the composite application!

    You're done. Upon deployment, the RSS feed will be polled and if the XPath filter matches, a file named archive.xml will be created with the contents and your IM client (or JMS queue) will receive the message. Have fun with it! If you are having issues with the filter, you can modify the DSL to read:
     ...
    from(cnn).multicast().to(im, archive);
     ...
    

For your convenience, you can download a zip containing the completed CamelContentFilter project and composite application here. I recommend rewiring the XMPP BC to use another component (e.g. another file or JMS) if you don't want to install the XMPP server and client.

Scenario extensions

There's more to Camel than just DSL. Here are some examples:

  1. Using POJOs and Spring
  2. Do you want to also route the message to a POJO for handling? It's easy to do with the Camel SE. Firstly, create Logger.java in the cameljbimodule package:
    
    package cameljbimodule;
    
    public class Logger {
    
        public void log(String body) {
            System.out.println("+++LP: Received a message with contents: " + body);
        }
    }
    
    

    Now modify the camel-context.xml in META-INF/spring to include a new bean entry:

    ...
    <bean id="logger" class="cameljbimodule.Logger" />
    ...

    You can now access that bean from your AppRouteBuilder class as shown in the next case.

  3. Applying interceptors using POJOs
  4. You can also add interceptors across your defined routes. For instance, to define an interceptor which will override all of your routes and simply log anything that comes in on the RSS feed using the above Logger, add the following code to your configure() method:

    ...
    intercept().to("log:foo");
    ...

    In comparison, Fuji allows you to build and deploy interceptors that work across composite applications as OSGi bundles and without needing to modify other deployments. They can be turned on or off simply by starting or stopping the bundle respectively. See the interceptor screencast for more details.


Summary: Camel SE and Open ESB v2

If you can't wait for GlassFish v3 and Open ESB v3, don't want to use BPEL or write your own EIP framework, it is worth considering using the Camel SE with Open ESB v2. In summary:
  • Camel supports most of the known EIPs. See the list here.
  • The Camel SE provides a simple way to expose Open ESB's JBI BCs / SEs to existing Camel users.
  • The Camel SE enables Spring and interceptor support for Open ESB v2.
  • Camel supports multiple DSL approaches but does not provide a GUI for service composition. On the other hand, Fuji's IFL can be composed in a browser.
  • Unlike Fuji, using Camel SE with Open ESB currently requires some knowledge of WSDL. The tooling may advance to hide WSDL further but it is important not to underestimate the role that WSDL contracts play in a JBI service oriented architecture.
  • In my experience, the required Java DSL in Camel is not always possible to construct. See my Nabble post on the confusion caused by using multicast with choice.
  • Message routing using splitters, filters and aggregators may be quick to implement using XPath and Camel. However, if you require more complex mapping capability, it may be easier and quicker to use the BPEL editor directly. Alternatively, you can use the XSLT SE and/or BPEL SE in combination with the Camel SE since it's just another JBI endpoint.
So don't get the hump, try Camel and Camel SE with Open ESB today. If you've got mountains to climb, I suggest you also take a look at Project Fuji!
Comments:

Hi Louis,
This is a very interesting article. We are developing an application that uses the Sun stack of Software e.g. Glassfish, Netbeans, OpenJMS etc. We need to develop a high speed message routing hub (about 2000+ msg/sec) to route messages between JMS queues. The routing rules need to be flexible and either described using an IDE or a simple language by system integrators. Would BPEL be fast enough and do you have any thoughts on implementing it to be fast, or would something like Apache Camel be a better choice?
What are your thoughs or ideas?
Thanks inadvance for your thoughts.
Cedric

Posted by Cedric Franz on September 01, 2008 at 08:26 AM BST #

Hi Cedric. Can you supply more details? Is 2000 msg/sec a combined throughput across all the queues in the system? Will message persistence / guaranteed delivery / HA be a requirement of the system. What is the average message size you will be sending and can the ESB run in parallel in order to process messages on inbound queues without the need to maintain outbound message order?

Also, how complex will the routing rules be? Will they operate on content based in the messages themselves or will this be 'duplicated' in JMS message headers? XML parsing and transformation logic will of course be a bottleneck. Is the plan to use the ESB as a simple router here or will it need to transform messages before they go to the different queues?

It may also be an idea to continue this discussion on users@open-esb.dev.java.net to get some feedback from the community and also some of my colleagues who are working on OpenESB benchmarking.

Thanks,
Louis

Posted by Louis Polycarpou on September 03, 2008 at 07:17 AM BST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Louis Polycarpou

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today