Antony Reynolds' Blog

Student SOA Survey

2008-07-22T13:43:42Z

Support for SOA Survey

I just received an EMail from a student, Galin Monev, at Cork ...

I am currently a student of MBS “Information Systems for Business
Performance” at University College Cork, Ireland. As a final part of the
programme I have to carry out an industry-based project. Therefore, I am
writing on the behalf of my group regarding our thesis project. We are
investigating how SOA can influence the IT capability of a firm and to what
extend this strategy can become a major initiative for changing the
underlined business approach of an organization.
We came across your blog and we are wondering if you could help us by
filling out a survey. We would really appreciate if you could forward our
survey to friends (through your blog) and colleges who you think can
contribute positively to this survey.

The survey is in two parts and will take no longer than 5-10 minutes. It is
located at:

Part 1

Part 2

We would also appreciate any additional comments you may have or additional
assistance you would like to provide.

If you have any questions or comments, please contact me at galin.monev@gmail.com. I am hoping for your kind consideration and
participation.

So if you get chance why not support him, I will ask him to share his findings later.

Fusion Middleware Roadmap

2008-07-01T17:53:53Z

I spent last week in Redwood Shores listening to product development outline their plans and directions for Fusion Middleware in the light of the BEA acquisition. Today we made those plans public through a webcast hosted by Charles Phillips and Thomas Kurian.

I think some of the former BEA folks were a little surprised to see WebLogic Server chosen as the Oracle JEE platform over OC4J, but it makes a lot of sense – which product has the larger deployed base, which has the greater developer mindshare, which did we just spend billions of dollars acquiring. I was delighted that Thomas and Steve Harris (our VP for Java products) were able to make such a clear decision so early on, as it gives the market a great deal of clarity.

Similarly I think the decision to promote the AquaLogic Server Bus into the Oracle SOA Suite as the preferred ESB is again the right decision. Personally this is causing me a lot of problems because I am in the midst of writing a book about the SOA Suite and now I have to revise it to use the Oracle Service Bus (nee ALSB) rather than the old Oracle ESB :-( But otherwise it is a good thing.

Over the last few months as I have looked into the BEA product set I have been surprised how complementary much of the technology actually is. For example one area where Oracle has been weak in product is around the governance space. This area will receive a big boost from the Oracle Enterprise Repository.

The area of Business Process Management is an interesting one, because in the past we have pushed BPEL – which is really a service orchestration engine – as a BPM tool. The addition of BPEL to ALBPM in the BPM Suite will strengthen the ALBPM story and at the same time continue to allow BPEL to be used as a service orchestration engine in the SOA Suite. In the longer term the plan to converge the two products into a single run time will be worth watching.

I think the most surprising area to me was in the area of transaction processing. Certainly there are strong synergies between JRockit Real Time and Coherence – boosted by the WebLogic Application Grid offering. But the surprising bit was the emphasis that Thomas Kurian made on Tuxedo. It seems as though the Tuxedo guys are being rehabilitated after years in the wilderness at BEA since the WebLogic acquisition. I was amazed at the increase in connectivity and functionality that has occurred in Tux since I last came into contact with it some ten years ago.

So all told, I am happy about the stated product directions. They all seem to drive towards Thomas’ goal of a single, complete and integrated middleware suite. The next couple of years may be a bit bumpy as we smooth out the differences between the two product sets, but even today I think we have a cracking offering that is a world beater – not that I am biased of course!

Throttling Files

2008-06-14T05:06:45Z

Throttling Files

I'm sure everyone has been tempted to grab a file by the throat and squeeze it until it departs for that great filing cabinet in the sky, but that is something to discuss quietly with your psychiatrist. In this entry I would like to investigate how to limit the concurrency of file handling procedures.

A Concurrency Problem

By default the interaction with the file adapter is a one way interaction. The file adapter posts a message into the queue for a process and then carries on about its business. This is good because it allows multiple files, or multiple batches from a single file to be processed concurrently. However if there are a large number of concurrent processes started then this can cause performance problems as the process manager starts to worry about which process to execute when. One sympton of this is undelivered messages visible on the BPEL console.

This is bad because it means we are feeding BPEL process manager faster than it can consume, and so it gags a little and throughput is reduced as it gags.

Stopping Force Feeding

So how do we tell the file adapter that BPELs mouth is full and please wait until I have eaten this bit. Well the answer is in the observation that interactions with the file adapter are usually one way. They do not need to be! It is possible to modify the WSDL generated by the file adapter wizard to support a two way interaction. This causes the file adapter thread to wait until the reply before sending another message. For the reasons why changing from a one-way to a two-way interaction causes this behaviour look at my earlier entry on BPEL threading. The answer is left as an exercise to the reader.

Changing the File Adapter WSDL

To change the one-way interaction to a two-way interaction we do the following in the generated WSDL file:

Make sure that the definitions root element has a namespace reference to XML schema by adding the following

xmlns:xsd="http://www.w3.org/2001/XMLSchema"

Add a dummy message that will be used in a reply.

Add a reply to the read operation

Add a reply to the binding

This gives a file like the one below

     name="ReadPhoneBookFileSvc"
     targetNamespace="http://xmlns.oracle.com/pcbpel/adapter/file/ReadPhoneBookFileSvc/"
     ...
     xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    >
   ...















                ...
          OpaqueSchema="false" >







     ...

Modifying the BPEL Process

Having modified the WSDL we must modify the BPEL to now do a reply. The reply will release the file handling thread in the adapter to do further work. The modified process is available as project BigBatchProcess1.2 in the FileManipulation.zip file. Details of using this process can be found in an earlier entry on batch processing of files. Note that there is an additional test file included in the src directory - phonebook2.csv - that has 5000 records in it to give a larger test case.

Other Scenarios

The above scenario is not the only one where we may wish to use a two-way interaction with the file adapter. The following additional scenarios may also occur.

Reading a File Sequentially

We may need to process a file in strict record order yet still want to have it batched because it is a large file. In this case the problem is that we may have multiple batches submitted concurrently and there is no guarantee of the order that they will be processed in. In this case just making the interaction two-way is not enough because there may be more than one file processing thread.

Controlling the Number of File Processing Threads

The number of threads used to process files in the file adapter is controlled by a setting in the file $ORACLE_HOME/bpel/system/services/config/pc.properties. To alter the number of threads change the following property:

oracle.tip.adapter.file.numProcessorThreads=1

Note that this is a global setting and so may impact other file activation agents that have been configured in the system. The net result of this is to limit file processing to a single thread. Hence by using a single thread and a two-way interaction we can single thread the file processing and so guarantee processing of records in order in a file.

Processing Files in Date Order

Another scenario where these techniques are applied is when there is a requirement to process files in order of last modification date, for example when later files may contain reversing transactions for earlier files. This can currently only be achieved in 10.1.3.1 via a patch but it should also be available at some point in 10.1.3.3 and later releases.

Final Thoughts

The use of two-way interactions and limiting threads processing files is very powerful but it needs to be considered in the context of other file interactions in the system. One way to limit the impact is to use singleton processes to receive the processing requests, enabling messages to be queued in order, releasing the file processing threads for other processes to use whilst still ensuring correct ordering in the required processes.
Hope this helped.

More on Batch Processing in BPEL

2008-06-11T22:23:32Z

More on Batch Processing in BPEL

I was asked a follow up question recently on my entry about batch processing in BPEL. The individual had implemented a BPEL process similar to the one shown below.

A Use Case

The process needs to refresh the corporate phone book. It does this by reading a file with the new phone details, deleting the old phone details from a database and then inserting the new phone details into the database.
A file is read into the process via ReceivePhoneBook activity. The ClearPhoneBook activity uses custom SQL to truncate the phonebook table. Finally after transforming the file format to database format the new phone details are inserted into the table using the InsertIntoPhoneBookActivity to insert multiple records into the table in a single call.

A Fly in the Ointment

This process works fine as long as all the records are received in a single message. If they are received in multiple messages then not all the records will be written as multiple processes will receive part of the input file, but all the processes will truncate the table, resulting in the possibility of some records being inserted by one process and deleted by another process that starts a little later. This is illustrated in the diagram below which shows three processes each processing a portion of the file.

Note that records 1-10 are written to the database at the same time as the table is truncated by the process receiving records 11-20, leaving the possibility that the write will complete only to be overridden by the truncate. If records 1-10 are not overwritten by the second process they will definitely be overwritten by the third process which only starts truncating the table after records 1-10 have been inserted into the table.

A Solution

What we need to do is to separate the deletion of the existing data from the insertion of the new data. We can do this by moving the truncation of the table (deleting the existing data) into a separate BPEL process. Such a process is shown below.

For this to work we need to know when a new file starts to be loaded so that this process can be invoked before any record processing is performed. Now there are lots of complicated ways that we could do this with singleton processes to maintain state and complex logic to make sure it all works. Or we could use a newish feature of the BPEL process manager (introduced in 10.1.3.3 I believe) to get it to invoke our process.

Batch Manager

In the previous discussion I ignored the partner link that initiates the record deletion process. This partner link implements the Batch Manager interface as specified in $ORACLE_HOME/bpel/system/xmllib/jca/BatchManager.wsdl. To create the above process first create a new empty BPEL process and then in the services stream right click and select new partner link. Click on the icon to browse for files from the local file system and select the BatchManager.wsdl file and choose to implement the BatchManagerInterfaceRole (i.e. choose this as "My Role" ). This is the interface used by BPEL Process Manager to notify a process that a file has started to be read. There are several methods available to receive different notifications

onBatchReadStart - tells when a file is started to be read
onBatchReadComplete - tells when a file has finished being read
onBatchReadFailure - tells when a record or records cannot be processed by the adapter framework

For this scenario we are interested in onBatchReadStart. Receiving this notification we know to truncate the table ready to receive the records.

Who to Tell?

How does the BPEL Process Manager know to call the notification process? The answer is that it is configured as an activation agent property in the bpel.xml file of the process receiving the records from the file adapter. To set up the notification it is necessary to add the following property tag to the bpel.xml at XPath location BPELSuitcase/BPELProcess/activationAgents/activationAgent :

bpel://default|FileNotificationProcess

Note that default is the name of your domain and FileNotificationProcess is the name of your process. Adding this property will cause the BatchManager interface on the given process to be invoked when a file is read.

Final Steps

With notification configured we now need to modify our BPEL process to not truncate the table because this is being done via a separate process. We also need to add a delay to the process to avoid race conditions that could cause records to be inserted into the database before the database has been truncated. The modified process is shown below, complete with a 30 second delay to avoid problems with multiple processes being invoked at the same time.

A Worked Example

I have created a sample to let you explore how all this works. To set it up do the following.

Download the project files in FileManipulation.zip.
Unzip FileManipulation.zip - this will create a FileManipulation directory with 3 sub-directories and a JDev 10.1.3.3 workspace
Open the workspace FileManipulation.jws in JDev 10.1.3.3
Create the following directories or modify the file adapter partner links in BigBatchProcess1.0 and BigBatchProcess1.1

C:FilesInbound
C:FilesOutbound

Create a test user in the database by running the script in FileManpulation/BigBatchProcess1.0/src/CreateUser.sql as a system user
Create a database connection in JDev called TestDS to connect to user Test (password test) in the database. You may need to rerun the adapter wizard if you are not running XE database on port 1521
As user test run the script in FileManpulation/BigBatchProcess1.0/database/CreateTable.sql to create the phonebook table.
Deploy the BigBatchProcess1.0 to the BPEL process manager.
Test it by copying file FileManpulation/BigBatchProcess1.0/src/PhoneBook1.csv to C:FilesInbound.
Verify the number of records stored by executing command in database "select count(*) from phonebook. There are 1000 records in the source file, you will probably receive a count less than 1000 due to race conditions around truncating the table and inserting records into it. This is the problem we are trying to avoid.
Deploy the BigBatchProcess1.1 to the BPEL process manager as version 1.1.
Deploy the FileNotificationProcess1.0 to the BPEL process manager.
Test it by copying file FileManpulation/BigBatchProcess1.0/src/PhoneBook1.csv to C:FilesInbound.
Verify the number of records stored by executing command in database "select count(*) from phonebook. There are 1000 records in the source file, you should now receive a count of 1000 in the database table, indicating thqat we have solved the problem.
Pat yourself on the back and have a nice drink.

Hope that some of you find the above useful.

Register of Interest

2008-05-17T06:06:06Z

Registering Interest

I have been involved in some interesting discussions recently on the use of a service registry. Oracle SOA Suite ships with a limited use license for a service registry and there is the option to upgrade it to a full use license and so the question comes up, Antony why would we want to use a registry and can we use it for all our service lookups.
Lets start by looking at what a registry is.

Glorified Yellow Pages

Uncharitably a registry may be described as a glorified yellow pages. It allows artifacts such as XML schema and service WSDLs to be stored in a searchable, categorized archive. Artifacts are stored under different categories and may have keywords associated with them to assist in searching. So at the end of the day the registry is just a repository of meta-data about artifacts. In the same way a car is just a large amount of beaten metal with a power unit that drives wheels. Calling it a repository of meta-data does not actually explain what it does or how it may be used.

Registry use Cases

Let me suggest a few registry use cases

Design Time Service Discovery
A registry can be used to catalogue existing services and associated artifacts. This encourages re-use by making it easier to discover existing services. The ability to promote items between registries also makes it possible to put in place a service approval process that vets new services before making them available to developers. This again promotes re-use and generalisation of existing services.
Run Time Service Discovery
A registry can also be used at runtime to provide the physical endpoint for a service. This makes it easy to change the physical provider of a particular service. This also simplifies migration of services between development test and production environments as outlined in the next use case.
Service Migration Mechanism
The use of multiple registries provided a managed path for services to be promoted between environments, either from a development perspective or from a runtime perspective.

Design Time Considerations

Use of a registry at design time is generally a good idea but it does a require a certain amount of discipline in its use otherwise it becomes yet another dumping ground for all design decisions, good and bad! However individual develpoment team can maintain their own local registry that has a well defined promotion process to a central registry, allowing teams to work on their development services which may later be migrated to a corporate registry. In my experience most customers are not making use of a registry even in their design time environments and this is probably the best place to start using a registry.

Run Time Considerations

Using a registry in a runtime environment promises a greater degree of de-coupling. However a similar amount of decoupling may be achieved through the use of an ESB alone. If a registry is used to look up an ESB endpoint then there is a potential cost to be paid in terms of an additional lookup. If an ESB endpoint is not looked up then there is the risk of coupling the data formats of unrelated services together, losing the use of an ESB to provide message transformation to/from canonical form. Some informal tests I ran indicated that the additional overhead of a registry lookup does not add much to the service invocation, but in high volume environments it may be the straw that breaks the camels back.
A good policy may be to begin using a service registry in design time, later trying it out in non-high volume environments. Using an ESB can make this migration to registry use easier, if less pure by having the ESB perform the registry lookup.

Using a Registry with Oracle BPEL PM

The current production release of Oracle BPEL PM has built in support for use of a registry. Al that is required to make a service lookup occur through a registry is to perform the following.

In the BPEL Console, go to the "Manage BPEL Domain" and set the following properties

uddiLocation - the inquiry address of the registry
uddiUsername - if it is a secured registry set it to a username for performing lookups, if not set it to urn:unknown
uddiPassword - if it is a secured registry set it to a password for performing lookups, if not set it to urn:unknown

In a BPEL process for each service endpoint (partner link) that you want to go through a UDDI lookup

Add a property "registryServiceKey" to the partner link with the "Entity Key" value assigned in the UDDI repository

For more on potential performance impacts of registry lookup see Chintan Shahs blog entry. Note that I haven't seen such a bad degradation as he reports, but I could believe it for a complex WSDL and high system load, meaning unresponsive threading, particularly in a Windows environment.

An Example of Dynamic Lookup

To give a feel for dynamic lookup I have uploaded 3 JDeveloper Projects in the this file.

JavaWS includes a simple Java web service (GreetingWS) that can be registered in a service registry. When registering it in the registry you will need to note the service key and copy it into the partner link property in Dynamic BPEL.
DynamicBPEL is a BPEL process that looks up the web service (GreetingWS) and invokes it in three different ways

Static, it uses the endpoint defined in a WSDL file
Dynamic, it sets the endpoint explicitly (passed in as a parameter to the process), showing how you can be very dynamic in the endpoint you invoke.
UDDI, it uses the UDDI repository to lookup the endpoint, showing how unintrusive it is on the rest of the process.

TestDynamic is a BPEL process that you invoke to test the DynamicBPEL process. It iterates over all three methods and calculates how long each method takes to complete the given number of iterations. The first invocation is not counted to allow the system to "warm up" for each call.

More on Time!

2008-05-13T00:51:41Z

More on Time!

Seems that dateTime is one of the most frustrating types in XML. I just had a colleague ask how to convert a string to an XML dateTime type. There are a rich range of functions to convert dateTimes to strings, but a paucity to do the reverse. So I present here one approach to this. I use XSLT variables to help in the parsing. Here is the XSLT fragment that converts a US formatted date (MM/DD/YYYY) to an XML dateTime format (CCYY:MM:DDTHH:MM:SS). Note that in an XML dateTime all fields are mandatory, the only optional pieces are the decimal part of the seconds (CCYY:MM:DDTHH:MM:SS.SS) and the timezone portion (CCYY:MM:DDTHH:MM:SSZ or CCYY:MM:DDTHH:MM:SS+02:00).
In the fragment below I take the input data from XPath expression /DateList/SingleDate/DateField. This has the date in format MM/DD/YYYY. I use this input to populate an element dateitem which is of dateTime type.
Lets go throught the XSLT a little at a time.

Obviously the parsing could be made to handle time formats and timezones as well.
Hoping that this helps someone struggling with parsing text strings into dates.

I have created a simple ESB project that takes data from a file in US date format and loads it into two columns in a database table, one formatted as a string the other as a SQL Date. When using the Database adapter the SQL Date format gets converted to an XML dateTime. The project is available for download here.

BEA Completes

2008-04-29T22:25:35Z

Today the EU gave approval for the Oracle acquisition of BEA and so completed the acquisition.
Now that the bean counters have had their fun it is the engineers turn. Although individual products will continue to be supported it will be exciting to see a convergent road map for Fusion Middleware, WebLogic and AquaLogic. Both companies invested heavily in standards based computing and so there are lots of opportunities for convergence and synergies between the product stacks.
The acquisition of BEA also gives Oracle access to a whole new set of contacts within their customer base. Although Oracle and BEA have a large overlap in their customer base they tend to talk to very different parts of the business. The combination of both companies will open up new opportunities for existing Oracle product lines as well as for the newly acquired BEA product lines.
The next few months will be very interesting times.

Finding the End(point)

2008-04-29T01:42:17Z

Finding the End(point)

or The Difference Between Deployed and Undeployed BPEL WSDLs

A colleague just complained to me about his inability to create an ADF business object from a BPEL process. He explained that when he pointed the JDeveloper wizard at the BPEL WSDL is gave an unknown endpoint error. When I asked if he had pointed it at the file from the JDeveloper BPEL project rather than the WSDL file on the BPEL server he confirmed that this was indeed the case. Pointing at the WSDL file on the server solved the problem. Why is quite instructive.

A Tale of Two WSDLs

When you build a new BPEL process, you generally end up with a new WSDL file that defines the inputs and outputs of the BPEL process. What this WSDL file is lacking however is any endpoint information. Endpoint information details the logical network name and port number of the server on which the process is running, as well as the path to the process. This data cannot be filled in until the process is deployed to a specific domain (part of the path of the request) and server (hostname and port number part of the request). Hence when retrieving the WSDL from the JDeveloper file, you have no location where the process is available and it is this lack of data that the ADF wizard complains about. When you retrieve the WSDL from the BPEL server then it includes the location information of the process and hence the ADF wizard is happier.
Obvious when you think about it, but it is a problem that plagued many people in the past so I thought I would share it with you.

Patterns and Methods

2008-04-25T05:19:27Z

Patterns & Methods

Thought I would share some thoughts on Oracle SOA Methodologies and SOA patterns. These are by no means complete, I just wanted to put down some ways in which SOA Suite can encourage good practise.

Methodologies

The Oracle SOA Success Methodology is a methodology with a small 'm'.
It is best viewed as a set of tools of techniques that can be applied
within the context of a more proscriptive methodology. That said it
works best with iterative methodologies.

The SOA Success Methodology is currently used with a restricted number
of Oracle customers prior to a full release.

Within the context of the SOA Success methodology is extensive
modelling of the business in terms of services. Oracle BPA Suite can
be used to drive this iterative modelling process.

BPA encourages a focus on the abstract modelling side of the SOA
success methodology. As the model becomes more reified then composite
services and processes may be made available to JDeveloper through a
shared repository.

Oracle is currently using BPA Suite to perform high level modelling in Fusion Applications development.

BPA Suite supports roundtripping to allow regeneration of processes
without loss of reifications applied to earlier versions, encouraging
an iterative cycle between business analyst and process developers.

SOA Patterns

Best source for SOA patterns in the Oracle space specifically is the BPEL cookbook
which covers a lot of common patterns and techniques. Both the latest
release of Oracle SOA Suite 10g, and 11g currently in beta, encourages
these best practises in a number of ways, including but not limited to
the following.

Rules abstraction for

True Business Rules - tight integration of rules and BPEL
encourage business process developers to abstract business logic into
the rules engine - this is also available in 10g
Routing Rules - tight integration of rules and Mediator encourage developers to abstract routing decisions into the rules engine

Service Abstraction

Encourages service level location abstraction by making it easy
to hide physical endpoint of service - in 11g all BPEL processes are
automatically packaged up in the same SCA assembly used to describe
service deployments
Encourages service level interface abstraction by encouraging
use of mediator to transform between canonical and physical
implementation formats

Process Abstraction

Definition of processes allows for lower level processes to be treated as services
Capturing of process flows in code allows for simulation and modelling based on real world data

Human workflow captures common best practise human interactions into re-usable templates

Process Migration

Champion/challenger processes - two versions of same process
can run in parallel and be compared to see which yields best results
based on concurrent versioning capabilities of BPEL process manager.
Managed process migration - ability to run multiple versions of
same process allows controlled introduction of new process versions
whilst existing instance continue to execute.

Service Discovery

Service promotion - use of a registry allows services to be
promoted from dev to test to production in a managed way, encouraging
best practise.
Service catalogue - use of UDDI allows such a catalogue to be
maintained to encourage re-use by simplifying service discovery process

There is lots more but this was a quick brain dump. Let me know which important ones I have missed.

Of Laundries and Lego

2008-04-11T01:05:36Z

The Laundry Model for SOA

A common analogy for SOA is to describe it as lego blocks. At one level that works reasonably well, but the same analogy holds for object orientation or structured programming, so there must be something more. The CBDI site suggested the analogy of laundry, and I have to confess I really like this model and have used it with several customers as well as internal people here at Oracle. Let me expand on the laundry model.

SOA is about Services

It might seem obvious but service orientation is about changing your thinking (orienting yourself) to think about IT systems as a collection of well defined collaborating components (services). A service has some sort of contract implying that the consumer of the service will provide X and in return will get Y. A laundry is a great example of this. A laundry defines an interface, I prefer to think of it as a basket, that says you give me a basket of dirty laundry and some money and I will give you a basket of clean laundry.

The Service Interface

The basket interface, defines that laundry is passed in a basket or a bag, and has various options (lets call them parameters) that allow the client to specify if they want stain removal on particular items, or if they want their laundry starched, or repaired or any of a number of different options. Note that these parameters specify what is to be delivered, not how it is to be done. In SOA we may use WSDL and XSDs to define the interfaces.

Composite Services

Our laundry service provider may decide to sub-contract part of the work to other companies, for example sending velvet clothing to a specialist velvet cleaning service that employs gnomes to pick out the dirt from the fabric. This use of other services makes our laundry a composite service, because it is built out of other services. The client of the laundry service is unable to tell that the service provider uses other services, hence there is no difference between composite services and atomic services. Often BPEL will be used to create composite services from existing services.

Virtualising the Service Interface

We may want to change our laundry provider, and as long as they provide the same service we don't really care who does it. We may change the provider based on cost, or service level. In any case we don't want to have to change the way we work with the laundry service just because it is a different provider. Many companies provide their employees with a laundry drop off service. The employee (client of the service) does not care who actually provides the service, he just drops off his dirty washing at the drop-off point and picks up the clean washing later. This virtualisation of the location of the service makes it easy for the service provider to be swapped out without changing the clients using the service.
In the SOA world we may use an ESB to virtualise the endpoint by providing a fixed address for the service. In addition to providing a fixed address we may also compensate for slight differences in the interface by using a mediator to map between the format the client wants to use and the format the new laundry service uses.

Personalising the Service

If we are a regular customer then the laundry may decide to offer us better rates or some other additional service. This type of customisation may need to be accessed from several different places, for example the laundry may automatically iron shirts for premium customers, this impacts both the process of processing the laundry (the process flow) and also the billing engine, which should not charge for certain services based on the value of the customer. Within a service oriented architecture a rules service will enable us to centralise common business rules, such as the free ironing service. Centralising business rules in a rules service allows them to be applied consistently across an organisation, and managed in a single location, making changing them much easier. In the SOA world a rules engine can provide this.

Finding a Laundry

Where to find a good laundry. One option is to look in yellow pages for a laundry convenient to yourself. The equivalent in the SOA world is to look it up in a service registry.

Protecting Your Smalls

An obvious concern is that you don't want your personal clothing stolen on the way to the laundry or tampered with in any way. You expect the transport mechanisms and the procedures at the laundry to be such that your privacy will be protected and your smalls kept private. You don't view this as part of the service interface but as a more fundamental attribute of the service.
In the SOA world we can use declarative security to confirm who is sending the message, and also to encrypt all or part of the message to prevent tampering with it. Within the Oracle stack this functionality can be provided by the Web Services Manager.

A Clean Solution

I hope the suggestions above show why I like the laundry model and why I feel it relates better to SOA than does the lego model. If you have other suggestions for extending the model please let me know. Similarly if you have better models then let me know as well.
In the meantime may your whites stay clean and starched.

Pairing Off in XSLT

2008-04-08T15:38:52Z

Pairing Off in XSLT

My friend Newton was just fighting to transform some XML that was structured in a awkward way. The machine generated XML he was working with basically paired off successive elements, for example to provide a credential prompt and then the following element contained the credential itself. It took a bit of fiddling to get the right XPath expression to perform the transformation. I thought it would be illuminating to share it with you and hopefully someone will find a better solution than the one we came up with.

The Source

The source XML looked like this

...

Note the pairing of the Attribute elements.

The Target

The target XML looked like this

The Problem

The challenge was after identifying a path with a SelectElement element how to select the following path that had a PasswordElement element.

The Answer is Axis

The trouble with XSLT is that I don't do enough of it to remember all the tools available. In this case it was very simple once I had remembered how axis work. Here is my solution

This solution iterates over all the SelectElement elements and then for each of those elements it picks out the following PasswordElement in the document. It does this by using the XPath expression "following::AttributeElement[1]/DisplayElement/PasswordElement/@attribute1".
The "following::" axis selects all nodes in the document that come after the given SelectElement element. This flattens the hierarchy enabling us to choose the next AttributeElement element by selecting the AtrributeElement element with index 1 (AttributeElement[1]). We can then traverse that part of the XML using the normal child axis. Simple when its done but when you rarely use axis other than the child axis it takes a while to remember how they work.

Reference

I find the two O'Reilly books on XSLT to be very helpful when struggling with XPath and XSLT. These are the Doug Tidwell XSLT book and the Sal Mangano XSLT Cookbook.

I hope this saves sombody some time.

Stringing Together XML

2008-03-07T18:16:48Z

Stringing Together XML

Someone just asked how to convert a string into XML. A common use case is to retrieve XML stored as a string in the database. Although you could argue that it would be better to store XML using the XML facilities of the Oracle database the fact is we still see lots of XML stored as VarChar2 or CLobs. So how can we read these into BPEL? Well in addition to explaining this I thought it was also a good excuse to explain the use of the append rule within an assign and also a use for the anyType.

Reading XML

Fortunately the developers foresaw the problem of converting strings to XML and have provided a handy function ora:parseEscapedXML which takes a string as a parameter and returns an XML document. The string can be either a full XML document or just a document fragment. Similarly it can have an explicit namespace or no namespace.
I have uploaded a simple BPEL process that takes an XML string as input and returns the DOM representation of that string.

Using an anyType

For the input I take a string type but for the output I have used an anyType. This allows me to legitimatly assign it any value. Note that although anyType is not supported by the graphical XPath builder in the XPath wizard it is supported by the editor, which will drill you down to the anyType.

Using Append instead of Copy

The other thing I have done in the assign is to use an append rule to add the parsed XML underneath the base output element. This allows us to see the entire result. If I had used a copy rule then the top level element of the parsed XML would have been thrown away and only its contents copied. Using the append rule causes the entire subtree selected by the XPath expression to be placed underneath the target element. Because the target is an anyType I don't have to worry about the XML conforming to any particular schema, I can dump any well formed XML in there.

Testing it Out

You may want to try it on the following input data to see how it works. Note that there is no difference in the way documents and document fragments are handled. the only requirement is that the XML be well formed.

Fragment Without a namespace

AntonyReynolds

Fragment With a namespace

AntonyReynolds

Document Without a namespace

AntonyReynolds

Document With a namespace

AntonyReynolds

Threading It All Together

2008-02-20T17:26:17Z

Threading It All Together

or Thread Usage by BPEL Processes

Based on comments I have received and questions I have been asked it seems that when it comes to thread usage by BPEL processes there are lot of confused people around. So I thought I would unravel a few mysteries about threading in BPEL. Understanding the threading model is important for BPEL because the thread model can affect scalability of processes and the BPEL engine itself.
How threads are allocated depends on the interaction pattern of the link activating the receive activity.

Interaction Patterns

The interaction pattern can be thought of as how the BPEL process is called. There are two types of interaction pattern, one-way and two-way or request-response as it is sometimes known.

One-Way Interaction

A one-way interaction pattern means that the process is invoked and is then left to run, possibly returning the results via another interaction, possibly not. We often think of this as asynchronous because the caller does not wait for the process to complete. Don't confuse the actual interaction pattern with the way the process works. For example many processes have a request reply process model that is implemented through two one-way interactions. The client calls the process and continues working. The process does its job and then calls the client to pass back the results. The process has a request-reply model but it works through two one-way interactions. A one-way interaction pattern is characterised by a process that has a receive activity but no corresponding reply activity. A request-reply process using one-way interaction patterns would be characterized by a receive activity with no corresponding reply but it would have a corresponding invoke.

Two-Way or Request-Response Interaction

A request-response interaction pattern means that the process is invoked and the caller then waits for the process to return a result before the caller continues its own processing. We often think of this as synchronous because the caller appears to get the result immediately. A request-response interaction pattern is characterised by a process that has a receive activity with a corresponding reply activity. Note that it is possible to have a process that combines an initial request-response interaction to return an initial result, such as a correlation token, with a later one-way interaction that returns the final result to the caller.
More information on what this looks like is available in the BPEL docs.

Instrumentation

To investigate thread usage in BPEL processes I wrote a small amount of Java code that is embedded into the process inside an exec activity. The code is shown below and stores the currently executing thread group and thread name into a variable called "CurrentThread".


      Thread t = Thread.currentThread();
      setVariableData("CurrentThread",
t.getThreadGroup().getName()+":"+t.getName());
    ]]>

This variable is then surfaced in the output of the process.

Simple Request-Response Thread Usage

My first test was to run a simple request-reply interaction pattern in a process. In this case the thread is taken from the inbound HTTP listener thread pool and a single thread is used to receive the request and process it. The thread group is "HTTPThreadGroup" and the actual thread I got was "AJPRequestHandler-RMICallHandler-56". The last digits vary. So in this case the BPEL server receives the request on the same thread that is used for all processing within the process. This process is available as ThreadExplorer1.zip.

Simple One-Way Thread Usage

My next test was to run a simple one-way interaction pattern, with another one-way onteraction to return the result. in this case the message is received on one thread and then passed onto a pool of worker threads to execute the BPEL process. The thread group is "main" and the actual thread I got was "WorkExecutorWorkerThread-120". The last digits vary. So in this case the BPEL server received the request and then queued it for subsequent execution. This allows for greater scalability by limiting thread usage within the BPEL server. This process is available as ThreadExplorer2.zip.

Combination of Request-Reply and One-Way Thread Usage

The obvious next question is what happens when I have both interaction patterns in the same process. Well that is what I did in my next test. I created a process that returns an immediate result (request-response interaction) and then later posts a further response as a one-way interaction. In this case the message is received on the listener thread and processed on that thread up to the reply. At that point the remainder of the work is passed off to a worker thread. This is exactly as one would expect from combining the two previous usages. This process is available as ThreadExplorer3.zip.

Effect of Flow on Threading

The next question is what happens in a flow activity. Surely this will cause multiple threads to be spawned. Well by default the answer is no. ThreadExplorer4 process uses a request-reply interaction and within the processing it has a 3-way flow statement. All statements in the flow are executed in the same thread. This is exactly the same as if we had used a request-response interaction with a while activity instead of a flow. This may be counter-intuitive but the reasoning behind this is as follows:

For scalability we want to limit thread usage
A flow is really intended to support waiting for multiple events such as responses from services with two one-way interactions and so we don't need to execute them in parallel, pseudo-parallel will do.
If one branch of a flow is waiting for an event the same thread can start executing the next branch of the flow, avoiding the need for a JVM thread context switch.

So there is little real need for a truly parallel flow statement. This process is available as ThreadExplorer4.zip.

More Detail of Effect on Flow on Threading

If we put a wait or receive into our flows then we find that the thread itself is not suspended, but is returned to the thread pool. If there are other legs of the flow capable of execution then the thread will be used to execute those before being returned to the worker pool. This same behaviour applies when a leg of a flow wakes up due to an event such as a timer (wait activity) expiring or a message arrive (receive activity). This is all shown in ThreadExplorer5 process which has a 3 way flow that uses the initial worker thread to put all three legs into a wait activity and then uses a different worker thread when they wake up to carry on their processing. This process is available as ThreadExplorer5.zip.

Creating True Parallelism in an Flow

One common use case for flow is to make multiple request-reply calls in parallel to reduce their latency. But as we saw in the previous section this doesn't work out of the box so how can it be achieved? If we add a property "nonBlockingInvoke" to the partner link we are calling and set it to "true" then BPEL will use a seperate thread to make the call. This is shown in ThreadExplorer6. Note that when you run this the initial request is being processed on the thread that received the message, a listener thread. When we do the invokes on a partner link with nonBlockingInvoke property set to true we see that those invokes are processed in a seperate thread. Because we are calling another BPEL process the server does some optimisation and uses the same thread for all three invokes. If the partner link were on a different server then 3 threads would have been used. Note that these threads are all worker threads, not the thread from the listener. Finally when the three flow legs join together a new worker thread is used to carry on processing. This process is available as ThreadExplorer6.zip.
You can read more about the nonBlockingInvoke in the BPEL documentation.

More on Thread Processing in BPEL

You can read more about thread processing in BPEL in the documentation which gives more reasoning for the way the cases above behave as they do. Hope this was useful to you.

Aggressive Cleanup

2008-01-21T18:20:07Z

A Problem of Aggressive Cleanup

Just been working on an internal proof of concept and came across some very irritating behaviour in BPEL Designer in JDeveloper 10.1.3.3. I deleted a Partner Link and accepted the suggestion of JDeveloper BPEL Designer to delete the associated WSDL file. Unfortunately it turns out that this deletes not only the partner link specific WSDL file but also a shared WSDL file (DBAdapterOutboundHeader.wsdl) that is used by all DB Adapter Partner Links in the project. Net result is that this breaks all the remaining DB Adapter based Partner Links in the project. Fortunately there is a simple solution. Copy the DBAdapterOutboundHeader.wsdl from any other project with a DB Adapter in it into the project/bpel directory and your problems will be solved.
For a view of some of the problems this may cause have a look at this blog.

BEA Joins the Team

2008-01-18T17:21:54Z

BEA Acquisition Thoughts

You may not be surprised to me hear me say this but I am really excited by Oracles recent acquisition of BEA, expected to complete in October of this year. If you haven't read about it yet then here is the official Oracle announcement and press release. On Wednesday Larry and Alfred took to the phone to explain the move to analysts (see transcript). Already the pundits are giving their views which are generally positive. Below are a few views I found on the net.

I think this is great news for a number of reasons.

Firstly the combination of BEA and Oracle will really give IBM something to worry about in the middleware space, the combined company and product stack will show superiority over IBM in almost every area of middleware.
Secondly the combination will provide a Java middleware powerhouse that will increasingly carry the fight to .Net, particularly in the user experience area.
Thirdly it removes the chance of SAP buying BEA (I know they are chalk and cheese in philosophy and management styles but it was still a worry).
Synergies will be easy to find. Both BEA and Oracle have a strong standards focus. It took Thomas Kurian to get Oracle Middleware firmly onto a standards based view of the world, but it is now there. BEA has always had a strong standards based approach in its WebLogic range. Because of these standards based appproaches key pieces of Fusion Middleware already run on the WebLogic, providing customers with lots of ways to rapidly combine the best of both Oracle and BEA middleware. It will also enable the engineering teams to share a lot of functionality across what will probably become convergent platforms.
Fusion Middleware has a "hot-pluggable" approach that allows components to be mixed and matched. The acquistion of BEA will give a new impetus to making sure that "hot-pluggable" is easy and provides a mechanism
From a purely selfish view it will let Oracle focus on its two main midleware competitors - Microsoft and IBM.

Of course we will have to wait for October 2008 for the deal to complete so we won't actually see any action on the ground until 2009, but then it will get interesting.

Disclaimer - these are my own views and not those of Oracle Corporation.