Setting the Context with Oracle Enterprise Manager

The last few weeks I have being trying to get my head around some of Oracles recent acquisitions in the operations management space and it struck me how they all talk about context but all mean something different by it.  Within Oracle these products come under the Enterprise Manager banner and specifically I have been looking at Oracle Real User Experience Insight (acquired from Moniforce), Oracle Application Diagnostics for Java and Oracle latest acquisition Clearapp.

What do they mean by context?

Each of these products talks about context but means different things.

Oracle RUEI means the goup of pages that make up a web session, that is context as expressed through a user session being held between web pages.  This allows it to monitor usage of web resources and provide information for example about which stage of a sales process customers give up on, providing useful insight into how look/buy or look/book ratios might be improved.  So in this case context is used to drive direct business information.

Oracle AD4J means the call stack and any related database sessions, so much of this is focused around stack traces, threads and related database sessions.  This allows it to monitor and identiofy code bottlenecks across the Java and database tiers in the context of the sequence of call made, providing context for a single request in a JVM and its related database activity.

ClearApp has yet another definition of context, in this case context is at the SOA level of relationships between components.  So context means which entry point drives traffic into which other components in the system.  This context provides information about how one component calls another in a SOA infrastructure, potentially across multiple JVMs and multiple invocations.

The same but different!

I just thought it was sort of intersting how the same word got used for different meanings.I

f I am analysing web site performance then I am interested in the context of calls from a single user perspective and what type of experience that user has of my web site.  This is the RUEI context.

If I am trying to monitor dependencies between SOA components and identify bottlenecks for particular SOA entry points then I am interested in the relations between components and which components are placing limits on the scalability or performance of my system.  This is the ClearApp context.

Finally if I have performance problems with my JEE code and its database usage then I am interested in the code path taken for particular servlets, jsps and ejbs.  I can use this to examine where my code is spending its time for a particular entry point and take action to improve its performance. This is the AD4J context.

At the moment these are all separate components in separate packages in Enterprise Manager but there are some obvious synergies between them and I can see operators wanting to take information gleaned from one and use it in another.  For example I detect a slow web page using RUEI.  I examine that web pages code usage in AD4J and discover that it is spending most of its time in a web service call.  I note that web service call and examine it using ClearApp where I discover that the problem is actually related to a BPEL process which I then ask my developers to tune to improve the web [age response time.

More SOA toys but this time more for operators than developers.

Getting Latest Version of SOA Suite

I keep a VM image with snapshots of various clean installs of SOA Suite.  When a new patch set or point release comes out I then apply it to the original vanilla install.  This keeps me with nice clean SOA Suite installs.  I thought people might be interested in what is the current latest software release level and how to get it / install it.

Current SOA Suite release level is SOA Suite 10.1.3.4 and there is a patchset available known as MLR#1.  Don’t confuse MLR#1 with earlier patchsets for 10.1.3.3 which got up into the teens.

SOA Suite 10.1.3.4 is installed as a patch to 10.1.3.1 except in the case of BAM where it is a patch to 10.1.3.3.  Note that the patch can also be applied to pretty much any patched versions of SOA Suite such as 10.1.3.3 MLR #10

Installing SOA Suite 10.1.3.4 MLR#1

To get a vanilla install of 10.1.3.4 MLR#1 we do the following:

1. Download SOA Suite 10.1.3.1 from OTN.  It is CD1 for your platform (Windows and Linux links).
2. Unzip/extract SOA Suite and install by running installer.  Note that if you are either not on Windows or wish to use an existing database then you need to have an existing database installed and ready to go.
3. Verify install worked by bringing up BPEL console, ESB console, App Server Console, etc.
4. Download SOA Suite 10.1.3.4 patch from OTN.  It is labeled 10.1.3.4 patch for your platform (Windows and Linux links).
5. Unzip and install by running installer.  Run database upgrade scripts as instructed.
6. Verify update worked by bringing up BPEL console, ESB console, App Server Console, etc.
7. Download MLR#1 patch from MetaLink.  This can only be done if you have a MetaLink account for SOA Suite.  If you are evaluating the software then you should be fine without MLR#1.  MLR#1 is patch number 7375086 and is a generic patch but is tagged as a Linux patch because MetaLink doesn’t have a generic platform.  The patch itself is valid on all 10.1.3.4 platforms, not just Linux.  Check the readme for install instructions.
8. Unzip and apply the patch.
9. Verify update worked by bringing up BPEL console, ESB console, App Server Console, etc.

Congratulations, you are now the proud owner of a SOA Suite 10.1.3.4 MLR#1 installation.

Installing BAM 10.1.3.4

To get a vanilla install of 10.1.3.4 BAM we do the following:

1. Download BAM 10.1.3.3 from OTN.  It is CD1 and is only available for Windows.
2. Unzip/extract BAM and install by running installer.  Note that if you will need an existing database.
3. Verify install worked by bringing up BAM console.
4. Download BAM 10.1.3.4 patch from OTN.  It is labeled 10.1.3.4 patch for Windows and is the same as the 10.1.3.4 SOA Suite patch so you can use the one yopu downloaded in step 4 in the previous section.
5. Unzip and install by running installer.  Run database upgrade scripts as instructed.
6. Verify update worked by bringing up BAM console.

There is currently no MLR patch for BAM 10.1.3.4.

Checking Versions

You can verify that you have the latest version of SOA Suite by going to the home page and you should see the version number as “Welcome to Oracle SOA Suite (10.1.3.4.0)” – note that patch levels are not shown here and must be interrogated by using “opatch lsInventory”.

To check what version of BAM you are running log onto any of the BAM tools, Active Viewer for example, and select the About box.  This will show the release as 10.1.3.3.0 but will have 10.1.3.4.0 mentioned in the Updates section.

Software as a Service for Technologists

With the Oracle cloud computing announcements at OpenWorld I thought it would be worth putting down a few thoughts about Software as a Service (SaaS), which is related to but not the same as Cloud Computing.  Software as a service is all about providing a useful function to the customer without the customer being concerned about it is delivered.  In that sense it is SOA taken taken to the next level.

For a good introduction to SaaS take a look at the Wikipedia page.  There is also an excellent article on MSDN by Frederick Chong and Gianpaolo Carraro that outlines a SaaS maturity model and indicates how SaaS can be used to bring the benefits of enterprise software to smaller businesses by making it available in a pricing model and at a price point that they can afford.  To track what is happening in the SaaS world there is an interesting blog on ebiz site.

Larry Ellison has an interesting position on SaaS that I think might be summarised as “nice model, lets wait and see how to make money at it”.  There is an interesting commentary on some of Larrys comments on ZDnet.

Pricing

Software as a service needs a pricing model based on usage.  This may be transaction charging or per user pricing. The important thing is that the pricing model has to be easily translated into the service the customer is receiving.  Consider an airline booking application, there are several models that may be used here.

• Number of Passengers Booked – essentially a booking fee, ties in very well to customer business model who will receive ticket value for each booking made which is then directly related to passengers booked.  However service provider may be worried about very high look to book ratios during fare wars and promotions forcing the provider to have additional capacity with no additional revenue, hence they may prefer the next model.
• Charges per View – each hit on the site is charged at a set amount.  This has the problem that some operations are more valuable and potentially more expensive to perform than others.  For example a booking operation may cost 100 times as much resources as a view operation.  Hence it may be that different operations are charged at different rates.
• Number of Passengers Booked Plus a Charge per View – essentially a combination of booking fee and an additional charge based on the number of views generated by potential purchasers of flights.  The additional charge may be zero up to a certain look/book ration and then there may be a sliding scale of charges beyond this ratio.

As can be seen from the example above, the pricing model, although tied to the service being provided may very rapidly become complex.

Hosting

Some people find it difficult to distinguish between hosting and software as a service.  A key feature of hosting is that clients rent hardware and floorspace to run applications to which the client has purchased the license directly.  This requires clients to be responsible for sizing and software architecture.  In contrast SaaS has clients draw down some computing service, such as CRM, and pay for it based on their usage of the resource.

In the hosting model the client must specify the capacity and takes the risk of the getting it wrong.  In the SaaS model the client is only interested in the service provided, not the capacity planning requirement.  Typically any capacity planning required should be related to the nature of the application, number of registered users for example rather than number of CPUs.

Architectures

A key feature of software as a service should be minimal footprint in the client organisation.  Generally services should be provided through the browser or across HTTP based web services for application access.  Some services may require tighter integration with software at the client site, and it may be that an appliance approach can be taken to simplify integration with other customer systems.

Because clients are no longer responsible for sizing the hardware needed to provided a service there may be a removal of risk from the client to the service provider.  As the service provider is presumably providing the same service to multiple customers they will be in a better position to calculate and provision appropriate resources to host the application.

Basic software as a service provides dedicated hardware for each customer.  This has the advantage of being simple to implement but is expensive in terms of hardware resources.  It also makes scaling the service difficult as the granularity of scaling is at the single customer level. Use of software virtualisation technologies can make this easier to manage but run into the problems identified in the next paragraph.

To overcome the problems of one machine per customer it is possible to have multiple software installations per machine, each installation being dedicated to a single customer.  This approach assumes that the software is capable of supporting multiple independent installations on the same machine, not all software can do this due to use of shared operating system configuration locations.  in addition to the inability of some software to provide this level of co-habitation, there are still inefficiencies in having multiple instances of the same software competing for the same processor, memory, disk and network resources.  Scaling the system is still problematic and complicated as if additional machines are required then the complexity of configuration increases.

The highest level of software as a service enables multiple customers to be hosted in the same software instance, known as multi-tenanting.  This allows a more efficient use of resources.  Scaling is also simpler in this environment, assuming the software can be scaled across multiple machines.  This level of co-habitation however requires customers to be convinced that their information is safe. In the section below on supporting technologies I will look at some of the features that can make multi-tenanted easier to achieve

Supporting Technologies

Operating System

The ability of modern operating systems to support hardware resource virtualization, such as Solaris makes the model of a dedicated hardware per customer more palatable.  Also playing a role in this space are software virtualisation technologies such as VMWare or Oracle Virtual Machines.

Database

At the database level then technologies each client company is provided with a different account that has access to shared application schemas.  By using technology such as Oracles Virtual Private Database we can allow developers to build software that will run in a single instance without the need for extensive filtering of individual queries, each customer will be automatically restricted to the data visible to their own user account.  In addition to different database accounts the data can be partitioned by customer, making backup and recovery possible on a per customer basis, potentially allowing different backup strategies for different customers.

Application Server

Ideally all customers would be using a single shared applications server cluster, taking advantage of load balancing and fail over within that cluster.  Security profiles within the applications server can be used to separate end users into groups by client, each client having a different set of resources available.  For example when queue are being used it would be necessary to have separate queues for each client to ensure that they did not interact.

SOA Services

At the SOA level tools such an ESB and process orchestration must provide support for identity propagation through the system.  At every level in the architecture it is important to be able to logically isolate one customers data from another customers.  Key to this is being able to propagate the client ID on all messages to ensure that they are correctly handled, for example routing may be different depending on the level of service a customer has contracted for.

Web 2.0 & AJAX

Web 2.0 technologies, particularly AJAX allow a much richer user experience through the browser, allowing more sophisticated interactions with software as a service than earlier static HTML implementations.

WS-* Standards

Exposing services that may be consumed by other applications is made much easier with the widespread adoption of webs service standards.  At the basic level this includes XML, SOAP and WSDL.  But the development of security standards such as SAML and messaging and transaction standards will allow for richer application to application interactions.

Adoption and exposure of these standards in SaaS services will allow hybrid models of SaaS providing some functionality but being tightly integrated with the rest of an enterprises IT infrastructure.

Summary

SaaS is not a case of taking existing software and tweaking it a bit to make it run in a SaaS environment.  To take full advantage of SaaS there needs to be some fairly fundamental thinking in the design about how it will be impacted by a multi-tenancy environment.  The actual coding practices may not be much different but there is a need for different testing régimes to handle the additional complication of multiple clients, each with multiple customers.

BPEL Deployment Framework

One of the coolest features in BPEL 10.1.3.4 is the deployment framework which makes the job of moving processes between dev, test and production environments much easier.  Modifying the bpel.xml file or altering descriptors through the console provide a degree of customisation for different environments, but it is all done on a single property at a time basis and requires a lot of work for each environment, especially when it is considered that individual WSDL files may also need to be updated. There is a better way in SOA Suite 10.1.3.4; the deployment framework.

The deployment framework combines the BPEL suitcase with a deployment plan that updates multiple files in the BPEL suitcase with the correct values for the deployment environment. Different deployment plans can be created and maintained for each deployment plan.

It is possible to generate a template BPEL deployment plan from a BPEL suitcase which can then be customised and used with the base BPEL suitcase at deployment time to update the various URLs and properties.

The steps to customise the BPEL Suitcase for each environment are as follows.

• Create a deployment template from the BPEL project or suitcase which will be used as the basis for the deployment plans.
• Create a deployment plan based on the template for each target environment.
• Attach the appropriate deployment plan to the BPEL suitcase or project prior to deploying in the target environment.
• Deploy the BPEL process into the target environment.

Creating a Deployment Plan Template

To create a deployment template we need to add the following commands to the build.xml ant file that is built by JDeveloper and found in the Resources folder of our BPEL project.

<target name="generate_plan_from_project">
  <generateplan planfile="${process.dir}/planfile.xml"
                verbose="true"
                overwrite="true"
                descfile="${process.dir}/bpel/bpel.xml"/>
</target>
<target name="generate_plan_from_suitcase">
  <generateplan planfile="${process.dir}/planfile.xml"
                verbose="true"
                overwrite="true"
                suitecase="${process.dir}/output/bpel_${BPELSuitcase.BPELProcess(id)}_${rev}.jar"/>
</target>

This creates two new ant targets, generate_plan_from_project and generate_plan_from_suitcase. These both create a template deployment plan, either directly from the project files, or from a generated suitcase. A BPEL suitcase is only generated when a BPEL project is deployed or when the project is “made”, so if using the option to generate from a suitcase it is necessary to ensure that the suitcase has previously been created. If the suitcase generated is a revision other than 1.0 then it is necessary to set the revision property in the build.properties file that is found in the Resources folder of the BPEL project in JDeveloper.

# Change below if deploying with process revision other than 1.0
rev = 1.1

The generateplan command in ant uses four attributes

• suitecase or descfile is the source information for the deployment plan.
• planfile is the location of the planfile template to be generated.
• overwrite will replace any existing planfile of the same name.
• verbose turns up the level of reporting.

A sample deployment plan template is shown below. Note the use of two elements:

• <replace> is used to replace the value of a property within a specific part of the bpel.xml
• <searchReplace> is used to <search> for a string in WSDL and XSD files and <replace> it with another string.

<?xml version="1.0" encoding="UTF-8"?>
<BPELDeploymentPlan xmlns="http://schemas.oracle.com/bpel/deployplan">
  <BPELProcess id="SimpleFileProcess">
    <configurations/>
    <partnerLinkBindings>
      <partnerLinkBinding name="ReadNewCustomerFileService">
        <property name="wsdlLocation">
          <replace>ReadNewCustomerFileService.wsdl</replace>
        </property>
      </partnerLinkBinding>
      <partnerLinkBinding name="WriteNewCustomerDBService">
        <property name="retryInterval">
          <replace>60</replace>
        </property>
        <property name="wsdlLocation">
          <replace>WriteNewCustomerDBService.wsdl</replace>
        </property>
      </partnerLinkBinding>
      <partnerLinkBinding name="CardValidatorService">
        <property name="wsdlLocation">
          <replace>CardValidatorService.wsdl</replace>
        </property>
        <property name="location">
          <replace>http://w2k3/chapter18/CardValidatorServiceSoapHttpPort</replace>
        </property>
      </partnerLinkBinding>
    </partnerLinkBindings>
  </BPELProcess>
  <wsdlAndSchema name="CardValidatorService.wsdl|DBAdapterOutboundHeader.wsdl|fileAdapterInboundHeader.wsdl|NewCustomerFile.xsd|ReadNewCustomerFileService.wsdl|WriteNewCustomerDBService.wsdl|WriteNewCustomerDBService_table.xsd">
    <searchReplace>
      <search/>
      <replace/>
    </searchReplace>
  </wsdlAndSchema>
</BPELDeploymentPlan>

Creating a Deployment Plan

Having created a template we can use this to create deployment plans for each specific environment. We do this by creating a copy of the deployment plan by selecting Save As from the file menu in JDeveloper and then editing the <search> and <searchReplace> tags to match our target environment.

We will search and replace all instances of our local development machine hostname, w2k3, with the name of our test server, testserver, across wsdl and xsd files. To do this we modify the search and replace elements as shown below.

<wsdlAndSchema name="*">
  <searchReplace>
    <search>w2k3</search>
    <replace>testserver</replace>
  </searchReplace>
</wsdlAndSchema>

This will cause the BPEL process manager to search all WSDL and schema files “*” in the suitcase at deployment time and replace the string “w2k3” with the string “testserver”. Note that it is possible to have multiple <searchReplace> elements.

Attaching a Deployment Plan to a BPEL Suitcase

Having created and saved a deployment plan specific for one or more specific environments we will want to deploy our process into an environment. Before doing this we must first attach the specific deployment plan to the BPEL suitcase. We do this using the following ant command.

<target name="attach_plan">
  <attachplan planfile="${planfile}" verbose="true"
               overwrite="true"
               suitecase="${process.dir}/output/bpel_${BPELSuitcase.BPELProcess(id)}_${rev}.jar"/>
</target>

This will create a file bpeldeployplan.xml in the BPEL suitcase. This is the deployment plan that will be used at deployment time by the BPEL process manager. Note that the name of the deployment plan to use is encoded as an Ant property planfile that must be set in the build.xml. Once attached the deployment plan will be executed when the BPEL suitcase is deployed.

Modifying Ant to Use Deployment Plan

In addition to adding the two tasks above to the build.xml file, it is possible to add the attachment of the plan file as part of the regular deploy process by modifying the dependencies of the process-deploy task by adding the attach_plan dependency after the compile dependency.

<target name="process-deploy"
          depends="validateTask, compile, attach_plan, deployProcess,
deployTaskForm, deployDecisionServices" />

Now when building and deploying with ant the deployment plan will be attached to the suitcase before the suitcase is deployed to the target environment. This allows us to provide a different deployment plan for each environment, which can then be deployed from the command line in a reproducible fashion.

Summary

The deployment plans allow a lot of automation in the move from development to production, and because they can be used with editable files and command lines they are easy to include in the version control systems and automated build environments.  Congratulations to the BPEL developers on a useful piece of functionality that appeals to those responsible for ensuring consistency of environments rather than developers.

Time Travel for Oracle Web Services Manager Administrators

I watched the Sarah Jane Adventures, a Doctor Who spinoff, with my family this week in which Sarah Jane has the opportunity to travel back in time to see her parents before they died.  Of course it all goes horribly wrong as usual and Sarah-Jane and her young assistants will have to save the world, again!  Reminds me of IT, you do something which seems like a good idea only to discover that it is a disaster.  Fortunately many of the tools we use have some sort of versioning that allows us to rollback the changes and restore the world to the way we want it.

Oracle Web Services Manager has just such a facility.  Every time a change is made to policies applied to a component in OWSM the previous set of policies is saved as a version, stored, ready and waiting to be restored if the change causes the fabric of space time to unravel.  When reviewing the policies applied to a component there is a View Versions label next to a link labelled Version.

  Clicking the Version link brings up a list of all the previous policy sets that were applied.  Not only does it list all the previous policy sets, it also tells us when they were applied.

Note that we have the option to view the details of a previous policy set by clicking on the view icon .

This shows us the policies that were applied to the component at a particular point in time.  By clicking on the View Details icon we can see the actual policy steps.

If after reviewing a previous policy set and the steps associated with the policies we realise that we need to restore that policy we can do so by selecting the restore link from the list of previous policy sets.

OWSM will ask for confirmation before restoring the previous policy set as the current working set.

Note that the current policy set is not obvious from the Versions page.  To check exactly what is being applied you need to open up the current policy and verify it is the version that you want before clicking OK.

You can drill down into the View Details to make absolutely sure that it is doing what you think before you commit the changes and so publish the new policies to the policy enforcement points.

So now you can travel back in time to correct errors in your OWSM policies.  But wait, there is more. When Sarah Jane went back in time she saved her parents and as a result destroyed her own future.  Similarly when we rolled back our policy to a previous time, we may also have lost the service mappings that relate policies to service endpoints.  The policy versions are actually policy set versions.  A policy set is the collection of policies available to a component.  If you fall back to a policy set that does not include a policy that you have applied to an endpoint, then that policy to endpoint mapping is also lost.  In the TV show Sarah Jane went back in time to undo her change.  Similarly you can also reverse your meddling in time by restoring a later policy and any lost endpoint mappings will also reappear.

Happy time travelling.

License to Use or Tracking Service Usage

I was on a training course this week around an Oracle SOA Reference Architecture.  This led to an interesting conversation about what is a service.  There was a difference in opinion on some of the characteristics of a service, most noticeably does a service have to be re-used to be a service.  There was however, general agreement that the definition of a service presented was a good one so I thought I would share it with you.  The service definition had four parts

1. Service Contract – the requirements document for the service in terms of what does the service do and what promises does it provide in terms of response time, scalability, availability.
2. Service Interface – the formal definition of operations and data formats provided by the service.
3. Service Implementation – the physical endpoint and code that provides the service.
4. Service Usage Agreement – the contract between the client and the service.

The one that may not be immediately obvious is the fourth one, service usage agreement.  This indicates the expected level of use of the service by the client, number of requests per time period, message size and concurrency for example.  This is important because SOA is about using services in place, sharing not just the service code, but also its implementation resources such as network bandwidth, memory and cpu resources.  Usage by one client may impact another and so it is important to set expectations of how a given client may use the service.  Service usage agreements may be realised by policies within a service bus that enforce some of the constraints agreed within the usage agreement.

So all told I think a very useful way to think about services and I will certainly be making sure I do a better job of formalising the service usage agreement in future.