Wednesday Jan 30, 2013

OSB Performance Tuning - RouterRuntimeCache

Many customers start out with smaller projects for an initial release.  Typically, these applications require 20-30 Proxy services.  But as time goes on and later phases of the project rollout, the number of proxy services can increase drastically.  The RouterRuntimeCache is a cache implemented by OSB to improve performance by eliminating or reducing the amount of time spent on compiling the proxy pipeline. 

By default, OSB will not compile a pipeline until a request message for a given service is received.  Once it has been compiled, the pipeline is cached in memory for re-use.  You have probably noticed in testing that the first request to a service takes longer to respond than subsequent requests, and this is a big part of the reason.  Since free heap space is often at a premium, this cache can not be infinite in size so this cache has a built in limit.  When the cache is full, the least recently used entry is released and the pipeline that is currently being requested is placed into cache in its place.  The next time a request comes in for the service who's pipeline was released, that pipeline has to be re-compiled and placed in cache, again forcing out the least recently used pipeline.  Once a pipeline is placed in cache it is never removed from cache unless forced out by a full cache scenario as above, or if the service is updated, forcing it to be recompiled.

The default size limit of the RouterRuntimeCache is 100 entries (or pipelines).  It is limited by the number of services in the cache, not the memory used by the cache so the amount of memory used by a full cache will vary greatly based on the complexity of the services, the extent and complexity of inline xquery, etc.  If your project grows beyond 100 proxy services, system performance can degrade significantly if the cache size is not increased to hold all frequently used services. 

Unfortunately, the way to tune this cache is not exposed through the OSB console.  As of 11g PS5, the only way to set this parameter is via a system property specified on the Java command-line.  The property name is com.bea.wli.sb.pipeline.RouterRuntimeCache.size.   For example,

“java … -Dcom.bea.wli.sb.pipeline.RouterRuntimeCache.size=500 … weblogic.Server …”. 

In this example, OSB will cache 500 proxies, instead of the default 100.  Because increasing the RouterRuntimeCache.size value will require more space in the heap to hold the additional proxies, be aware that you may need to reevaluate your JVM memory settings to allow OSB to continue to perform optimally.

Friday Sep 28, 2012

OSB, Service Callouts and OQL - Part 2

This section of the "OSB, Service Callouts and OQL" blog posting will delve into thread dump analysis of OSB server and detecting threading issues relating to Service Callout using ThreadLogic. We would also use Heap Dump and OQL to identify the related Proxies and Business services. The previous section dealt with threading model used by OSB to handle Route and Service Callouts.

[Read more..]

OSB, Service Callouts and OQL - Part 1

Oracle Fusion Middleware customers use Oracle Service Bus (OSB) for virtualizing Service endpoints and implementing stateless service orchestrations. Behind the performance and speed of OSB, there are a couple of key design implementations that can affect application performance and behavior under heavy load. One of the heavily used feature in OSB is the Service Callout pipeline action for message enrichment and invoking multiple services as part of one single orchestration. Overuse of this feature, without understanding its internal implementation, can lead to serious problems.

This post will delve into OSB internals, the problem associated with usage of Service Callout under high loads, diagnosing it via thread dump and heap dump analysis using tools like ThreadLogic and OQL (Object Query Language) and resolving it. The first section in the series will mainly cover the threading model used internally by OSB for implementing Route Vs. Service Callouts.

[Read more...]

OSB, Service Callouts and OQL - Part 3

In the previous sections of the "OSB, Service Callouts and OQL" series, we analyzed the threading model used by OSB for Service Callouts and analysis of OSB Server threads hung in Service callouts and identifying  the Proxies and Remote services involved in the hang using OQL.

This final section of the series will focus on the corrective action to avoid Service Callout related OSB Server hangs.

[Read more..]

Monday Aug 13, 2012

Automatically Disable Proxy Service to avoid overloading OSB

A very frequently asked question of using Oracle Service Bus (OSB) is how to avoid OSB to be overloaded once an endpoint becomes temporarily unreachable. The same question is equally asked for an endpoint doesn't respond as quickly as expected and, in worse case, the situation deteriorates gradually. The concern is raised for the potential risk of overwhelming OSB if the relevant proxy service continues receiving requests from callers.

From the design practice point of view, it is always recommended to make your endpoint highly available. With the help of either load balancer or OSB's failover capability across multiple endpoints, you should be able to make your OSB safer and less vulnerable from endpoint errors. In addition, you'd better implement OSB best practice by configuring a Work Manager attached to Proxy Service and enabling throttling control on Business Service. The details of these two features will not be covered in this blog. You can find the details of using work manager with OSB here and How to enable and use throttling control here in the documentation.

However, this is not the whole picture of the story. Once an unexpected problem occurs on endpoint, you certainly want to be noticed for the incident and take some appropriate action to cope with it especially in the case where you have only one endpoint or the endpoint systems reside externally and out of your control.

Now, let's see how can you get notified when the following two common problems occur at endpoint:

  • the endpoint becomes unreachable
  • the endpoint still alive but the response becomes slow and results in SLA violation

OSB provides a nice feature to offline the unresponsive endpoint URI due to communication error. A Service Level Agreement (SLA) rule can be defined to trigger the alert once endpoint URI is marked as offline. You can also define a SLA regarding response time. If the response time is longer than the defined value, the alert can be fired automatically by OSB.

Once alerted, it is up to you to decide what are the appropriate actions to take. If the problem on the endpoint persists and cannot be resolved quickly, a possible action you might want to take is to stop the Proxy Service for not receiving requests anymore until the problem is sorted out. You can manually disable the Proxy Service via OSB console or WLST to achieve this.  This blog shows an alternative approach which is to disable Proxy Service automatically via OSB's Java API when SLA is violated due to endpoint abnormalities.

The following steps highlight how to implement the solution:

  1. create a SLA rule which triggers alert if the response time is longer than the expected one or the endpoint is unreachable
  2. configure alert to be sent to a designated JMS queue
  3. create an "administrative" Proxy Service to listen on the JMS queue
  4. make a Java callout within the "administrative" Proxy Service to disable the relevant Proxy Service once the alert is received from JMS queue.  

 The details are as follows:

Create an alert destination to send alert to a designated JMS queue.

 Create the Business Service and enable the offline endpoint URI. If the endpoint URI is unreachable, OSB changes its status to offline automatically.

Define the SLA. An alert will be triggered if the max response time > 5000 ms OR the endpoint is marked as offline.

Create the "administrative" Proxy Service listening on the designated JMS queue

 The Proxy Service, upon the receipt of alert message, get the JMS headers set by OSB when sending alert to JMS queue. In this blog, we simply use alert message's ServiceName header to find out which Business Service fired this alert.  The XAPTH expression to achieve this is:

$inbound/ctx:transport/ctx:request/tp:headers/tp:user-header[@name="ServiceName"]/@value.

Based on the Business Service name, you can configure some parameters to find out which Proxy Service is supposed to be disabled. And finally, make a Java Callout to disable that Proxy Service. In this blog, the way to determine the Proxy Service is simply hard coded.

The following is the sample Java code which disables the Proxy Service. It shows how to programatically enable or disable a Proxy Service via Java API. The same code can be applied to Business Service as well.

package com.oracle.ateam;

import java.io.IOException;
import java.net.MalformedURLException;
import java.util.Hashtable;

import com.bea.wli.sb.management.configuration.SessionManagementMBean;
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.naming.Context;
import weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean;
import weblogic.management.jmx.MBeanServerInvocationHandler;

import com.bea.wli.config.Ref;
import com.bea.wli.sb.management.configuration.ProxyServiceConfigurationMBean;

public class ServiceManager {

    private static JMXConnector initConnection(String hostname, int port,
            String username, String password)
    throws IOException,MalformedURLException
    {
        JMXServiceURL serviceURL =
        new JMXServiceURL("t3", hostname, port,
        "/jndi/" + DomainRuntimeServiceMBean.MBEANSERVER_JNDI_NAME);

        Hashtable<String, String> h = new Hashtable<String, String>();
        h.put(Context.SECURITY_PRINCIPAL, username);
        h.put(Context.SECURITY_CREDENTIALS, password);
        h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");

        return JMXConnectorFactory.connect(serviceURL, h);
    }

    private static Ref convertServiceURI2Ref(String resType,String serviceuri){
        Ref ref = null;
        if((serviceuri.equals(""))||(serviceuri==null))
                return ref;

        String[] uriData = serviceuri.split("/");
        ref = new Ref(resType,uriData);
        return ref;
    }

    public static void changeProxyServiceStatus(String serviceref,boolean status)throws Exception{
        JMXConnector conn = null;
        SessionManagementMBean sm = null;
        String sessionName = "Session.ByApp." + System.currentTimeMillis();

        try{

            conn = initConnection("localhost", 7001, "weblogic", "welcome1");
            MBeanServerConnection mbconn = conn.getMBeanServerConnection();
            DomainRuntimeServiceMBean domainService = (DomainRuntimeServiceMBean) MBeanServerInvocationHandler.
                 newProxyInstance(mbconn, new ObjectName(DomainRuntimeServiceMBean.OBJECT_NAME));

            sm = (SessionManagementMBean) domainService.
                 findService(SessionManagementMBean.NAME,
                             SessionManagementMBean.TYPE, null);

            sm.createSession(sessionName);

            ProxyServiceConfigurationMBean proxyConfigMBean = (ProxyServiceConfigurationMBean) domainService.
             findService(ProxyServiceConfigurationMBean.NAME + "." + sessionName,
                     ProxyServiceConfigurationMBean.TYPE, null);
            Ref ref = convertServiceURI2Ref("ProxyService",serviceref);
            String msg = "";
            if(!status){
               proxyConfigMBean.disableService(ref);
               msg="Disabled the Proxy Service : " + serviceref;
            }
            else {             
                proxyConfigMBean.enableService(ref);   
                msg="enabled the Proxy Service : " + serviceref;
            }                                 

            sm.activateSession(sessionName, msg);
            System.out.println(msg);
            conn.close();
        }catch(Exception ex){
            if(null != sm) {
                try{
                    sm.discardSession(sessionName);
                }catch(Exception e) {
                    System.out.println("discard session error");
                }
            }
            throw ex;
        }finally{
            if(null != conn)
                try{
                    conn.close();
                }catch(Exception e) {
                    e.printStackTrace();
                }
        }
    }

}

By putting a sleep of 10 seconds at the endpoint which is greater than the defined 5 seconds in SLA, the Proxy Service is automatically disabled as shown below.

Make another round of test by killing the endpoint, the same result (automatically disabled the Proxy Service) outcomes as expected. At the same time, we can also see the endpoint is automatically marked offline by OSB.

This blog demonstrates a way to automatically turn off a Proxy Service when the endpoint is unreachable or the SLA is violated. However, you need to be cautious when considering the automation. The reason is there are so many factors and criteria to consider and evaluate to make the decision of whether or when the Proxy Service should be disabled. Alternatively, you can also break the above sample into two steps for being alerted and disabling Proxy Service via Java API, then add a manual decision making step in between.      

Tuesday Apr 03, 2012

How to deal with transport level security policy with OSB

OSB 11g PS4 consuming a Web service, that is secured by HTTP transport level security policy.[Read More]

Friday Nov 04, 2011

An example of how to process a zipped file in OSB

This post describes processing a zipped payload when passed as part of a multipart MIME message via HTTP.[Read More]
About


This is the blog for the Oracle FMW Architects team fondly known as the A-Team. The A-Team is the central, technical, outbound team as part of the FMW Development organization working with Oracle's largest and most important customers. We support Oracle Sales, Consulting and Support when deep technical and architectural help is needed from Oracle Development.
Primarily this blog is tailored for SOA issues (BPEL, OSB, BPM, Adapters, CEP, B2B, JCAP)that are encountered by our team. Expect real solutions to customer problems, encountered during customer engagements.
We will highlight best practices, workarounds, architectural discussions, and discuss topics that are relevant in the SOA technical space today.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today