Monday Aug 13, 2012

Automatically Disable Proxy Service to avoid overloading OSB

A very frequently asked question of using Oracle Service Bus (OSB) is how to avoid OSB to be overloaded once an endpoint becomes temporarily unreachable. The same question is equally asked for an endpoint doesn't respond as quickly as expected and, in worse case, the situation deteriorates gradually. The concern is raised for the potential risk of overwhelming OSB if the relevant proxy service continues receiving requests from callers.

From the design practice point of view, it is always recommended to make your endpoint highly available. With the help of either load balancer or OSB's failover capability across multiple endpoints, you should be able to make your OSB safer and less vulnerable from endpoint errors. In addition, you'd better implement OSB best practice by configuring a Work Manager attached to Proxy Service and enabling throttling control on Business Service. The details of these two features will not be covered in this blog. You can find the details of using work manager with OSB here and How to enable and use throttling control here in the documentation.

However, this is not the whole picture of the story. Once an unexpected problem occurs on endpoint, you certainly want to be noticed for the incident and take some appropriate action to cope with it especially in the case where you have only one endpoint or the endpoint systems reside externally and out of your control.

Now, let's see how can you get notified when the following two common problems occur at endpoint:

  • the endpoint becomes unreachable
  • the endpoint still alive but the response becomes slow and results in SLA violation

OSB provides a nice feature to offline the unresponsive endpoint URI due to communication error. A Service Level Agreement (SLA) rule can be defined to trigger the alert once endpoint URI is marked as offline. You can also define a SLA regarding response time. If the response time is longer than the defined value, the alert can be fired automatically by OSB.

Once alerted, it is up to you to decide what are the appropriate actions to take. If the problem on the endpoint persists and cannot be resolved quickly, a possible action you might want to take is to stop the Proxy Service for not receiving requests anymore until the problem is sorted out. You can manually disable the Proxy Service via OSB console or WLST to achieve this.  This blog shows an alternative approach which is to disable Proxy Service automatically via OSB's Java API when SLA is violated due to endpoint abnormalities.

The following steps highlight how to implement the solution:

  1. create a SLA rule which triggers alert if the response time is longer than the expected one or the endpoint is unreachable
  2. configure alert to be sent to a designated JMS queue
  3. create an "administrative" Proxy Service to listen on the JMS queue
  4. make a Java callout within the "administrative" Proxy Service to disable the relevant Proxy Service once the alert is received from JMS queue.  

 The details are as follows:

Create an alert destination to send alert to a designated JMS queue.

 Create the Business Service and enable the offline endpoint URI. If the endpoint URI is unreachable, OSB changes its status to offline automatically.

Define the SLA. An alert will be triggered if the max response time > 5000 ms OR the endpoint is marked as offline.

Create the "administrative" Proxy Service listening on the designated JMS queue

 The Proxy Service, upon the receipt of alert message, get the JMS headers set by OSB when sending alert to JMS queue. In this blog, we simply use alert message's ServiceName header to find out which Business Service fired this alert.  The XAPTH expression to achieve this is:

$inbound/ctx:transport/ctx:request/tp:headers/tp:user-header[@name="ServiceName"]/@value.

Based on the Business Service name, you can configure some parameters to find out which Proxy Service is supposed to be disabled. And finally, make a Java Callout to disable that Proxy Service. In this blog, the way to determine the Proxy Service is simply hard coded.

The following is the sample Java code which disables the Proxy Service. It shows how to programatically enable or disable a Proxy Service via Java API. The same code can be applied to Business Service as well.

package com.oracle.ateam;

import java.io.IOException;
import java.net.MalformedURLException;
import java.util.Hashtable;

import com.bea.wli.sb.management.configuration.SessionManagementMBean;
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.naming.Context;
import weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean;
import weblogic.management.jmx.MBeanServerInvocationHandler;

import com.bea.wli.config.Ref;
import com.bea.wli.sb.management.configuration.ProxyServiceConfigurationMBean;

public class ServiceManager {

    private static JMXConnector initConnection(String hostname, int port,
            String username, String password)
    throws IOException,MalformedURLException
    {
        JMXServiceURL serviceURL =
        new JMXServiceURL("t3", hostname, port,
        "/jndi/" + DomainRuntimeServiceMBean.MBEANSERVER_JNDI_NAME);

        Hashtable<String, String> h = new Hashtable<String, String>();
        h.put(Context.SECURITY_PRINCIPAL, username);
        h.put(Context.SECURITY_CREDENTIALS, password);
        h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");

        return JMXConnectorFactory.connect(serviceURL, h);
    }

    private static Ref convertServiceURI2Ref(String resType,String serviceuri){
        Ref ref = null;
        if((serviceuri.equals(""))||(serviceuri==null))
                return ref;

        String[] uriData = serviceuri.split("/");
        ref = new Ref(resType,uriData);
        return ref;
    }

    public static void changeProxyServiceStatus(String serviceref,boolean status)throws Exception{
        JMXConnector conn = null;
        SessionManagementMBean sm = null;
        String sessionName = "Session.ByApp." + System.currentTimeMillis();

        try{

            conn = initConnection("localhost", 7001, "weblogic", "welcome1");
            MBeanServerConnection mbconn = conn.getMBeanServerConnection();
            DomainRuntimeServiceMBean domainService = (DomainRuntimeServiceMBean) MBeanServerInvocationHandler.
                 newProxyInstance(mbconn, new ObjectName(DomainRuntimeServiceMBean.OBJECT_NAME));

            sm = (SessionManagementMBean) domainService.
                 findService(SessionManagementMBean.NAME,
                             SessionManagementMBean.TYPE, null);

            sm.createSession(sessionName);

            ProxyServiceConfigurationMBean proxyConfigMBean = (ProxyServiceConfigurationMBean) domainService.
             findService(ProxyServiceConfigurationMBean.NAME + "." + sessionName,
                     ProxyServiceConfigurationMBean.TYPE, null);
            Ref ref = convertServiceURI2Ref("ProxyService",serviceref);
            String msg = "";
            if(!status){
               proxyConfigMBean.disableService(ref);
               msg="Disabled the Proxy Service : " + serviceref;
            }
            else {             
                proxyConfigMBean.enableService(ref);   
                msg="enabled the Proxy Service : " + serviceref;
            }                                 

            sm.activateSession(sessionName, msg);
            System.out.println(msg);
            conn.close();
        }catch(Exception ex){
            if(null != sm) {
                try{
                    sm.discardSession(sessionName);
                }catch(Exception e) {
                    System.out.println("discard session error");
                }
            }
            throw ex;
        }finally{
            if(null != conn)
                try{
                    conn.close();
                }catch(Exception e) {
                    e.printStackTrace();
                }
        }
    }

}

By putting a sleep of 10 seconds at the endpoint which is greater than the defined 5 seconds in SLA, the Proxy Service is automatically disabled as shown below.

Make another round of test by killing the endpoint, the same result (automatically disabled the Proxy Service) outcomes as expected. At the same time, we can also see the endpoint is automatically marked offline by OSB.

This blog demonstrates a way to automatically turn off a Proxy Service when the endpoint is unreachable or the SLA is violated. However, you need to be cautious when considering the automation. The reason is there are so many factors and criteria to consider and evaluate to make the decision of whether or when the Proxy Service should be disabled. Alternatively, you can also break the above sample into two steps for being alerted and disabling Proxy Service via Java API, then add a manual decision making step in between.      

About


This is the blog for the Oracle FMW Architects team fondly known as the A-Team. The A-Team is the central, technical, outbound team as part of the FMW Development organization working with Oracle's largest and most important customers. We support Oracle Sales, Consulting and Support when deep technical and architectural help is needed from Oracle Development.
Primarily this blog is tailored for SOA issues (BPEL, OSB, BPM, Adapters, CEP, B2B, JCAP)that are encountered by our team. Expect real solutions to customer problems, encountered during customer engagements.
We will highlight best practices, workarounds, architectural discussions, and discuss topics that are relevant in the SOA technical space today.

Search

Archives
« August 2012 »
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
31
 
       
Today