Friday Jan 31, 2014

Handling Large Objects (e.g., Audio/Video files) in Oracle NoSQL DB

Recently, I have worked in a project that required using Oracle NoSQL Database to store complex structures like audio and video files, and make those files rapidly accessible through an web application. When I started the development of file retrieval from the KVStore, I got surprised about how bad documented this scenario is, specially if you are dealing with LOB's. So I decided to document examples of how I got the scenario solved.

Writing LOB Values into KVStore

So you have a file stored in the file system and you would like to store it in the KVStore to enable further retrieval? The best way to accomplish this is to extract the InputStream from the file and put its value into KVStore like this:

package com.oracle.bigdata.nosql;

import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.concurrent.TimeUnit;

import oracle.kv.Durability;
import oracle.kv.KVStore;
import oracle.kv.Key;

public class HowToStoreFilesFromFileSystem {

	// The reference below should be injected
	// by some runtime IoC mechanism or initialized
	// through constructor or setter methods...
	
	private KVStore store;

	public void storeValue(List<String> majorKeys, List<String> minorKeys,
			InputStream inputStream) {

		Key key = null;

		try {

			key = Key.createKey(majorKeys, minorKeys);
			store.putLOBIfAbsent(key, inputStream, Durability.COMMIT_NO_SYNC,
					10, TimeUnit.SECONDS);

		} catch (IOException ex) {

			throw new RuntimeException(
					"Error trying to store files into NoSQL Database", ex);

		}

	}

}

Reading LOB Values from the KVStore

Once you are ready to retrieve LOB values from the KVStore, perhaps wondering about how to rebuild the file in the file system, you can proceed this way:

package com.oracle.bigdata.nosql;

import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.concurrent.TimeUnit;

import oracle.kv.Consistency;
import oracle.kv.KVStore;
import oracle.kv.Key;
import oracle.kv.lob.InputStreamVersion;

public class HowToLoadFilesFromKVStore {

	// The reference below should be injected
	// by some runtime IoC mechanism or initialized
	// through constructor or setter methods...

	private KVStore store;

	public File buildFileFromKVStore(List<String> majorKeys,
			List<String> minorKeys, String fileName) {

		Key key = null;
		InputStreamVersion isVersion = null;

		int _byte = 0;
		File file = null;
		InputStream inputStream = null;
		BufferedInputStream bis = null;
		ByteArrayOutputStream baos = null;
		FileOutputStream fos = null;

		try {

			key = Key.createKey(majorKeys, minorKeys);
			isVersion = store.getLOB(key, Consistency.NONE_REQUIRED, 10,
					TimeUnit.SECONDS);

			if (isVersion != null) {

				inputStream = isVersion.getInputStream();
				bis = new BufferedInputStream(inputStream);
				baos = new ByteArrayOutputStream();

				while ((_byte = bis.read()) != -1) {
					baos.write(_byte);
				}

				file = new File(fileName);
				fos = new FileOutputStream(file);
				fos.write(baos.toByteArray());
				fos.flush();

			}

		} catch (Exception ex) {

			throw new RuntimeException(
					"Error trying to get files from NoSQL Database", ex);

		} finally {

			if (fos != null) {

				try {
					fos.close();
				} catch (IOException ioe) {
					ioe.printStackTrace();
				}

			}

		}

		return file;

	}

}

In the example above, the file is completely rebuilt in the file system to be accessed from any application. Note that once you have the BufferedInputStream in place, you don't actually need to rebuild the file completely in the file system in order to play with it. Perhaps you can transfer all those bytes to an mechanism that is perfectly capable to handle that stream in-memory, taking full advantage of handling data on-demand based on the streaming API of LOB's.

Saturday Jan 18, 2014

Capturing Business Events for Future Playback through Oracle Event Processing Platform

In the heart of OEP application development there is the understanding of the business events anatomy, the knowledge about its structure, frequency about when they happen, its relationship with other types of events and of course, its volume. Business events play a critical role in event-driven applications because they are the input for the problem that you are trying to solve. Without the business events, an event-driven application would be like a car engine without fuel.

When you are designing an EPN (Event Processing Network) of an OEP application, you are actually expressing the way about how the business events will be received, processed and perhaps transformed in some kind of output. Like any other technology, the behavior of this output depends heavily of which data you have used as input. For one specific volume, the EPN could work, for another volume, maybe not. For one specific ordering, the EPN could work, with another ordering, it could not. It will only work when the right set of events is being used.

Its very common the situation when you deploy the first version of your OEP application and after a while users start complaining about undesired results. This happens because no matter how many times you tested your application, you will never get close to the mass of events present in the customer environment. The ordering, volume, size, encoding and frequency of the events found in the customer environment is always different of that mass of events that you used during functional testing. If this situation happens to you, its time to think in a way to record those events and bring them back to the development environment to figure out what is wrong.

This article aims to explore the record and playback feature of Oracle Event Processing Platform. Through the examples shown here, you will be able to record and replay a mass of business events to simulate some behavior that maybe you have forgot to capture at your EPN. This feature is also pretty cool for prototyping scenarios. Instead of bringing with a you a simulator of events, you can just replay a recorded set of events and present your OEP application.

Configuring the OEP Default Provider

In order to the record and playback features work, OEP needs to relies in a repository in which events will be stored. To make the developer work easier, OEP brings out-of-the-box an default provider for this repository based on the Berkeley DB technology. Located inside of every OEP domain, there is an instance of Berkeley DB (aka "BDB") which can be adjusted through the configuration file of the domain. In most cases, you don't need to change anything before start using the record and playback features, but there is one catch that you should be aware of.

BDB stores each recorded event as an entry in this repository. Each entry has a default size of 1KB (1000 bytes) pre-defined in the moment that you create the domain. The information about the size of each entry is important in terms of performance and how BDB will growth its storage. So if you know the average size of your events, it is a good idea to inform the size of each entry in the domain configuration file. For instance, suppose that we are dealing with a mass of events with 32KB of average size for each event. You can adjust BDB editing the domain configuration file located in <DOMAIN_HOME>/<SERVER_NAME>/config/config.xml:

As you can see in the example above, you need to adjust the cache-size property of the bdb-config section. This property is measured in terms of bytes. The value of this property represents the amount of memory that BDB need to allocate for each entry.

Recording the Events

In order to record events, you need to access the OEP Visualizer Console of your domain. Log in onto the console and navigate to the EPN of the OEP application that you want to record events.

In the EPN of your OEP application, you can right-click any of the elements inside, no matter if it is an adapter, channel, processor or an cache. But make totally sense to record the events from the adapters perspective. They are the genesis of your EPN, and all the events that will flow through it will come from the adapters. Right-click the input adapter of your EPN and choose the "Record Event" option:

You will open the record tab of the adapter. In the bottom of the page, in a section named "Change Recording Schedule", you will find an button named "Add". Click on it.

Adding a new record configuration scheme enables the UI for full-filling some fields. Enter in the "DataSet Name" field the name of this recording session. This field will be used to construct the physical repository of BDB. You are also required to set the "Selected Event Type List" field. You need to describe in this field which incoming events you are expecting to record. After this you can click in the "Save" button.

We are all set to start recording. Double-check if there are events being sent to your OEP application and click in the "Start" button. Doing this will start a recording session and all events received by the adapter will be recorded. At any point of time, you can click in the "Stop" button to stop the recording session.

Playing Back the Events

To start the playback of any recorded session, you need to go back to the EPN screen and right-click the channel that will receive the events. This will bring you to the playback tab of the selected channel. Just like the recording session, you need to add one playback configuration scheme clicking in the "Add" button.

The group of fields to be full-filled is almost the same. Pay attemption that in the "DataSet Name" field you need to inform the exactly name that you informed in the recording session. Since you can have multiple recording sessions under different names, when you playback you need to tell OEP which session you would like to playback. As such, you need to inform which events you would like to playback, filling the "Selected Event Type List" field.

Optionally, you can set some additional parameters related to the behavior of the playback. In the "Change Playback Schedule Parameters" you can find fields to define which interval of time you want to playback. This is useful when you recorded hours/days of event streaming and you would like to reproduce just one piece of it. You can also define the speed in which events will be injected into the channel, useful to test how your EPN behave in a high speed velocity situation. Finally, you can set if the playback will be repeated forever. Since the recording session is finite, you can playback it as many times you want, or can just set the "Repeat" field as true to automatically restart from the beginning when the recorded streaming ends.

When you are done, save the playback configuration scheme and click in the "Start" button to start the playback session according your settings. An "Playing..." message will be flipping in the screen during the playback session. At any time, you can stop the playback clicking in the "Stop" button.

All the configuration you made so far is stored in the OEP domain along with your application so don't be shy to restart the OEP server and potentially lose your job: all the settings will remain there when the server come back to life. This statement is true as long you don't undeploy your application from the OEP domain. If you undeploy your application, all the settings will be lost. If you want make those settings permanent, you can set those into your application. Access this link to learn how to configure a component to record events, and this link to learn how to configure a component to playback events.

Wednesday Jan 08, 2014

Service Enablement of CORBA Applications through Oracle Service Bus

An quick overview in the definition of an ESB will tell us that one of its main responsibilities is among other things, the enablement of existing systems to provide new fresh services, using the same or maybe different protocols and/or contracts. This is when the ESB make magical things happen, virtualizing existing services and making them available for the outside world, abstracting from the applications that will consume the service details about the underlying system.

With this statement in mind, it is reasonable to think that an good ESB must be able to handle different types of systems and technologies found in legacy systems, no matter if this system was built last year, five years ago or even in the last decade. Those systems represents the assets of an organization in terms of their business building blocks, so there is a huge chance that those systems carries a substantial number of business services that could be leveraged by an SOA initiative.

CORBA is a distributed technology very powerful, that was pretty popular in the 90's and the beginning of 2000 year. Many industries that demands an robust infrastructure to handle their business transactions, critical by nature and extreme sensitive in terms of performance and reliability, relied on CORBA as technology implementation. It is pretty common to find communications companies like internet providers, mobile operators and pre-paid services chain that built its foundation (also known as engineering services) on top of CORBA systems.

This article will show how to enable CORBA systems through OSB, the ESB implementation from Oracle that is part of the SOA Suite. Through the steps showed here, you will be able to leverage existing CORBA systems and expose that business logic (defined as CORBA objects) in different transports, protocols and contracts, making the reuse of that business logic both possible and viable. This article will not cover any CORBA specific ORB to make the techniques available here reproducible in different contexts.

The Interface Definition Language

The definition of any CORBA object is written in an neutral description language called IDL, acronym of Interface Definition Language. For this example, I will consider that OSB will service enable an functionality that sends SMS messages, and this functionality is currently implemented as an object of an CORBA system. The IDL of this object is described below:

module corbaEnablementExample
{

    struct Message
    {

        string content;
        string phoneId;

    };

    interface SMSGateway
    {

        oneway void send(in Message message);

    };

};

As you can see, this is a very simple object that accepts an message as main parameter and the message has attributes that represents the content to be sent as an SMS message, and the mobile phone number that will receive the content.

The CORBA Server Application

It does not matter for the didactic of this article in which programming language the server part of the CORBA application will be implemented. What really matters is which ORB the CORBA server application will register its implementation stub. To illustrate the example, lets suppose that this CORBA object is implemented in Java.

package corbaEnablementApplication;

import org.omg.CORBA.ORB;
import org.omg.CORBA.Object;
import org.omg.CosNaming.NameComponent;
import org.omg.CosNaming.NamingContextExt;
import org.omg.CosNaming.NamingContextExtHelper;
import org.omg.PortableServer.POA;
import org.omg.PortableServer.POAHelper;

import corbaEnablementExample.SMSGateway;
import corbaEnablementExample.SMSGatewayHelper;

public class ServerApplication {

	public static void main(String[] args) {
		
		ORB orb = null;
		Object tmp = null;
		POA rootPOA = null;
		SMSGatewayImpl impl = null;
		Object ref, objRef = null;
		SMSGateway href = null;
		NamingContextExt ncRef = null;
		NameComponent path[] = null;
		
		try {
			
			System.setProperty("org.omg.CORBA.ORBInitialHost", "soa.suite.machine");
			System.setProperty("org.omg.CORBA.ORBInitialPort", "8001");
			orb = ORB.init(args, null);
			
			tmp = orb.resolve_initial_references("RootPOA");
			rootPOA = POAHelper.narrow(tmp);
			rootPOA.the_POAManager().activate();
			
			impl = new SMSGatewayImpl();
			ref = rootPOA.servant_to_reference(impl);
			href = SMSGatewayHelper.narrow(ref);
			
			objRef = orb.resolve_initial_references("NameService");
			ncRef = NamingContextExtHelper.narrow(objRef);
			
			path = ncRef.to_name("sms-gateway");
			ncRef.rebind(path, href);
			
			System.out.println("----------------------------------------------------");
			System.out.println("  CORBA Server is Running and waiting for Requests  ");
			System.out.println("----------------------------------------------------");
			
			orb.run();
			
		} catch (Exception ex) {
			ex.printStackTrace();
		}
		
	}

}

The code listing above shows an CORBA server application that connects onto a ORB available on one 8001 TCP/IP port. After retrieve the POA from the ORB, it get access to the naming service that will be used to register the object implementation. Finally, the application binds the object implementation under the name of "sms-gateway", the name which the CORBA object will be known from the outside world. In order to test this CORBA server application, start an ORB under the port 8001 and execute the program using one JVM. If you don't have any commercial ORB available, you can use the ORB which comes with the JDK. Just enter in the /bin folder of your JDK and type:

orbd -ORBInitialHost soa.suite.machine -ORBInitialPort 8001

To check if this remote object is working properly, you need to write an CORBA client application. Here is an example of an CORBA client written upon the same IDL interface which the server was written:

package corbaEnablementApplication;

import java.util.Random;
import java.util.UUID;

import org.omg.CORBA.ORB;
import org.omg.CORBA.Object;
import org.omg.CosNaming.NamingContextExt;
import org.omg.CosNaming.NamingContextExtHelper;

import corbaEnablementExample.Message;
import corbaEnablementExample.SMSGateway;
import corbaEnablementExample.SMSGatewayHelper;

public class ClientApplication {
	
	private static final Random random = new Random();

	public static void main(String[] args) {
		
		ORB orb = null;
		Object objRef = null;
		NamingContextExt ncRef = null;
		SMSGateway smsGateway = null;
		
		try {
			
			System.setProperty("org.omg.CORBA.ORBInitialHost", "soa.suite.machine");
			System.setProperty("org.omg.CORBA.ORBInitialPort", "8001");
			orb = ORB.init(args, null);
			
			objRef = orb.resolve_initial_references("NameService");
			ncRef = NamingContextExtHelper.narrow(objRef);
			
			smsGateway = SMSGatewayHelper.narrow(ncRef.resolve_str("sms-gateway"));
			Message message = createNewRandomMessage();
			smsGateway.send(message);
			
		} catch (Exception ex) {
			ex.printStackTrace();
		}
		
	}

	private static Message createNewRandomMessage() {
		String content = UUID.randomUUID().toString();
		String phoneId = String.valueOf(random.nextLong());
		Message message = new Message(content, phoneId);
		return message;
	}

}

The Business Services Layer

In order to OSB get access to the remote object, it is necessary to create an mechanism that can translate the IIOP protocol (the protocol used in pure CORBA systems) for one protocol that OSB can understand, which could be RMI/IIOP or pure RMI. To accomplish that, the best way is to implement the wrapper pattern. Write down one EJB 3.0 service that encapsulates the CORBA remote object, and delegates its service calls to this object. The interface for this EJB 3.0 service should be something simpler like this:

package com.oracle.fmw.soa.osb.corba;

public interface SMSGateway {
	
	public void sendMessage(String content, String phoneId);

}

The implementation of this EJB 3.0 service should perform a job similar to the CORBA client application described previously, but quite different in terms of how it connect to an ORB:

package com.oracle.fmw.soa.osb.corba;

import javax.annotation.PostConstruct;
import javax.annotation.Resource;
import javax.ejb.Remote;
import javax.ejb.Stateless;

import org.omg.CORBA.ORB;
import org.omg.CORBA.Object;
import org.omg.CosNaming.NamingContextExt;
import org.omg.CosNaming.NamingContextExtHelper;

import corbaEnablementExample.Message;
import corbaEnablementExample.SMSGatewayHelper;

@Remote(value = SMSGateway.class)
@Stateless(name = "SMSGateway", mappedName = "SMSGateway")
public class SMSGatewayImpl implements SMSGateway {
	
	private corbaEnablementExample.SMSGateway smsGateway;
	
	@Resource(name = "namingService")
	private String namingService;
	
	@Resource(name = "objectName")
	private String objectName;
	
	@PostConstruct
	@SuppressWarnings("unused")
	private void retrieveStub() {
		
		ORB orb = null;
		Object objRef = null;
		NamingContextExt ncRef = null;
		
		try {
			
			orb = ORB.init();
			objRef = orb.resolve_initial_references(namingService);
			ncRef = NamingContextExtHelper.narrow(objRef);
			smsGateway = SMSGatewayHelper.narrow(ncRef.resolve_str(objectName));
			
		} catch (Exception ex) {
			
			throw new RuntimeException("EJB wrapper failed in the retrieval of the CORBA stub.", ex);
			
		}
		
	}

	@Override
	public void sendMessage(String content, String phoneId) {
		
		smsGateway.send(new Message(content, phoneId));
			
	}

}

The code is very similar to the CORBA client application showed before, with one important difference: it has no information about which ORB to connect. In this case, the EJB will reside in on WebLogic JVM. Each WebLogic JVM has one ORB implementation out-of-the-box. So when you write EJB objects that will wrap-up CORBA remote objects, you don't need to worry about which ORB to use. WebLogic it is already an ORB.


Note: as explained before, this article will not enter in details of any commercial ORB available, in defense of the clarity and didactic of the article. But keep in mind that the steps shown here for the stub retrieval can be quite different if you are using another ORB. If you are using Borland VisiBroker for instance, there is a unique way to access the ORB which is using an service called "Smart Agent", which dynamically finds another objects in the network. IONA Orbix has another unique way to connect to an ORB, which is by the use of the domain configuration location of the Orbix network.


Create one WebLogic domain and execute one or more WebLogic managed servers, and re-run the CORBA server application again. Remember that now, the CORBA server application should point to the WebLogic port since the ORB now should be the one available in the WebLogic subsystem. If you check the JNDI tree of the WebLogic JVM, you should see something like this:

This means that the remote CORBA object was properly registered in the CosNaming service available in the WebLogic ORB. Package the EJB 3.0 implementation into a JAR or an EAR and deploy it in the same WebLogic JVM that the CORBA remote object was registered. Now we have everything in place to start the development of the OSB project. For the purposes of this article, I will assume that the EJB 3.0 object is available under the following JNDI name: "SMSGateway#com.oracle.fmw.soa.osb.corba.SMSGateway".

The OSB Project

In the OSB side, all you have to do is to create an business service that points to one or more endpoints of the EJB 3.0 that is running in the one or more servers of the WebLogic domain. In order to accomplish that, you will need to teach OSB about how to communicate with this foreign WebLogic domain. This is done creating an JNDI provider for the OSB configuration scheme:


OSB also needs to access the EJB 3.0 interfaces (and any other helper classes) to instantiate client proxies, so you need to package all the EJB 3.0 artifacts (except of course from the enterprise bean implementation) and deploy it onto your OSB project:

Now we have everything in place. It is time to create the business service that will point to the EJB 3.0 wrapper. Create one business service and set its service type to "Transport Typed":

Configure the business service protocol as "EJB" and set its endpoint URI to the prefix "ejb:" plus the name of the JNDI provider and plus the JNDI name of the EJB 3.0:

Finally, you need to configure the client interface of the EJB 3.0 endpoint in the business service configuration page. Check the "EJB 3.0" checkbox and choose from the drop-down list which interface will be used for message communication.

Finish the creation of the business service and save the changes. You can now test your business service using the testing tool available on OSB:

After making an request to the business service using the OSB testing tool, you can check the CORBA server application log to see the results of this invocation. Here is an example:

With the business service in place, you can easily create one or more proxy services to access the remote CORBA object with minimal effort. For the OSB perspective, it is all about routing messages to the business service that you created, making the fact that this business service is a CORBA remote object really irrelevant.

No matter what will be your use case, now you have the CORBA remote object available in OSB for virtually anything. You can expose it directly using one of the available transports, you can forward messages for it in the middle of your pipeline, you can use it as enrichment mechanism using service callouts or you can just use the business service as one of the choices of an dynamic routing. If you choose to expose this business service into a new protocol, you can play with SOAP, REST, HTTP, JMS, Email, Tuxedo, File and FTP with zero-coding. OSB will take care of the protocol translation during messages exchanges.

You can download the project artifacts created in this article here.

About

Ricardo Ferreira is just a regular person that lives in Brazil and is passionate for technology, movies and his whole family. Currently is working at Oracle Corporation.

Search

Categories
Archives
« January 2014
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
9
10
11
12
13
14
15
16
17
19
20
21
22
23
24
25
26
27
28
29
30
 
       
Today