Friday Aug 29, 2014

Managing SLAs with Oracle Event Processing

Do you need to manage your SLAs in real-time? This year's Oracle OpenWorld hands-on lab for Oracle Event Processing will show you how to quickly develop an application that can help you manage your service level agreements. Whether you are running SOA composites or have any data source containing information about your processing of requests, this hands-on lab will show you how to process the data in real-time and generate an alert if you are about to miss a service level agreement, even if these SLAs vary by customer or type of request. No prior knowledge of Oracle Event Processing is required. This session will fill up quickly, so be sure to register for HOL 9396: Hands-on Fast Data development with Oracle Event Processing 12c.

Thursday Apr 26, 2012

Testing with OCEP Shell

OCEP Shell

The OCEP shell is a new profile of the OCEP server 11g PS5 that has been created to facilitate the testing of OCEP applications.

To enable this profile, you need to unzip OCEP Shell to the modules directory of your domain.

For example, if you are using the default domain that comes with the product installation at Oracle/Middleware/user_projects/domains/ocep_domain/defaultserver, you would unzip the referenced file so that the content of defaultserver/modules becomes:





Start the server as usual, and you will get a shell prompt, as follows (in a Unix environment):

<Mar 13, 2012 8:02:16 AM BRT> <Notice> <Server> <BEA-2046000> <Server STARTED>


Oracle CEP Shell (using Apache Felix Gogo)


The OCEP shell uses Apache Felix Gogo shell. Please, take a quick look at Gogo’s documentation before proceeding. It can be found at: .Felix Gogo is based upon OSGi RFC 147, which provides a standard shell command prompt for OSGi frameworks.

Using the OCEP shell, you can invoke commands to test, and manage the OCEP server, as well as to test and manage the OSGi framework itself. The commands are organized in categories, named by a prefix. You can get a list of all commands and their categories by invoking the help command. Currently, the following categories of commands are supported:

epn, felix, gogo, mngt, obr

In the next section, we take a look at how to create a simple EPN containing CQL queries for testing.

EPN Commands

We start by creating a new EPN session. You can think of an EPN session as a stand-alone EPN that can be manipulated dynamically. When you are done, you can simply end the session, and create a new one if we want to test different things.

You begin a session by invoking the command:

shell> begin

Because the command begin is unique across all categories, you can omit the epn prefix. You can get descriptive help for a command by invoking help <command>, such as in the following example:
shell> help begin

begin - Begins new session for invoking EPN commands

scope: epn


By default, an EPN session has an implicit CQL processor that is connected to an event sink that prints all outputs to the shell console. So all we need to do to test CQL queries is to create an input channel, define the queries, and send events.

The following example does this:


channel MyChannel [msg=String]

query "select * from MyChannel"

send MyChannel [msg='Hi']


The result is:
1:19:39 PM -> insert event: msg=Hi

The channel command creates a channel named “MyChannel”, whose event-type has a single event property named msg of type String. You could also have specified a Java Class name as the event-type name.

The query command registers the query “select * from MyChannel” in the session’s implicit CQL processor. You can remove the query by using the remove command.

The send command dispatches an insert event to a channel. If the session only has a single channel, then the name of the channel is optional. In other words, for this particular example, the following two commands would be equivalent: “send MyChannel [msg=’Hi’]” and “send [msg=’Hi’]”.

As it should be obvious from the Gogo documentation, the syntax “[msg=’Hi’]” creates a map containing a single key-value pair, whose key is “msg” and value is “Hi”. This matches with the event-type we created previously while defining the channel.

Further, you can use the update command and the delete command to respectively send an update event and a delete event to a relation-based channel. The insert command is equivalent to the send command.

Finally, you can check the current registered statements in a session with the statement command. Likewise, you can find out the channels you have created in a session with the channels command. Finally, the eventtypes commands allows you to find out the structure of all the event-types currently in the server.

And that’s it, by running these simple set of commands, you can fully construct an EPN and try out CQL statements on the fly. In the next section, we take a look at how to manage the OCEP server and to test an existing OCEP application using the OCEP shell.

Management Commands

In this section, we take a look at the management category of commands. Let’s start by finding out all the deployed OCEP applications in the running server. You can do this by invoking the command mngt:listapps. Here is an example:

shell> listapps


Further, you can list all the OCEP libraries using the command listlibs:
shell> listlibs




Noticed how this command lists all the bundles we have unzipped in the modules directory to run the OCEP shell.

Next, you can install a new application using the deployapp <URL> command:

shell> deployapp file:///Users/ocepapps/helloworld.jar

The application is deployed and started immediately, as if it had been deployed by using the Visualizer management web-console. Conversely, you can un-deploy an application by invoking the command undeployapp.

You can send events to an existing OCEP application by specifying the full name of the channel when using the send, insert, update, and delete commands. For example, to send an event to the helloworldInputChannel in the helloworld application, you should use the name “helloworld:helloworldInputChannel”, as in the following example:

shell> event1 = createevent HelloWorldEvent

shell> $event1 message “Hi Shell!”

shell> send helloworld:helloworldInputChannel $event1

First, we create an event instance of the event type HelloWorldEvent. This is accomplished using the createevent command. Next, we populate the event with the value “Hi Shell!”, and then send it using the send command.

There is one caveat, the channel must have been advertised, otherwise the shell won’t be able to find it, so make sure that the advertise attribute is set to true in the application’s EPN assembly, as follows:

<wlevs:channel id="helloworldInputChannel"

event-type="HelloWorldEvent" advertise="true" >

You can also subscribe to receive events from a channel. You can do this using the subscribe command. The events subscribed will be send either to the shell console or to a file, if a file-name is specified. For example, the following command subscribes to all output from the helloworld application:
shell> subscribe helloworld:helloworldOutputChannel

As in the previous case, make sure that the channel being subscribed has also been advertised.

You can use the introspect command to list all the stages that are public, that is, have been advertised, in an application. The following example shows the result you get when invoking the introspect command in the out-of-the-box helloworld application:

shell> introspect helloworld

Application 'helloworld' provides the following OCEP services:

Event Channel 'helloworldOutputChannel' for Event Type 'HelloWorldEvent'

Using the Gogo shell facilities you can retrieve any OSGi service and invoke standard Java methods. In addition, the OCEP shell adds the mbean command that allows you to retrieve a EPN JMX MBean and invoke its operations. In the following example, we retrieve the CQLProcessorMBean for the helloworld processor, and invoke its operation “getAllQueries”.
shell> proc = mbean helloworld:helloworldProcessor CQLProcessor

shell> $proc allQueries


select * from helloworldInputChannel

The OCEP server exposes a rich set of JMX operations, which can now be easily tested and manipulated using the mbean command.

Finally, when you are all done, you can invoke the shutdown command to stop the shell and the OCEP server.

Regression Testing

The OCEP shell facilitates the testing of OCEP applications. The next step is to be able to automate this process. This can be done by running the OCEP shell head-less using scripts. For example, consider the following script, named send-event.ocep:


channel -a [a=Long]

query "select * from ch0"

send 0 [a=1]

send 1 [a=2]


You can execute this script by specifying the gosh.args system property. To do this, edit the last line of the file (startwlevs.cmd in Windows), as in the following example:
"$JAVA_HOME/bin/java" -Dgosh.args=send-event.ocep $JVM_ARGS $DEBUG_ARGS -Dwlevs.home="$USER_INSTALL_DIR" -Dbea.home="$BEA_HOME"  -jar "${USER_INSTALL_DIR}/bin/wlevs.jar" $ARGS

One approach is to have a test-driver script that invokes other scripts and directs their output, as in this example:
source send-event.ocep | tac test-output/log/send-event.out

source test-delete.ocep | tac test-output/log/test-delete.out


The OCEP shell provides a quick and easy way to prototype, test, and regression test OCEP applications, EPN assemblies, and CQL queries.

Just keep in mind that the OCEP shell is currently a technology preview for Oracle CEP 11g PS5. We will keep improving it, so please send us your feedback! Enjoy.

Tuesday Nov 29, 2011

Aggregating cache data from OCEP in CQL

There are several use cases where OCEP applications need to join stream data with external data, such as data available in a Coherence cache. OCEP’s streaming language, CQL, supports simple cache-key based joins of stream data with data in Coherence (more complex queries will be supported in a future release). However, there are instances where you may need to aggregate the data in Coherence based on input data from a stream. This blog describes a sample that does just that.

For our sample, we will use a simplified credit card fraud detection use case. The input to this sample application is a stream of credit card transaction data. The input stream contains information like the credit card ID, transaction time and transaction amount. The purpose of this application is to detect suspicious transactions and send out a warning event. For the sake of simplicity, we will assume that all transactions with amounts greater than $1000 are suspicious. The transaction history is available in a Coherence distributed cache. For every suspicious transaction detected, a warning event must be sent with maximum amount, total amount and total number of transactions over the past 30 days, as shown in the diagram below.

Application Input

Stream input to the EPN contains events of type CCTransactionEvent. This input has to be joined with the cache with all credit card transactions. The cache is configured in the EPN as shown below:

    <wlevs:caching-system id="CohCacheSystem" provider="coherence"/>
    <wlevs:cache id="CCTransactionsCache" value-type="CCTransactionEvent" 
                 key-properties="cardID, transactionTime"

Application Output

The output that must be produced by the application is a fraud warning event. This event is configured in the spring file as shown below. Source for cardHistory property can be seen here.
    <wlevs:event-type type-name="FraudWarningEvent">
          <wlevs:properties type="tuple">
              <wlevs:property name="cardID" type="CHAR"/>
              <wlevs:property name="transactionTime" type="BIGINT"/>
              <wlevs:property name="transactionAmount" type="DOUBLE"/>
              <wlevs:property name="cardHistory" type="OBJECT"/>

Cache Data Aggregation using Java Cartridge

In the output warning event, cardHistory property contains data from the cache aggregated over the past 30 days. To get this information, we use a java cartridge method. This method uses Coherence’s query API on credit card transactions cache to get the required information. Therefore, the java cartridge method requires a reference to the cache. This may be set up by configuring it in the spring context file as shown below:

    <bean class="">
        <property name="cache" ref="CCTransactionsCache"/>

This is used by the java class to set a static property:

    public void setCache(Map cache)
        s_cache = (NamedCache) cache;

The code snippet below shows how the total of all the transaction amounts in the past 30 days is computed. Rest of the information required by CardHistory object is calculated in a similar manner. Complete source of this class can be found here. To find out more information about using Coherence's API to query a cache, please refer Coherence Developer’s Guide.

public static CreditHistoryData execute(String cardID)
     Filter filter = QueryHelper.createFilter("cardID = :cardID and transactionTime > :transactionTime", map);
        CardHistoryData history = new CardHistoryData();
        Double sum = (Double) s_cache.aggregate(filter, new DoubleSum("getTransactionAmount"));
    return history;

The java cartridge method is used from CQL as seen below:

select cardID,
          CCTransactionsAggregator.execute(cardID) as cardHistory
from inputChannel
where transactionAmount>1000

This produces a warning event, with history data, for every credit card transaction over $1000.

That is all there is to it. The complete source for the sample application, along with the configuration files, is available here. In the sample, I use a simple java bean to load the cache with initial transaction history data. An input adapter is used to create and send transaction events for the input stream.

Thursday Feb 10, 2011

Extending Oracle CEP with Predictive Analytics


OCEP is often used as a business rules engine to execute a set of business logic rules via CQL statements, and take decisions based on the outcome of those rules. There are times where configuring rules manually is sufficient because an application needs to deal with only a small and well-defined set of static rules. However, in many situations customers don't want to pre-define such rules for two reasons. First, they are dealing with events with lots of columns and manually crafting such rules for each column or a set of columns and combinations thereof is almost impossible. Second, they are content with probabilistic outcomes and do not care about 100% precision. The former is the case when a user is dealing with data with high dimensionality, the latter when an application can live with "false" positives as they can be discarded after further inspection, say by a Human Task component in a Business Process Management software.

The primary goal of this blog post is to show how this can be achieved by combining OCEP with Oracle Data Mining® and leveraging the latter's rich set of algorithms and functionality to do predictive analytics in real time on streaming events. The secondary goal of this post is also to show how OCEP can be extended to invoke any arbitrary external computation in an RDBMS from within CEP. The extensible facility is known as the JDBC cartridge.

The rest of the post describes the steps required to achieve this:

We use the dataset available at to showcase the capabilities. We use it to show how transaction anomalies or fraud can be detected.

Building the model:

Follow the self-explanatory steps described at the above URL to build the model.  It is very simple - it uses built-in Oracle Data Mining PL/SQL packages to cleanse, normalize and build the model out of the dataset.  You can also use graphical Oracle Data Miner®  to build the models.

To summarize, it involves:

  • Specifying which algorithms to use. In this case we use Support Vector Machines as we're trying to find anomalies in highly dimensional dataset.
  • Build model on the data in the table for the algorithms specified.

For this example, the table was populated in the scott/tiger schema with appropriate privileges.

Configuring the Data Source:

This is the first step in building CEP application using such an integration.  Our datasource looks as follows in the server config file.  It is advisable that you use the Visualizer to add it to the running server dynamically, rather than manually edit the file.

            <test-table-name>SQL SELECT 1 from DUAL</test-table-name>


Designing the EPN:

The EPN is very simple in this example.

Thumbnail image for dm-epn.jpg

We briefly describe each of the components.

  • The adapter ("DataMiningAdapter") reads data from a .csv file and sends it to the CQL processor downstream. The event payload here is same as that of the table in the database (refer to the attached project or do a "desc table-name" from a SQL*PLUS prompt). While this is for convenience in this example, it need not be the case. One can still omit fields in the streaming events, and need not match all columns in the table on which the model was built. Better yet, it does not even need to have the same name as columns in the table, as long as you alias them in the USING clause of the mining function. (Caveat: they still need to draw values from a similar universe or domain, otherwise it constitutes incorrect usage of the model).
  • There are two things in the CQL processor ("DataMiningProc") that make scoring possible on streaming events.

1.      User defined cartridge function

Please refer to the OCEP CQL reference manual to find more details about how to define such functions. We include the function below in its entirety for illustration.

<?xml version="1.0" encoding="UTF-8"?>







              <function name="prediction2">           

                     <param name="CQLMONTH" type="char"/>

                     <param name="WEEKOFMONTH" type="int"/>

                     <param name="DAYOFWEEK" type="char" />

                     <param name="MAKE" type="char" />

                     <param name="ACCIDENTAREA"   type="char" />

                     <param name="DAYOFWEEKCLAIMED"  type="char" />

                     <param name="MONTHCLAIMED" type="char" />

                     <param name="WEEKOFMONTHCLAIMED" type="int" />

                     <param name="SEX" type="char" />

                     <param name="MARITALSTATUS"   type="char" />

                     <param name="AGE" type="int" />

                     <param name="FAULT" type="char" />

                     <param name="POLICYTYPE"   type="char" />

                     <param name="VEHICLECATEGORY"  type="char" />

                     <param name="VEHICLEPRICE" type="char" />

                     <param name="FRAUDFOUND" type="int" />

                     <param name="POLICYNUMBER" type="int" />

                     <param name="REPNUMBER" type="int" />

                     <param name="DEDUCTIBLE"   type="int" />

                     <param name="DRIVERRATING"  type="int" />

                     <param name="DAYSPOLICYACCIDENT"   type="char" />

                     <param name="DAYSPOLICYCLAIM" type="char" />

                     <param name="PASTNUMOFCLAIMS" type="char" />

                     <param name="AGEOFVEHICLES" type="char" />

                     <param name="AGEOFPOLICYHOLDER" type="char" />

                     <param name="POLICEREPORTFILED" type="char" />

                     <param name="WITNESSPRESNT" type="char" />

                     <param name="AGENTTYPE" type="char" />

                     <param name="NUMOFSUPP" type="char" />

                     <param name="ADDRCHGCLAIM"   type="char" />

                     <param name="NUMOFCARS" type="char" />

                     <param name="CQLYEAR" type="int" />

                     <param name="BASEPOLICY" type="char" />                      






              AS probability

            FROM (SELECT  :CQLMONTH AS MONTH,                 

                          :WEEKOFMONTH AS WEEKOFMONTH,

                          :DAYOFWEEK AS DAYOFWEEK,

                          :MAKE AS MAKE,

                          :ACCIDENTAREA AS ACCIDENTAREA,


                          :MONTHCLAIMED AS MONTHCLAIMED,


                          :SEX AS SEX,

                          :MARITALSTATUS AS MARITALSTATUS, 

                          :AGE AS AGE,

                          :FAULT AS FAULT,

                          :POLICYTYPE AS POLICYTYPE, 


                          :VEHICLEPRICE AS VEHICLEPRICE,

                          :FRAUDFOUND AS FRAUDFOUND,

                          :POLICYNUMBER AS POLICYNUMBER,

                          :REPNUMBER AS REPNUMBER,

                          :DEDUCTIBLE AS DEDUCTIBLE, 

                          :DRIVERRATING AS DRIVERRATING,




                          :AGEOFVEHICLES AS AGEOFVEHICLES,



                          :WITNESSPRESNT AS WITNESSPRESENT,

                          :AGENTTYPE AS AGENTTYPE,

                          :NUMOFSUPP AS NUMOFSUPP,

                          :ADDRCHGCLAIM AS ADDRCHGCLAIM, 

                          :NUMOFCARS AS NUMOFCARS,

                          :CQLYEAR AS YEAR,

                          :BASEPOLICY AS BASEPOLICY

                FROM dual)






2.      Invoking the function for each event.

Once this function is defined, you can invoke it from CQL as follows:

<?xml version="1.0" encoding="UTF-8"?>

<wlevs:config xmlns:wlevs="">





       <query id="q1"><![CDATA[

                    ISTREAM(SELECT S.CQLMONTH,


                                   S.DAYOFWEEK, S.MAKE,



                                   C.F AS probability


                                StreamDataChannel [NOW] AS S,




                                      S.MAKE, ...,

                                      S.BASEPOLICY) AS F of char) AS C)






  • Finally, the last stage in the EPN prints out the probability of the event being an anomaly. One can also define a threshold in CQL to filter out events that are normal, i.e., below a certain mark as defined by the analyst or designer.

Sample Runs:

Now let's see how this behaves when events are streamed through CEP. We use only two events for brevity, one normal and other one not.

This is one of the "normal" looking events and the probability of it being anomalous is less than 60%.

Event is: eventType=DataMiningOutEvent object=q1  time=2904821976256 S.CQLMONTH=Dec, S.WEEKOFMONTH=5, S.DAYOFWEEK=Wednesday, S.MAKE=Honda, S.ACCIDENTAREA=Urban, S.DAYOFWEEKCLAIMED=Tuesday, S.MONTHCLAIMED=Jan, S.WEEKOFMONTHCLAIMED=1, S.SEX=Female, S.MARITALSTATUS=Single, S.AGE=21, S.FAULT=Policy Holder, S.POLICYTYPE=Sport - Liability, S.VEHICLECATEGORY=Sport, S.VEHICLEPRICE=more than 69000, S.FRAUDFOUND=0, S.POLICYNUMBER=1, S.REPNUMBER=12, S.DEDUCTIBLE=300, S.DRIVERRATING=1, S.DAYSPOLICYACCIDENT=more than 30, S.DAYSPOLICYCLAIM=more than 30, S.PASTNUMOFCLAIMS=none, S.AGEOFVEHICLES=3 years, S.AGEOFPOLICYHOLDER=26 to 30, S.POLICEREPORTFILED=No, S.WITNESSPRESENT=No, S.AGENTTYPE=External, S.NUMOFSUPP=none, S.ADDRCHGCLAIM=1 year, S.NUMOFCARS=3 to 4, S.CQLYEAR=1994, S.BASEPOLICY=Liability, probability=.58931702982118561 isTotalOrderGuarantee=true\nAnamoly probability: .58931702982118561

However, the following event is scored as an anomaly with a very high probability of  89%. So there is likely to be something wrong with it. A close look reveals that the value of "deductible" field (10000) is not "normal". What exactly constitutes normal here?. If you run the query on the database to find ALL distinct values for the "deductible" field, it returns the following set: {300, 400, 500, 700}

Event is: eventType=DataMiningOutEvent object=q1  time=2598483773496 S.CQLMONTH=Dec, S.WEEKOFMONTH=5, S.DAYOFWEEK=Wednesday, S.MAKE=Honda, S.ACCIDENTAREA=Urban, S.DAYOFWEEKCLAIMED=Tuesday, S.MONTHCLAIMED=Jan, S.WEEKOFMONTHCLAIMED=1, S.SEX=Female, S.MARITALSTATUS=Single, S.AGE=21, S.FAULT=Policy Holder, S.POLICYTYPE=Sport - Liability, S.VEHICLECATEGORY=Sport, S.VEHICLEPRICE=more than 69000, S.FRAUDFOUND=0, S.POLICYNUMBER=1, S.REPNUMBER=12, S.DEDUCTIBLE=10000, S.DRIVERRATING=1, S.DAYSPOLICYACCIDENT=more than 30, S.DAYSPOLICYCLAIM=more than 30, S.PASTNUMOFCLAIMS=none, S.AGEOFVEHICLES=3 years, S.AGEOFPOLICYHOLDER=26 to 30, S.POLICEREPORTFILED=No, S.WITNESSPRESENT=No, S.AGENTTYPE=External, S.NUMOFSUPP=none, S.ADDRCHGCLAIM=1 year, S.NUMOFCARS=3 to 4, S.CQLYEAR=1994, S.BASEPOLICY=Liability, probability=.89171554529576691 isTotalOrderGuarantee=true\nAnamoly probability: .89171554529576691


By way of this example, we show:

  • real-time scoring of events as they flow through CEP leveraging Oracle Data Mining.
  • how CEP applications can invoke complex arbitrary external computations (function shipping) in an RDBMS.
[Read More]

Thursday Jan 27, 2011

Using EclipseLink from OCEP

[Read More]

Thursday Sep 16, 2010

Oracle CEP Events at OpenWorld 2010

[Read More]

Wednesday Sep 08, 2010

Generating Complex Events from a Partitioned Stream

[Read More]

Tuesday Aug 10, 2010

How to Programmatically Access Oracle CEP MBeans

[Read More]

Wednesday Jul 28, 2010

Oracle CEP: Enriching the Results of CQL Aggregation Queries

[Read More]

Monday Jul 26, 2010

Oracle CEP Applications and CQL Aggregation

[Read More]

Wednesday May 26, 2010

Oracle CEP on OTN

[Read More]

Wednesday May 12, 2010

Using Open MQ as an Oracle CEP Event Source

[Read More]

Tuesday Oct 13, 2009

Oracle CEP at Oracle OpenWorld 2009

[Read More]

Wednesday Jul 08, 2009

Oracle CEP and Twitter

[Read More]

This blog contains information about Oracle Event Processing (OEP)


« March 2015