X

@OracleIMC Partner Resources & Training: Discover your Modernization options + Reach new potential through Innovation

Exploring data streams with Oracle Stream Explorer - part 1

Webcast: Fast Data-Turning High-Velocity Data into ValueOracle recently released Oracle Stream Explorer, an add-on for Oracle Complex Event Processing which simplifies rapid creation of event-driven
applications for any type of Real Time business solution, enabling the
Enterprise to now really immerse itself in next generation real time
applications, with times to market of minutes rather than days or weeks.

In this blog I'll describe how it can be used to analyse twitter data on the fly.

Getting the twitter stream

Twitter provides a few different options to get live streams. Firehose - the one which provides all data in real time - is a commercial product. Another option is filtered stream, which does not guarantee delivery of all twits, but is sufficient for our demo.

I will not get into details how the twitter part is implemented. Suffice it to say we listen to two keywords - #mobile and #cloud in my example - and as they come to the stream, we push it to the stream explorer over a rest service. You can download the source here, change the twitter keys/tokens/secrets to your twitter app's values and try it out.

Starting the Stream Explorer

Stream explorer is distributed as a patch for Oracle Complex event processing. There are some blogs out there on how to install and run it. Let me just add that after you start the service, you need to wait a bit before /sx url is available. If you get 404 it might be that Stream Explorer is still starting. If it's still not there in a minute or so, you probably have a real issue though.

Creating your first stream

First we need to create a stream. We'll ignore Stream Explorer quick starts and go straight to catalog (top right), and there we create new Stream item.

Then we'll configure the stream in 3 easy steps:

We'll call our stream "twitter" and configure source type as REST. We don't want to create default exploration yet, so you can uncheck that.

Then we provide context path for our REST service. What Stream explorer does under the hood is really cool. It will create REST service accessible at e.g. http://localhost:9002/twitter, no coding required. But we need to describe the service and we do that in step 3.

Here we provide fields and types accepted by our rest service. We'll collect the whole twit text, keyword which triggered the event, twitter user and so on. 

After clicking create, you can post data to the service using postman or similar tool. Here is sample json payload accepted by the service:

{"text": "Testing 123","keyword": "#test","user": "tester","retweetCount": 0,"favoriteCount": 0}

Now you should be able to start the TwitterSream app and send data to the Stream Explorer - just check URLs and your twitter tokens. 

We'll explore streams in part 2 of this post.


Jernej Kaše is a Fusion Middleware Specialist working closely with Oracle Partners
in the ECEMEA region to grow their business by leveraging Oracle technology.
Facebook Google+ Twitter LinkedIn Pinterest Delicious Digg Addthis

Join the discussion

Comments ( 5 )
  • guest Thursday, April 2, 2015

    Excellent article! Just to be clear, support for REST is not any REST service, but just the REST interface exposed by Oracle Event Processing (OEP), correct? Thus, it's up to us to feed that OEP REST interface (as you have done in your example here.


  • guest Thursday, April 16, 2015

    It goes both ways. When we configure a stream, it can receive REST messages and will in fact create REST service to which clients can send events.

    Explorations can also also invoke outbound rest services, meaning sending (filtered and processed data) to a REST service. Just make a REST service with object which has the same names of members than the exploration, and it should work. Maybe I'll describe this in part 3.


  • guest Wednesday, August 5, 2015

    HiJernej,

    Nice Article. Thanks for sharing such post.For POC I am trying to upload JSON file as input for the Stream.

    So is it like I have to create a java class and need to write one method which will read the JSON file and we need to expose that method as REST web service and in Context path we have to give the url of the webservice?

    And is it require to deploy that JAVA application to OEP server?


  • Jernej Thursday, August 27, 2015

    If you want to post data to a stream, you 1st need to create REST type stream as described in this article. Note that this will create a REST endpoint which expects JSON according to the configured rest shape. So in case your existing JSON maps 1-to-1 to the payload expected by the REST stream, you have nothing more to do than calling the REST with JSON payload. In case however that your existing payload differs, you firs need to transform it before feeding it to the stream.

    Actually this example receives twitter feeds in twitter JSON format and transposes them (using twitter4j) to a much simpler format. The only difference is that twitter4j hides original JSON complexity and provides java classes to work with incoming twitter JSON data.

    Regarding the 2nd question, yes, OEP and StremExplorer do require Java.


  • guest Monday, November 2, 2015

    I installed SOA + OEP

    know how to Execute OEP in the browser?


Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.