Upon returning from vacation this week, I noticed that an interesting discussion had taken place on The Server Side about the use of Twitter as a source (and sink) of events for CEP engines/platforms. A lot of the discussion focused on the merits (and morality) of using Twitter as a source of news for algorithmic trading applications. The level of interest in querying Twitter data streams using CEP technology made me think that it would be worth mentioning how this can be done using the Oracle CEP product.
As you may know if you are reading this blog, Oracle CEP is an all-Java platform for creating event-driven applications. One of Oracle CEP's major strengths is that it provides a straightforward set of API's for writing the Java components that make up an event processing application. In this case, we are interested in a particular type of component called an "adapter". Adapters are responsible for connecting to external event sources and feeding events to the rest of the application and also for sending events generated by the application to external clients. Twitter is an example of both an external event source and an external client that can receive events, so creating an Oracle CEP application that works with Twitter basically requires writing a "Twitter adapter".
I actually wrote a sample adapter several months ago using the Twitter4j open source library to simplify the adapter implementation. For example, when using Twitter4j the adapter can retrieve data from Twitter using a single line of code:
List<Status> statuses = twitter.getPublicTimeline(lastTweet);
It doesn't get much easier than that! Once the adapter has retrieved the data from Twitter it converts the data to an Oracle CEP event and then sends the event to any of its event listeners in the usual way:
streamSender.sendInsertEvent(statusEvent);
Hopefully, this short example has provided a flavor for the simplicity with which Twitter can be accessed from an Oracle CEP application. I welcome any comments that people have on this topic, especially suggestions for new features that would enhance Oracle CEP's support for Twitter.
Comments (5)
Hi,
nice post. This is exactly what I'm starting to do at the moment. I built an adapter as well and now I send the tweets into Oracle CEP.
Next step is to aggregate tweets by certain criteria (with CQL) and then apply text mining (clustering) techniques on the aggregated tweets, which themselves are a new event stream.
Before clustering data cleansing (stemming, stop words,etc) is applied as well.
And maybe here a proposal for a new feature arises:The possibilites of integrating modules within the visualizer that are not directly concerend with stream processing but are necessary for event modification/transformation.
I can do this in the code of course by calling the "transformation class" from a StreamSink class. But it would be great to inject event modification into the stream and make this perceptible in the visualizer, e.g. I create a "TransformEvent" class or "NormalizeEvent"class and inject this into several EventBeans.
As I just started with the new Oracle CEP suite maybe this feature is already existing and I just missed it up to now.
In general it would be great to integrate some more additional workflow features into the visualizer (e.g. BPEL/BPMN).
Cheers,
Andreas
Posted by Andreas | July 9, 2009 1:27 AM
Posted on July 9, 2009 01:27
Hi Andreas,
Sounds like you are doing some cool stuff. Thanks for the suggestion. It is possible (using Spring) to inject a reference to a "transform" or "normalize" component into a custom event bean (an event bean that you write). Is that what you were thinking?
Regards,
Seth
Posted by Seth White | July 9, 2009 3:42 PM
Posted on July 9, 2009 15:42
Hi,Andreas
It is nice you work with Visualizer and rasise some cool proposals.
It seems you try to update the enp dynamically. You try to insert a customized processor between the adapter and a streamsink, the responsibility of this processor is to midify/transform events.
Does it make sense?
Thanks
Tony
Posted by Tony | July 10, 2009 8:37 PM
Posted on July 10, 2009 20:37
Thanks for your comments
@Seth: The hint with Spring is good. As far as I can see Spring is used in lots of places withn the CEP suite. So it's no problem to insert this within the code. What I was thinking about, was a way to visualize this, e.g. right click in the epn viewer and say "Inject custom class". So I can see that there is something happening.
@Tony: Thanks for your hint. In my understanding you use processors to, e.g. create a new stream from an existing stream (Istream). I thought of something that needs more computational effort, e.g. transform attributes of a stream into a n-dimensional array, add this to a event class as an attribute and send this event into the stream. And this computing class I would like to visualize, so I can see this in the epn.
Posted by Andreas | July 14, 2009 1:04 AM
Posted on July 14, 2009 01:04
Andreas,
Thanks for the suggestion. I have forwarded your idea to our IDE developers.
Regards,
Seth
Posted by Seth White | July 14, 2009 1:46 PM
Posted on July 14, 2009 13:46