Role and Tool for Real Time Processing in IoT
By shupadhy on Nov 22, 2013
Everyone acknowledges that IoT will improve our productivity by automating mundane tasks like getting relevant information from a machine to a Cloud Service for Repair or Replenishment. Large numbers of folks also agree that Big Data has an important role to play. For example, crunching data from hundreds of sensors in a turbine and predicting failures. However, relatively less number of folks understand the role of Real Time processing in Internet of Things. After all if I have the ability to reduce my usage based automobile insurance premium, I would appreciate if instead of sending a monthly report I can get an alert to change my behavior while I am still in the middle of my long drive.
If Real Time Processing of data was difficult in simple transactional systems, consider the complexity when you are dealing with streaming data plus you have to combine multiple data sources like information of my trip and the data coming in constantly from the Telematics module in my car. So you have to learn a new tool. While you might be familiar with Hadoop, you should familiarize yourself with Storm. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.
Oracle's solution for building application to process streaming data in real time is Oracle Event Processing. Oracle also offers a lighter weight version of OEP for Embedded Development on ARM and x86 devices. Such devices include Home Automation Gateways, Smart Meter Aggregators, Healthcare Hubs, Industrial Gateways. Yes, if you can take decision without needing any more data from the backend, then why not do it at the Gateway/Aggregator level and not send all the streaming data back to the data center. You might still need to send the data back for other use cases, but at least your real time decision making latency is reduced considerably.
So what are the strengths of OEP. Here are some key points -
1) OEP provides infrastructure and guidelines for developers to create scalable and highly available Event processing networks (EPN).
2) OEP provides out of the box Adapters for easily consuming data coming in different protocols and formats. Of course you are free to write your own adaptors.
3) Oracle CQL is a rich formalized Query Language optimized for concise expression of temporal queries similar to how SQL is optimized for expression of static data queries. Out of the box functions augment the core language.
4) Oracle CQL is augmented by data cartridge framework allowing you to tightly integrate other domain data types and functions. Following Data Cartridges are available – Oracle Java, Oracle Spatial, Oracle JDBC, Hadoop. Oracle JDBC cartridge allows you to write applications merging streaming data with geo fencing functions available from Oracle Spatial or predictive analytics capabilities of Oracle Advanced Analytics.
5) Pre integration with Oracle Coherence for local and distributed cache. You can integrate a cache system with your Oracle Event Processing application so that the cache is available as source or destination for data your application uses, including event data. Integrating a cache can provide access to relatively static data at a speed that is well suited to an application that handles streaming data.
6) Persist the events that flow out of a component of the EPN to a store and then play them back at a later stage. This helps debug a problem with a currently running application. If you have been recording the events at a node in the EPN when the problem occurred, you can later playback the same list of events to recreate the problem scenario for debugging purposes.
7) OEP Visualizer is a rich Web 2.0 application that displays OEP Data in a useful and intuitive way for admins e.g. latency and throughput graphs display the amount of time it takes an event to pass through the specified stage or path in the EPN or the number of events passing through, respectively. EPN Configurations can be changed through the Visualizer. All this can also be done programmatically through available JMX APIs.
8) OEP architecture is highly modular based on OSGi. Adapters and other components involved in implementing data feeds can be dynamically configured/activated/suspended via a lifecycle API (SuspendableBean, ResumableBean). Changes can be made dynamically to CQL queries. In business implementations, business rules are very dynamic. This also limits testing required for changes.
9) OEP integrates well with Oracle SOA. It is available as a part of Oracle SOA Suite and can be used in SOA Composites.
So explore the role of real time processing in your IoT application with OEP and share with us what cool application you came up with. Fast Data enabled by OEP brings more value to high velocity sensor/device data. This enables new services, better customer experience, improved efficiency, higher quality in operations.