NoSQL : Key-Value stores and range based operations

In the NoSQL database space, the key-value store (as opposed to Document, Graph, Object, XML stores) has the strongest reputation for being able to scale out well, providing high throughput writes along with highly concurrent reads.  That is possible because of the power in developing the database architecture over a simplistic data model where the key is easily used to hash into a growing number of logical groups (physical nodes).

However, in some cases, that simplistic key-value model could tie your hands when you want to provide a use case that needs a data access pattern involving a range or collections (especially if large) of data.  It turns out, this is a fairly common use case pattern showing up in areas of analysis involving time-series, sensor data aggregation, activity grouping, and a whole bunch of use cases where you've got logically related, nested data models.  

So, one of the key questions to ask is whether or not your key-value store is providing facilities to help in the implementation of those types of use cases.  Further, if there is anything in the design / implementation of the store which optimizes the system resource usage to accommodate those access patterns.  For example, it can be all well and good if your store offers a getKeys(start, stop) type of API, but if the underlying implementation of the store causes a disk seek for each key value in the range between stop-start then your blazingly fast key value store is about to get really slow.  

Of course, a secondary index overlay can also be used to optimize range and aggregate access patterns.  However, I am sure you will recollect, secondary indexes are what the relational database uses to optimize this kind of access and while they can help on the read side, they come with the unwanted side effect of slowing down insertions and updates and are at the heart of some of their inherent scaling issues.  In the end, you want to minimize your use of those indexes otherwise the aggregate ability to perform and scale is diminished.   Also very important for developers to realize, the secondary index support for nearly all of the NoSQL databases are not consistent with data operations, because most of those NoSQL databases do not support transactions, providing no guarantee for the update of the index and the data.  So, you end up with divergence between your index and data and that means over time your queries start returning incorrect results.     

The Oracle NoSQL Database (ONDB) was designed with these range and aggregate use case access patterns in mind.  At the heart of the distribution model is the notion of a major and minor portion to the key, combined representing the unique key value. Within ONDB, those concepts of major-minor play a role in the distribution and physical storage of the data.  The major portion of the key will provide locality from a networking perspective and the minor key portion provides further locality from a disk perspective.  This major-minor key data designation for distinct values, will cause locality of storage that can be leveraged to optimize both network and disk access for range and aggregate based operations. 

An example to help conceptualize is a sensor network for a smart building complex.  The sensors in such a complex have a hierarchical structure the e.g. Complex>>Subdivisions>>Buildings>>Floors>>Offices.  A building complex will have 100,000's of sensors (aircon,lights,motion,entry,exit,smoke,gas,flood,etc) and their data can become very large with samples coming in every second (even milliseconds). Monitoring of the building complex will involve readings in aggregation from different levels (e.g. aircon per floor). Sensors can then be modeled using these hierarchies as keys and individual sensors within specific areas will then be able to be accessed efficiently because all of the sensors for a particular building's floor will be in the same logical group which means in the same (or few) network location in ONDB's data distribution.  Further, a given sensors minor keys will hold data samples from the sensor in the same physical disk location. So, operations involving sensor monitoring over a particular time interval can be obtained with a single network hop and a sequential disk read, providing the optimization of system resources on data access.  

In addition, lots of operations can be performed within a major key in a completely transactional manner, either all the changes happening or none of them (a full rollback), so you can also take advantage of localized consistency groups. However, this is the topic of another discussion.

For those who want to get a little more practical, on the read side, here is a great blog created that talks about how the capabilities discussed in this blog show up in the API and what the behavior looks like when storing and retrieving range/aggregate data sets.    


Post a Comment:
  • HTML Syntax: NOT allowed

This blog is about everything NoSQL. An open place to express thoughts on this exciting topic and exchange ideas with other enthusiasts learning and exploring about what the coming generation of data management will look like in the face of social digital modernization. A collective dialog to invigorate the imagination and drive innovation straight into the heart of our efforts to better our existence thru technological excellence.


« July 2016