Thursday Apr 28, 2016

Submit your OOW 2016 Proposals

If you have a great story or solution that includes Oracle NoSQL Database and want to talk about it, please submit at: 

https://www.oracle.com/openworld/call-for-proposals.html

Closes on May 9, 2016

  • Conference location: San Francisco, California, USA
  • Date: Sunday, September 18 to Thursday, September 22, 2016

Friday Apr 22, 2016

Oracle Open World 2016 - Call for Proposals

The call for proposals for Oracle Open World 2016 is open.  Please submit your proposals by May 9, 2016.  The proposal web site is:

 https://www.oracle.com/openworld/call-for-proposals.html

 Submit if you have an interesting use of Oracle NoSQL Database to talk about. 

Wednesday Mar 16, 2016

Oracle NoSQL Database Cluster YCSB Testing with Fusion ioMemory™ Storage

Highly distributed systems with large data stores in the form of NoSQL databases are becoming increasingly important to enterprises, not just to hyperscale organizations. NoSQL databases are being deployed for capturing patient sensors data in health care, smart meter analysis in utilities, customer sentiment analysis in retail, and various other use cases in different industries. NoSQL database systems help organizations store, manage, and analyze huge amounts of data on distributed system architecture. The sheer volume of data and the distributed system design needed to manage this large data at a reasonable cost necessitated a different category of database systems, leading to NoSQL databases. Oracle NoSQL Database is part of the NoSQL database family and is based on a distributed, key-value architecture.

This technical white paper describes a three-node Oracle NoSQL Database Cluster deployment procedure on Fusion ioMemory™ storage. The following points are emphasized:

  • Highlights performance and scalability advantages compared to traditional spinning disks.
  • Because enterprises evaluate and assess new technologies for enterprise-wide adaptability, Yahoo Cloud Serving Benchmark (YCSB) is the standard benchmark tool employed for testing and is the same tool used in this paper to evaluate Oracle NoSQL Database for YCSB Benchmark Testing.
  • Analysis and discussion are provided for throughput and latency testing results with YCSB.

 

Download now at:   https://www.sandisk.com/content/dam/sandisk-main/en_us/assets/resources/enterprise/white-papers/oracle-nosql-cluster-ycsb-testing-with-fusion-iomemory.pdf

Tuesday Mar 01, 2016

Support for Oracle NoSQL Database CE available

For those of you who have downloaded  Oracle NoSQL Database Community Edition  (CE) , did you know that support in two forms is available for this ?  Some customers may think that by downloading the CE Version that they are left on their own.  This is not true.  Questions and answers can be posted to:  

https://community.oracle.com/community/database/high_availability/nosql_database   where lots of users monitor the questions and respond to these inquiries. 

In addition, support from Oracle is available to anyone.  Go to:  https://shop.oracle.com/pls/ostore/f?p=DSTORE:PRODUCT:::NO:RP,6:P6_LPI:124789930244771286351865     where details are available.

Thanks to all who use Oracle NoSQL Database and continue to create innovative applications and use the family of Oracle products. 

Tuesday Feb 23, 2016

Oracle NoSQL Database Sessions at All India Oracle Users Group (AIOUG)

Come listen to two talks by members of the Oracle NoSQL team at the AIOUG event in Hyderabad, India on February 27, 2016. Ashutosh Naik and Anand Chandak will be presenting on how to set up an Oracle NoSQL Database environment and will discuss innovative customers and how they are taking advantage of the Oracle NoSQL Database. Details can be found below.  The full announcement is at:

https://blogs.oracle.com/NoSQL/resource/HydBigdataDay.pdf

Ganga Conference Hall

Tech Mahindra Learning world (Just Adjacent to Oracle India Office- Hi Tech City)

TechMahindra Infocity Campus
Hi-Tech City

 

Monday Feb 15, 2016

New Oracle NoSQL Database Whitepaper on Bulk Put and Bulk Get

A new whitepaper is available, describing the new Bulk Get and Bulk Put API's in the Oracle NoSQL Database product.  Download from our Oracle NoSQL Database page, under Whitepapers.

http://www.oracle.com/technetwork/database/database-technologies/nosqldb/overview/index.html

Monday Feb 08, 2016

SQL Developer 4.1 now supports Oracle NoSQL

SQL Developer now supports Oracle NoSQL database. Support includes Developer and DBA roles, KVLite Store deployment and seeding, Store Reporting, Store Statistics and Administration.  Learn more at the Oracle NoSQL main page. Click Here.  (Additional Resources)

Monday Feb 01, 2016

Oracle NoSQL BulkPut

Introduction

Our customers have often asked us “what’s the fastest and most efficient way to insert large number of records in Oracle NoSQL database? “ Very recently, a shipping company reached out to us with specific requirement of using Oracle NoSQL database for their ship management application,which is used to track the movements of their container ships that moves the cargo from port to port. The cargo ships are all fitted with GPS and other tracking devices, which relays ship's location after few seconds into the application . The application is then queried for  1) The location of all the ships displayed on the map 2) A specific ship's trajectory over a given period of time displayed on the map too.As the volume of the location data started growing, company is finding hard to scale the application and is now looking at a back-end system that can ingest this large data-set very efficiently.

Historically, we have supported the option to execute a batch of operations for records that share the same shard key, which is what our large airline customer (Airbus) has done. They pre-sort the data by the shard key and then perform a multi-record insert when the shard key changes. Basically, rather than sending and storing a record at a time they can send a large number of records in a single operation. This certainly saved network trips, but they could only batch insert records that shared same shard key. With Oracle NoSQL Database release 3.5.2, we have added the ability to do a bulk insert or a bulk put records across different shards,allowing application developers to work more effectively with very large data-sets.

The BulkPut API is available for table as well as the key/Value data model. The API provides significant performance gains over single row inserts by reducing the network traffic round trips as well as by doing ordered inserts in batch on internally sorted data across different shards in parallel. This feature is release in a controlled fashion, so there aren’t java docs available for this API with this release, but we encourage you to use it and give us feedback.

API

KV interface: loads Key/Value pairs supplied by special purpose streams into the store.

public void put(List<EntryStream<KeyValue>> streams, BulkWriteOptions bulkWriteOptions)

Table interface: loads rows supplied by special purpose streams into the store.

public void put(List<EntryStream<Row>> streams, BulkWriteOptions bulkWriteOptions)
streams the streams that supply the rows to be inserted.
bulkWriteOptions non-default arguments controlling the behavior the bulk write operations

Stream Interface :

public interface EntryStream<E> {
String name();
E getNext();
void completed();
void keyExists(E entry);
void catchException(RuntimeException exception, E entry);
}


Performance

We ran the YCSB benchmark with the new Bulk-Put API on 3x3 (3 shards each with 3 copies of data) NoSQL Cluster running on bare metal servers, ingesting 50M records per shard or 150M records across the datastore, using 3 parallel thread per shard or total 9 ( 3x3) for the store and 6 parallel input streams per SN or total 54 ( 6 *9) across the store. The results for the benchmark run are shown in the graph below


The above graph compares the throughput (ops/sec) of Bulk vs Simple Put API with NoSQL store having 1000 partitions with durability settings of None and Simple Majority. As seen from the above charts there is over a 100% increase in throughput with either durability settings.

Sample Example

Here's link program uploaded to the github repository, the sample demonstrate how to use the BulkPut API in your application code. refer to the readme file for details related to the program execution.

Summary

If you are looking at bulk loading data into Oracle NoSQL Database the latest Bulk Put API provides the most efficient and fastest (as demonstrated by the YCSB) way to ingest large amount of data. Check it out now and download the latest version of the Oracle NoSQL Database at: www.oracle.com/nosql.

I'd like to thanks my colleague Jin Zhao for inputs on the performance numbers.


Monday Jan 04, 2016

Oracle NoSQL BulkGet API

Recently, we have been getting queries from our developer community and customers who wanted to know what is the most efficient way to retrieve large amounts of data in a single operation using Oracle NoSQL Database. An example of such a request would be on an eCommerce website, where potential customers want to retrieve all the phones in the price range $ 200 to $ 500 from Apple, Samsung, Nokia, Motorola (for example) and a host of other manufacturers to return all the details including the images of the product.

[Read More]

Tuesday Nov 17, 2015

Turn Big Data into Fast Data Using Oracle NoSQL and Fusion ioMemory

Prasad Venkatachar at SanDisk has been working with the Oracle NoSQL Database and with Fusion ioMemory to see how using flash affects the performance of various NoSQL workloads.  The Fusion ioMemory application accelerators deliver extreme performance when running a variety of workloads. Specifically for the Oracle NoSQL Database, the results show an amazing increase in performance compared to hard disk drives. The YCSB benchmark was run using different configurations adn the results are explained for a Write heavy workload (50 % write / 50 % read), a Read heavy workload (5 % write / 95 % read) and a read only workload (100 % read).  The latencies remained constant over larger workloads as did the performance, even as the amount of data grew. Read more details about using Fusion ioMemory by clicking here

Friday Nov 06, 2015

Oracle NoSQL Database demonstrates record performance with SPARC M7

Oracle's SPARC T7-4 server containing the SPARC M7 processor delivered 1.9 million ops/sec on 1.6 billion records for the Yahoo Cloud Serving Benchmark (YCSB) 95% read/5% update workload. Oracle NoSQL Database was used in these tests. NoSQL is important for Big Data Analysis and for Cloud Computing. The details can be found at the Performance blog.

Friday Oct 30, 2015

Oracle NoSQL Database Mentioned in Press Release

See the Oracle press release from October, 27, 2015, regarding the Oracle Cloud Platform for Big Data.  Read more here.

Friday Oct 09, 2015

Announcement - Oracle NoSQL Database 12.1.3.4.7

Oracle announced Oracle NoSQL Database 3.4.7 on Oct, 2015. This release offers new features including new commands to perform the Failover and SwitchOver operation when there's a loss of quorum, Bulk Get API to take input a list of primary keys and return all the records matching those keys, Off Heap Cache to allow users to allocate and use additional memory outside the limits of the Java heap, reducing the impact of Java garbage collection and support for the Big Data SQL and Apache Hive integration. Please read the full annoucments http://bit.ly/1qBUMuP. To download visit our download (http://bit.ly/1mpL2f9) page for the latest releaseless

Check the release presentation here

Thursday Aug 20, 2015

Invoking OracleNoSQL based Java application from PL/SQL

Recently, we ran into an interesting use-case with one of large supermarket customers, who wanted to take the output of a PL/SQL process and store that in  Oracle NoSQL Database to be be later consumed by one of their retail application - very quickly and something that can scale very well to support high volume of data that they are expected.  Oracle NoSQL DB is the obvious choice because it can provide a high throughput, low latency read/write operation and can scale to support large volume of data.

Coming to the integration, one of the highlights of the OracleN SQL Database is that it integrates really very well with other Oracle Tech Stack. The simplest way to write to Oracle NoSQL DB from a PL/SQL procedure is to call a  Java procedure that uses the native NoSQL DB API in order to insert data into the database and the simplest way to read from Oracle NoSQL DB in a stored procedure is to use an External Table in the query so that data from Oracle NoSQL DB can be passed to the Oracle Database query processor. There's another possible option to  use Golden Gate to move data from the Oracle Database to NoSQL DB. We have already blogged about the GoldenGate Integration, so in this blog I am going to focus on the Java Stored procedure based approach.

In case if you are not familiar with  Java Stored Procedure : A Java stored procedure essentially contains Java public static methods that are published to PL/SQL and stored in an Oracle database for general use. This allows a Java stored procedure to be executed from an application as if it were a PL/SQL stored procedure. When called by client applications, a Java stored procedure can accept arguments, reference Java classes, and return Java result values.

So, to help our customer, we created a POC that showcases this integration. Am listing down steps involved in this integration 

  1. First, create a NoSQL DB tables  that would store the data from Oracle Database
  2. Create a Java Application using the native NoSQL Driver to perform CRUD operation on NoSQL DB.
  3. Load the Java Application classes that we created in Step#2 in the oracle database using the load-java utility.
  4. Create a Java Store stored procedure that takes the data from the PL/SQL and updates NoSQL Database
  5. Next, publish Java stored procedures in the Oracle data dictionary. To do that, you write call specs, which map Java method names, parameter types, and return types to their SQL counterparts.
  6. Finally, call the Java store procedure from the PL/SQL Block to perform the updates.

The POC is available for download in a zip file from our OTN page (refer: The PL/SQL Integration in the Demo/Sample Program). The READ-ME file bundled in the zip has all the detailed steps and files needed for this integration.

With this approach, the NoSQL access is transparent to the Oracle DB application . NoSQL DB is an excellent choice here and using this Java Stored Procedure approach, the customer can exploit the advantages of BOTH repositories effectively and with better TCO.



Wednesday Aug 12, 2015

Migrating/Importing MongoDB Documents into Nosql Tables

Summary 

This paper presents a how to to migrate documents in MongoDB's collections into tables and child tables in Oracle Nosql. The idea is to take as example a relatively complex document, define a mapping file to map the basic fields of the document into a table,  and to map the embedded collections of the document into child tables. The java class that we provide will generate the Nosql structure of the tables and insert the data. The set of components of each element of the collection is inserted in the same operation into the store.

A Json example 

 Let's use an example of a family item from a MongoDB collection:

{ "_id" : ObjectId("55c4c6576e4ae64b5997d39e"),

"firstname" : "lena",

"lastname" : "clark",

"gender" : "W",

"childrens" : [

{ "name" : "bob",

"schools" : [ {"street" : "90, pine street","id" : "Saint volume"},

{"street" : "134, mice street","id" : "Saint appearance"}

],

"hobbies" : ["soccer","photo"]

},

{ "name" : "joseph",

"schools" : [ {"street" : "168, merely street","id" : "Saint slipped"} ],

"hobbies" : ["tennis","piano"]

},

{ "name" : "sandy",

"schools" : [{"street" : "227, thread street","id" : "Saint discovery"}],

"hobbies" : ["football","guitar"]

}

]

}

In this case the main document has the the following fields : '_id', 'firstname', 'lastname', 'gender' and childrens'. 'childrens' is an embedded collection, containing 'name', 'schools' and 'hobbies'. 'schools' is again a nested collection with 'street and 'id' fields and 'hobbies' is a list. We can map them into several nested tables:

  • the main table represents FAMILY,
  • FAMILY.CHILDREN  gets 'childrens' items and
  • FAMILY.CHILDREN.SCHOOLS and FAMILY.CHILDREN.HOBBIES store schools and hobbies information.

The mapping file 

The mapping file, is a properties file, it contains also connect information to access MongoDB database and Nosql store:

  • the name of the Nosql store: Nosql.Store=kvstore
  • the host and port of the nosql store: Nosql.URL=bigdatalite:5000
  • the mongodb host: MongoDB.host=localhost
  • the mongodb port: MongoDB.port=27017
  • the mongodb database: MongoDB.DB=gadb

Mapping principles

Define the main collection, its fields and its main table mapping

For each field define its type and its mapping value. Note that this can be a recursive step.

For each table define the primary key index components. 

Mapping extracts

Mapping collection and table with its primary keys

  • mongo.collection=family
  • mongo.collection.map=FAMILY
  • FAMILY.indexcols=LASTNAME,FIRSTNAME
indexcols is the keyword to introduce the comma separated list of columns of the key, order is important. The indexcol prefix is a Nosql table name

Family fields

  • family.fields=lastname,firstname,gender,childrens
  • family.firstname.type=string
  • family.firstname.map=FIRSTNAME
  • family.childrens.type=collection
  • family.childrens.map=CHILDREN
fields is the keyword to introduce the comma separated list of fields of a collection. For each field type corresponds to the type of a column in a Nosql table (string, integer, long, float, double or boolean are accepted). Two other values are used: array or collection. array is for lists of basic types, collection is for more complex collections. When  type is a basic type, map indicates a column of the mapped table, when the type is array or collection, map introduces a new table.

Children mappings

  • CHILDREN.indexcols=NAME
  • childrens.fields=name,schools,hobbies
  • childrens.name.type=string
  • childrens.name.map=NAME
  • childrens.schools.type=collection
  • childrens.schools.map=SCHOOLS
  • childrens.hobbies.type=array
  • childrens.hobbies.map=HOBBIES

School mappings 

  • schools.fields=street,id
  • schools.indexcols=ID
street and id are basic string fields, their type and map are not shown.

Hobbies mappings

  • hobbies.fields=hobbies
  • hobbies.hobbies.type=string
  • hobbies.hobbies.map=HOBBY
  • HOBBIES.indexcols=HOBBY

children.hobbies is an array of strings mapped to child table HOBBIES, there is no name in the main collection for the field, I've chosen to use hobbies (the name of the collection) as the field name to be able to define a mapping. 

Tables generated

Get child tables from FAMILY   

kv-> show tables -parent FAMILY

Tables: 

FAMILY.CHILDREN

 FAMILY.CHILDREN.HOBBIES

 FAMILY.CHILDREN.SCHOOLS

Get table indexes

kv-> show indexes -table FAMILY

Indexes on table FAMILY

FAMILYIndex (LASTNAME, FIRSTNAME)

kv-> show indexes -table FAMILY.CHILDREN

Indexes on table FAMILY.CHILDREN

CHILDRENIndex (NAME)

kv-> show indexes -table FAMILY.CHILDREN.SCHOOLS

Indexes on table FAMILY.CHILDREN.SCHOOLS

SCHOOLSIndex (ID)

kv-> show indexes -table FAMILY.CHILDREN.HOBBIES

Indexes on table FAMILY.CHILDREN.HOBBIES

HOBBIESIndex (HOBBY) 

Getting data from tables

Get our example family

kv-> get table -name FAMILY -field LASTNAME -value "clark" -field FIRSTNAME -value "lena"

{"FIRSTNAME":"lena","LASTNAME":"clark","GENDER":"W"}

Get our family children

kv-> get table -name FAMILY.CHILDREN -field LASTNAME -value "clark" -field FIRSTNAME -value "lena"

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"bob"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"joseph"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"sandy"}

Get our family children schools

kv-> get table -name FAMILY.CHILDREN.SCHOOLS -field LASTNAME -value "clark" -field FIRSTNAME -value "lena"

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"bob","STREET":"134, mice street","ID":"Saint appearance"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"bob","STREET":"90, pine street","ID":"Saint volume"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"joseph","STREET":"168, merely street","ID":"Saint slipped"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"sandy","STREET":"227, thread street","ID":"Saint discovery"} 

Get our family children hoobies

kv-> get table -name FAMILY.CHILDREN.HOBBIES -field LASTNAME -value "clark" -field FIRSTNAME -value "lena"

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"bob","HOBBY":"photo"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"bob","HOBBY":"soccer"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"joseph","HOBBY":"piano"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"joseph","HOBBY":"tennis"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"sandy","HOBBY":"football"}

{"LASTNAME":"clark","FIRSTNAME":"lena","NAME":"sandy","HOBBY":"guitar"}

Running the example

jar files needed

MongoJavaDriver: mongo-java-driver-3.0.0.jar (this is the version we have used)

Nosql client: kvclient.jar (it should be a version containing tables 12.3+, it had been tested with 3.3.4)

main java class : mongoloadnosql.MongoDB2Nosql (java source code is here)

Parameters

The tool has 5 parameters:

  • -limit <integer>, number of documents to load, 0 is for all the documents
  • -skip <integer>, offset of the first document to load, similar to the skip function in MongoDB, 0 means no skip of documents
  • -mapfile <file>, properties file to load
  • -create [true|<anything else than true>], if true the Nosql API functions for creation of tables and indexes are issued
  • -insert [true|<anything else than true>], if true the Nosql API functions  for insertion are issued

Launch command

This command creates nosql tables and indexes if there do not exists, and insert the whole collection items using the properties file  mappingfam.properties:

java -classpath <tool_dir>/classes:<KVHOME>/lib/kvclient.jar:<MONGODB_CLASSPATH>/mongo-java-driver-3.0.0.jar mongoloadnosql.Mongo2Nosql -limit 0 -skip 0 -mapfile mappingfam.properties -create true -insert true

Caveats and warnings

Actually there is no possibility to map  MongoDB references (neither referenced relationships nor DBRefs)

Fields should be presented in the order defined by their primary keys (lastname,firstname) instead of (firstname,lastname) 

The java code attached is just  to illustrate how to  import/migrate MongoDB data into Nosql tables in an efficient and consistent way, it has not been tested in all kinds of situations and it is not intended to be free of bugs.

Bonus

The following mapping file, allows to map MongoDB documents having the structure of a post.  

In this case there is an embedded object "origine" which is defined as: {"owner" : "gus","site" : "recent_safety.com"}) which is not a collection.

There is no primary key other than the MongoDB '_id' field.

Enjoy trying this example also.

About

This blog is about everything NoSQL. An open place to express thoughts on this exciting topic and exchange ideas with other enthusiasts learning and exploring about what the coming generation of data management will look like in the face of social digital modernization. A collective dialog to invigorate the imagination and drive innovation straight into the heart of our efforts to better our existence thru technological excellence.

Search

Categories
Archives
« April 2016
SunMonTueWedThuFriSat
     
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
23
24
25
26
27
29
30
       
Today