Handling Large Objects (e.g., Audio/Video files) in Oracle NoSQL DB

Recently, I have worked in a project that required using Oracle NoSQL Database to store complex structures like audio and video files, and make those files rapidly accessible through an web application. When I started the development of file retrieval from the KVStore, I got surprised about how bad documented this scenario is, specially if you are dealing with LOB's. So I decided to document examples of how I got the scenario solved.

Writing LOB Values into KVStore

So you have a file stored in the file system and you would like to store it in the KVStore to enable further retrieval? The best way to accomplish this is to extract the InputStream from the file and put its value into KVStore like this:

package com.oracle.bigdata.nosql;

import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.concurrent.TimeUnit;

import oracle.kv.Durability;
import oracle.kv.KVStore;
import oracle.kv.Key;

public class HowToStoreFilesFromFileSystem {

	// The reference below should be injected
	// by some runtime IoC mechanism or initialized
	// through constructor or setter methods...
	
	private KVStore store;

	public void storeValue(List<String> majorKeys, List<String> minorKeys,
			InputStream inputStream) {

		Key key = null;

		try {

			key = Key.createKey(majorKeys, minorKeys);
			store.putLOBIfAbsent(key, inputStream, Durability.COMMIT_NO_SYNC,
					10, TimeUnit.SECONDS);

		} catch (IOException ex) {

			throw new RuntimeException(
					"Error trying to store files into NoSQL Database", ex);

		}

	}

}

Reading LOB Values from the KVStore

Once you are ready to retrieve LOB values from the KVStore, perhaps wondering about how to rebuild the file in the file system, you can proceed this way:

package com.oracle.bigdata.nosql;

import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.concurrent.TimeUnit;

import oracle.kv.Consistency;
import oracle.kv.KVStore;
import oracle.kv.Key;
import oracle.kv.lob.InputStreamVersion;

public class HowToLoadFilesFromKVStore {

	// The reference below should be injected
	// by some runtime IoC mechanism or initialized
	// through constructor or setter methods...

	private KVStore store;

	public File buildFileFromKVStore(List<String> majorKeys,
			List<String> minorKeys, String fileName) {

		Key key = null;
		InputStreamVersion isVersion = null;

		int _byte = 0;
		File file = null;
		InputStream inputStream = null;
		BufferedInputStream bis = null;
		ByteArrayOutputStream baos = null;
		FileOutputStream fos = null;

		try {

			key = Key.createKey(majorKeys, minorKeys);
			isVersion = store.getLOB(key, Consistency.NONE_REQUIRED, 10,
					TimeUnit.SECONDS);

			if (isVersion != null) {

				inputStream = isVersion.getInputStream();
				bis = new BufferedInputStream(inputStream);
				baos = new ByteArrayOutputStream();

				while ((_byte = bis.read()) != -1) {
					baos.write(_byte);
				}

				file = new File(fileName);
				fos = new FileOutputStream(file);
				fos.write(baos.toByteArray());
				fos.flush();

			}

		} catch (Exception ex) {

			throw new RuntimeException(
					"Error trying to get files from NoSQL Database", ex);

		} finally {

			if (fos != null) {

				try {
					fos.close();
				} catch (IOException ioe) {
					ioe.printStackTrace();
				}

			}

		}

		return file;

	}

}

In the example above, the file is completely rebuilt in the file system to be accessed from any application. Note that once you have the BufferedInputStream in place, you don't actually need to rebuild the file completely in the file system in order to play with it. Perhaps you can transfer all those bytes to an mechanism that is perfectly capable to handle that stream in-memory, taking full advantage of handling data on-demand based on the streaming API of LOB's.

Comments:

What version of the database was used?
And what version it supports these examples?

Posted by Oleg on February 01, 2014 at 02:27 PM BRST #

And what about С++?

Posted by Oleg on February 01, 2014 at 02:29 PM BRST #

For this tests, I have used version 2.1.57 (last one) but this API exists for ever in the 2.0 version of ONOSQL.

Currently, ONOSQL does not support C++ clients. It does pure C API. In that case, you can download its API here:

http://www.oracle.com/technetwork/database/database-technologies/nosqldb/downloads/index.html

Once done, you can interact will full C functions related to large objects such as:

kv_lob_delete()
kv_lob_get_for_read()
kv_lob_get_for_write()
kv_lob_get_version()
kv_lob_put_from_file()
kv_lob_read()
kv_lob_release_handle()

Cheers,

Ricardo Ferreira

Posted by Ricardo Ferreira on February 01, 2014 at 05:37 PM BRST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Ricardo Ferreira is just a regular person passionate for technology, traveling, movies and his family. Currently he is working at Oracle U.S in the FMW Architects Team, otherwise known as the A-Team.

Search

Categories
Archives
« March 2015
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    
       
Today