Friday Jan 31, 2014

Handling Large Objects (e.g., Audio/Video files) in Oracle NoSQL DB

Recently, I have worked in a project that required using Oracle NoSQL Database to store complex structures like audio and video files, and make those files rapidly accessible through an web application. When I started the development of file retrieval from the KVStore, I got surprised about how bad documented this scenario is, specially if you are dealing with LOB's. So I decided to document examples of how I got the scenario solved.

Writing LOB Values into KVStore

So you have a file stored in the file system and you would like to store it in the KVStore to enable further retrieval? The best way to accomplish this is to extract the InputStream from the file and put its value into KVStore like this:

package com.oracle.bigdata.nosql;

import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.concurrent.TimeUnit;

import oracle.kv.Durability;
import oracle.kv.KVStore;
import oracle.kv.Key;

public class HowToStoreFilesFromFileSystem {

	// The reference below should be injected
	// by some runtime IoC mechanism or initialized
	// through constructor or setter methods...
	
	private KVStore store;

	public void storeValue(List<String> majorKeys, List<String> minorKeys,
			InputStream inputStream) {

		Key key = null;

		try {

			key = Key.createKey(majorKeys, minorKeys);
			store.putLOBIfAbsent(key, inputStream, Durability.COMMIT_NO_SYNC,
					10, TimeUnit.SECONDS);

		} catch (IOException ex) {

			throw new RuntimeException(
					"Error trying to store files into NoSQL Database", ex);

		}

	}

}

Reading LOB Values from the KVStore

Once you are ready to retrieve LOB values from the KVStore, perhaps wondering about how to rebuild the file in the file system, you can proceed this way:

package com.oracle.bigdata.nosql;

import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.concurrent.TimeUnit;

import oracle.kv.Consistency;
import oracle.kv.KVStore;
import oracle.kv.Key;
import oracle.kv.lob.InputStreamVersion;

public class HowToLoadFilesFromKVStore {

	// The reference below should be injected
	// by some runtime IoC mechanism or initialized
	// through constructor or setter methods...

	private KVStore store;

	public File buildFileFromKVStore(List<String> majorKeys,
			List<String> minorKeys, String fileName) {

		Key key = null;
		InputStreamVersion isVersion = null;

		int _byte = 0;
		File file = null;
		InputStream inputStream = null;
		BufferedInputStream bis = null;
		ByteArrayOutputStream baos = null;
		FileOutputStream fos = null;

		try {

			key = Key.createKey(majorKeys, minorKeys);
			isVersion = store.getLOB(key, Consistency.NONE_REQUIRED, 10,
					TimeUnit.SECONDS);

			if (isVersion != null) {

				inputStream = isVersion.getInputStream();
				bis = new BufferedInputStream(inputStream);
				baos = new ByteArrayOutputStream();

				while ((_byte = bis.read()) != -1) {
					baos.write(_byte);
				}

				file = new File(fileName);
				fos = new FileOutputStream(file);
				fos.write(baos.toByteArray());
				fos.flush();

			}

		} catch (Exception ex) {

			throw new RuntimeException(
					"Error trying to get files from NoSQL Database", ex);

		} finally {

			if (fos != null) {

				try {
					fos.close();
				} catch (IOException ioe) {
					ioe.printStackTrace();
				}

			}

		}

		return file;

	}

}

In the example above, the file is completely rebuilt in the file system to be accessed from any application. Note that once you have the BufferedInputStream in place, you don't actually need to rebuild the file completely in the file system in order to play with it. Perhaps you can transfer all those bytes to an mechanism that is perfectly capable to handle that stream in-memory, taking full advantage of handling data on-demand based on the streaming API of LOB's.

About

Ricardo Ferreira is just a regular person that lives in Brazil and is passionate for technology, movies and his whole family. Currently is working at Oracle Corporation.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today