Monday Jun 27, 2016

Oracle Berkeley DB Java Edition 7.0 Announced

Oracle Berkeley DB Java Edition 7.0 is now available for download.

Download Page – Click Here

Documentation Page – Click Here

Oracle Berkeley DB Java Edition Version 7.0 – New Feature

· Time-To-Live – efficient aging out of “expired” data – This is a common IoT requirement.

Berkeley DB JE continues to enable the most powerful embedded database solutions

· Pure Java library

· Handle TBs of data with a small, lightweight, embedded library

· Flexible, transactional data management engine

· Runs on a variety of operating systems and platforms.

· Over 50 open source software projects embed BDB -- check them out on Wikipedia

· Various options for APIs

· Industrial quality and battle tested

Supporting Assets and Resources:

Collateral and sales assets continue to be updated.

· Oracle Berkeley DB Web Site

· Oracle Berkeley DB blog

· Oracle Berkeley DB Java Edition Datasheet

· Oracle Berkeley DB Family Datasheet

LinkedIn Group – https://www.linkedin.com/groups/2131985

Twitter @berkeleydb

Community Berkeley DB – https://community.oracle.com/community/database/high_availability/berkeley_db_family/berkeley_db

Community for Berkeley DB Java Edition - https://community.oracle.com/community/database/high_availability/berkeley_db_family/berkeley_db_java_edition

Blogs - https://blogs.oracle.com/berkeleydb/

Tuesday May 31, 2016

Update for Oracle Berkeley DB - Version 6.1.29

Oracle is pleased to announce an update to Berkeley DB (BDB), version 6.1.29.  We want to make sure that some important features that are in our latest release (6.2.23) are available to those of you who continue to use the 6.1 release.

The highlights include:

  • A new utility db_convert which will convert a database to the endianness of the machine it is running on.
  • Encryption can be enabled when Binary Large Object (BLOB)  files are stored external to the database.
  • An improved vote comparison routine used to select a new master in HA.

Please go to :   http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index-082944.html    to download this release, 6.1.29.

Check out the change log for details on all the changes in this 6.1.29 update release.  Click Here.

Monday May 23, 2016

BDB User Authentication

BDB User Authentication

(courtesy of  Brayden Zhang)

Introduction

In Berkeley DB 6.2, user authentication extension is added into the SQL interface. The new user authentication layer is both secure and easy to use. In this article, we will introduce how to use the user authentication, and also explain some details.

The SQLite user authentication

User authentication was first included in SQLite 3.8.7, and was described well in the following page: http://www.sqlite.org/src/doc/trunk/ext/userauth/user-auth.txt  There are four user authentication APIs:

sqlite3_user_authenticate(conn, username, pwd, nPwd); sqlite3_user_add(conn, username, pwd, nPwd, isAdmin);

sqlite3_user_change(conn, username, pwd, nPwd, isAdmin);

sqlite3_user_delete(conn, username);

 A database becomes a user authentication required database once the first user is added with a sqlite3_user_add() call. To visit an authentication-required SQLite database, an authenticated user must be logged into the database connection first; otherwise attempts to read or write from the database will fail with errors.

From the implementation view, SQLite stores user/password-hash in an internal sqlite_user table and verifies user login with that table.

This SQLite userauth will be activated in Berkeley DB SQL layer with -

DBDBSQL_USER_AUTHENTICATION compile option.

Security considerations about the SQLite userauth

Let’s rethink about user authentication. When we are talking about user login, we mean we are putting the sensitive data into a safe zone. If a user need to access the data, we will ask the user for his name/password. Once we are sure the user is authenticated, we will let him into the safe zone. Thinking about a client/server Database, it provides the safe zone as the data is physically stored in a backend host.

Next let’s go back into the SQLite userauth, where is the safe zone? I guess the answer is: “it does not exist (yet).” It will be SQLite application’s resposibilly to provide it. In some cases it should be not so easy, such as an Android application. Anyone who has the access to this Android device could read the database file in binary editor; also if you built a SQLite without the userauth compile option, you could access an authentication-required database freely(no need for a login) 

Encryption should be a common(and neccesary) way to provide a safe zone for SQLite user authentication. In SQLite, to visit an encrypted authentication-required database, we need a call sequence as:

sqlite3_open_v2();
sqlite3_key_v2();
sqlite3_user_authenticate();

/* Database is now usable */ 

The next question is: who will provide the key?

  • The application could keep the key in somewhere and call sqlite3_key_v2() with the key. This means the application needs to implement some mechanism to protect the key;
  • Or the key will be remembered by every user, and he needs to provide it as well as his username/password. This sounds less attractive. 

 

Keystore-based userauth in Berkeley DB

It will be much better if an authentication-required database is encrypted, but a user could access the database with only his user name/password. We provide such an enhancement with a keystore file. Encryption is mandatory for a keystore based authentication-required database and we take the responsibly to protect the database encryption key. This is totally transparent to the end user. A user just does the login with a call sequence as:

sqlite3_open_v2();
sqlite3_user_authenticate();

/* Database is now usable */ 

In our enhanced sqlite3_user_authenticate() , we actually do things as below: 

1. fetch the database encryption key from keystore file using user name and password;
2. apply the encryption key to the database(that is, calling sqlite3_key_v2() );
3. verify user/password via sqlite_user table, as the orignal userauth does. 

This enhanced keystore based userauth will be activated with - DBDBSQL_USER_AUTHENTICATION_KEYSTORE compile option.

How the keystore file works

We store keystore entries in the keystore file. Every entry is the database encryption key which is encrypted with a salted hash value of user’s password. Say we have items below:
DEK - database encryptio key
EDEK - encrypted database encryption key
PWD - user's password
SALT - a randomly generated byte string
USERNAME - user's name
encrypt - Berkeley DB's encryption function;
decrypt - Berkeley DB's decryption function;
hash - Berkeley DB's SHA1 Secure Hash Algorithm function

When a user was added, we get the database encryption key from memory cache, and we compute 

EDEK = encrypt(DEK, hash(SALT, PWD))

 then we save a keystore entry as: (USERNAME, SALT, EDEK) . During a user login, to get the database encryption key back, first we find the keystore entry with user’s name, then we compute the database encryption key with:

DEK = decrypt(EDEK, hash(SALT, PWD)); 

Thus, by using hash/encryption, We ensure that:

  • The database encryption key in the keystore file is encrypted and could never be leaked from the keystore file;
  • Only user’s password hash is stored, so user’s password could never be leaked from the keystore file.

 

Make the Berkeley DB keystore userauth robust

If keystore file get corrupted, some of the end users will not be able to access the database. We try to make our keystore authentication as robust as possible.

Checksum and backup/restore

We enable checksum for the keystore file and have the checksum verified every time when keystore file was used. We also implement a simple backup/restore mechanism of the keystore file.

Locking file for the keystore

We expect the database be visited by multiple processes. To avoid keystore file from messed by multiple process writes, we introduce a lock file. Only when a process could have the .lck file
created(exclusively), it could access the keystore file; when it finished, it will remove the .lck file.

In rare cases, if user application fails when accessing the keystore file,  the locking file may not be cleaned. In this case, user needs to clean the .lck file under the database environment.

Treat updates of sqlite_user table and keystore file as an atomic operation

When adding/deleting/updating a user, we will first update the sqlite_user database table then the keystore file. We need to ensure that: Updates to sqlite_user table and keystore file will both succeed or both fail. This is guaranteed by database transactions. We put the updates of sqlite_user table and keystore file in a transaction and only commit when we succeed to update the keystore file; if we
failed to update the keystore file, we will rollback the database updates. Please note as we use transactions internally in userauth API, you should not call the userauth API within an outside transaction. If you do that, userauth API will return an error message.

Monday May 16, 2016

Slices in BDB

Slices in Berkeley DB

(courtesy of Charles Koester) 

With the new Berkeley DB 6.2 release, there are a number of new features.  One of the more exciting ones is the use of Slices.

It is possible to improve the multiprocessor scaling of your DB databases by implementing slices. Slices cause your database records to be spread evenly across sub-environments and sub-databases using an internal hashing algorithm. Because your records are stored in separate physical environments and databases, this allows for better parallel processing support by spreading read and write workloads across separate, discrete processor cores.

The new “slices” feature can improve the multi threaded performance of IOT and many other workloads. Throughput in multi threaded applications can improve by 3x or more. Transactions which access a single key — e.g., solo DB->get(), DB->put, or DB->del() calls, or DB->get() followed by an update call — are ideal for the use of the slices feature.. Workloads in which related sets of keys are accessed inside a single transaction can also use slices. By using the new DB->set_slice_callback() API call to designate the slice-relevant portion of the key, a transaction can access many different keys as long as they share the slice callback value.

Read more about the new Slices feature in Berkeley DB at:  http://docs.oracle.com/cd/E17076_05/html/programmer_reference/bdb_slices.html

Tuesday May 10, 2016

BDB Write Forwarding

(Courtesy of Paula Bingham) 

Simpler High Availability with Write Forwarding



Do you have a simple Berkeley DB application that would benefit from higher availability or improved read scalability?

These are well-known benefits of using Berkeley DB High Availability (BDB-HA).  You can use BDB-HA to turn your application into a distributed application using two or more nodes.  One node is the master node on which you can perform read and write operations; the other nodes are replicas which only support read operations.  When one node is down, the other nodes keep your application available to users. If you have many read operations, they can be distributed among different nodes.

 You might ask "But wait, what if I don't want to restrict write operations to the master node?" and "Isn't it complicated to add BDB-HA to an existing application?"

If your application performs its write operations one-per-transaction without using cursors, it is a candidate for the new Replication Manager write forwarding feature.  When write forwarding is enabled, these simple write operations can be performed on a replica.

The code changes to add Replication Manager write forwarding to your application are relatively simple and fall into the following major categories:

  • Configure and start Replication Manager:  You need to explicitly enable write forwarding, configure your nodes, and start Replication Manager.
  • Manage database creation: You only need to create your databases on the initial master node.  BDB-HA replicates them to the replica nodes.
  • Error handling: Minor code changes to handle a few additional BDB-HA errors.

See "Configuring for Write Forwarding"  in the Berkeley DB documentation for more information and sample code illustrating these points.

Monday May 02, 2016

Information on a new feature in Berkeley DB 6.2 - BDB Server

Information on a new feature in Berkeley DB 6.2 

(courtesy of Yong Li)

BDB Server (aka. BDB as standalone server or thrift)

Beyond being an industry-leading embedded key-value database, BDB Server turns BDB into a key-value database server which supports a client-server style of access. User applications can now use BDB Server client drivers to access remote BDB Servers.

Currently, the following BDB features are supported by BDB Server:

* B-Tree, Hash and Recno access methods.

* Transactions. (BDB Server can be run in Transaction Data Store mode only)

* Secondary indices and foreign constraints

* Join cursors

* Sequences

The following BDB features are not supported by BDB Server now:

* Queue and Heap access methods

* Replication

* Most call-back functions

The following features are specific to BDB Server:

* Support for SSL over TCP

* Handle timeout. If a handle has not been access for a long time, it can be closed automatically by the server.

System requirement for BDB Server:

* JDK 8 or above

Currently BDB Server client drivers are only available in Java and require JDK 8 or above. The Java client driver API is very similar to the BDB Java API with a few functions to open and close server connections.

Get Berkeley DB version 6.2 at:  http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html

Monday Apr 25, 2016

Greatly Improved SQL Index Performance for Oracle Berkeley Database, Version 6.2

A new whitepaper has been published, "Greatly Improved SQL Index Performance for Oracle Berkeley Database, Version 6.2". Learn how to increase the performance of SQL Queries using BDB.

Go to: http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/learnmore/index.html     and look at the Whitepaper section. 

Friday Apr 22, 2016

Oracle Open World 2016 - Call for Proposals

The call for proposals for Oracle Open World 2016 is open.  Please submit your proposals by May 9, 2016.  The proposal web site is:

 https://www.oracle.com/openworld/call-for-proposals.html

 Submit if you have an interesting use of Oracle Berkeley Database to talk about. 

Monday Apr 18, 2016

Oracle Berkeley Database (BDB) 6.2 Announced

Oracle Berkeley Database Version 6.2 (12.1.6.2.23) is now available for download.

Download Page – Click Here

Documentation Page – Click Here

Oracle Berkeley Database Version 6.2 – New Features

· Slices – provides significantly better scalability on large SMP machines as compared to the previous release.

· Write Forwarding – No longer are application writes restricted to the Master node when this feature is enabled.  Simple write operations can now be performed on the replica and BDB-HA will forward the writes to the Master automatically.

· BDB as a Stand-Alone Server Gives user an out-of-the-box solution to a key-value database in a client-server architecture.

· SQL Performance Enhancements – Index-based SQL query performance has been improved by as much as 40% out of the box when compared to BDB 6.1.

· Authentication – User authentication has been added to the BDB SQL API.  The database can now be set up to authenticate users prior to them accessing data in the database.  The new authentication capability, coupled with existing encryption, can be used to create secure, high performance embedded database applications.

Berkeley DB continues to enable the most powerful embedded database solutions

· Handle TBs of data with a 1MB library

· Flexible, lightweight, transactional data management engine

· Runs on a wide variety of operating systems and platforms ranging from low power ARM devices to clusters of high-end servers

· Over 50 open source software projects embed BDB -- check them out on Wikipedia

· Completely customizable, with choice of 5 different access methods

· Industrial quality and battle tested

Supporting Assets and Resources

Collateral and sales assets continue to be updated.

· Oracle BDB Web Site

· Oracle BDB Blog web site – continue to check for new blogs

· Oracle BDB Database Product Family Datasheet

· Oracle BDB Database Datasheet

LinkedIn Group - https://www.linkedin.com/groups/2131985/profile

Twitter berkeleydb

What others are saying:

Open source Fedora package maintainer, Lubomir Rintel, says "Berkeley DB has quietly served behind the scenes as the database for the RPM Package Manager.   It has proven itself time and time again as a robust and efficient storage engine.   It stores the meta information of the installed rpms.  Under heavy workloads, BDB proves itself reliable. Countless people that use popular Linux distributions have used BDB through RPM and never knew it.  With this new release,   BDB continues its tradition of being a solid storage engine"

Oracle Tape Product Manager, Dan Deppen, says "Berkeley DB is integral to  Oracle StorageTek Storage Archive Manager (SAM-QFS).  We have been embedding Berkeley DB in our product for over a decade and it is vital to our disk archiving feature which is used to send files to remote data centers to enable disaster recovery.  Performance and scalability are critical because SAM-QFS supports some of the largest archive customers in the world.   HPC sites, research centers, national libraries and other customers requiring massive scalability and high reliability depend on SAM-QFS and Berkeley DB to maintain availability of their critical data."

Questions & Help

External Email Alias bdb@oss.oracle.com

Monday Jul 14, 2014

Berkeley DB 12cR1 (12.1.6.1) released

New Releases of Oracle Berkeley DB

Available For Download Now
Subject:
Release Update


The newest release of Berkeley DB 12cR1 (12.1.6.1) is available now. Here is a summary of the new features:

      upgrades to in-memory OLTP throughput & performance
new
HA improvement to identify a single master in a 2 site replication group
new
HA useability improvements
new Blob support added into replication
removed need for fail check monitor process
reduced the time for a database backup
and a lot more!

Berkeley DB continues to enable the most powerful embedded database solutions

      Handle TBs of data with a 1MB library
Flexible, lightweight storage engine, small footprint
Runs on low power ARM devices to cluster of high-end servers
Over 50 open source software projects embed BDB -- check them out on Wikipedia
Completely customizable, choose from 5 different access methods
Industrial quality and battle tested with over 200 million deployments

BDB is hands-down the best edge, mobile, and embedded database available to developers.  With the flexibility to place log files and/or database in any directory, applications can easily take advantage of the IO performance of flash caches, flash disks or SSDs.


Top notch performance
:

  • Berkeley DB performs over 5 million operations a second on a 100GB database running on a 1/8 rack Exadata V2 Database Machine configured with 256GB RAM and 12 cores.
  • Berkeley DB can insert 100 thousand records in 72 milliseconds and read those records in 30 milliseconds, running on a 8 core XEON-based commodity server.  The records contain a 4 byte key and a 48 byte data value.  This was run using the benchmark described on pages 34-36 in an ebook on SQL Server 2014 from Microsoft Press.
  • To put this into perspective, we compared Berkeley DB to SQL Server 2014's In-Memory OLTP feature (code name Hekaton)  which has similar technology.  Berkeley DB, an open source product, is about 20% faster than SQL Server 2014 which takes 94 milliseconds for the same 100k insert operations on an 8 core X64 Intel commodity box with 14 GB memory.

    We are making available a benchmark program in C that can be configured to validate Berkeley DB throughput for the 100K insert test here.


What folks are saying:

Open source Fedora package maintainer, Lubomir Rintel, says "Berkeley DB has quietly served behind the scenes as the database for the RPM Package Manager.   It has proven itself time and time again as a robust and efficient storage engine.   It stores the meta information of the installed rpms.  Under heavy workloads, BDB proves itself reliable. Countless people that use popular Linux distributions have used BDB through RPM and never knew it.  With this new release,   BDB continues its tradition of being a solid storage engine"

Oracle Tape Product Manager, Dan Deppen, says "Berkeley DB is integral to Oracle StorageTek Storage Archive Manager (SAM-QFS).  We have been embedding Berkeley DB in our product for over a decade and it is vital to our disk archiving feature which is used to send files to remote data centers to enable disaster recovery.  Performance and scalability are critical because SAM-QFS supports some of the largest archive customers in the world.   HPC sites, research centers, national libraries and other customers requiring massive scalability and high reliability depend on SAM-QFS and Berkeley DB to maintain availability of their critical data."

  Oracle Identity Management Vice President, Shirish Puranik, says "Berkeley DB is a critical component of Oracle Unified Directory (OUD) and Oracle Directory Server Enterprise Edition (ODSEE).  We have been using Berkeley DB in these products as a high performance, transaction embedded database repository for several years.  Berkeley DB has exceed our expectations for performance and stability. Our Berkeley DB based products are widely deployed in production at largest telcos and financial institutions all over the world.   The improvements for BLOB support and high availability in the 6.1 release are welcome."



Software Downloads

Questions?

Please direct product questions to our Product Help Mail list for Berkeley DB. You can also email the Berkeley DB Product Management team directly.

___________________________________________________
Berkeley DB Product Management
Internal -
https://stbeehive.oracle.com/teamcollab/wiki/Berkeley+DB+Inside+Scoop
OTN - 
http://www.oracle.com/technetwork/database/berkeleydb/overview/index.html

About

Information about Berkeley DB products directly from the people who build them.

Search

Archives
« June 2016
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today