Charles Lamb's Blog

I want to call attention to Carbonado, an open source persistence framework that Brian O'Neill at Amazon developed.  TheServerSide also posted an article about Carbonado back in October.  Mark Hayes also posted an article on the OTN forums about it.

It's interesting for a couple of reasons.  First, it allows a programmer to create Java persistence with some independence of the underlying engine.  By default, Carbonado uses Berkeley DB Java Edition, but Berekeley DB "C" Edition, and JDBC can also be specified.  This is extremely useful
when an abstraction that supports an SQL-based backend is a requirement,
or when there is a need to synchronize data between Berkeley DB and SQL
databases.  In the past, our users have asked for this "engine independence" and our answer has always been that they need to build it themselves at a higher layer.  Carbonado provides such a layer.  Secondly, Carbonado provides a bunch of higher level features that Berkeley DB doesn't provide (and probably won't, at least for the forseeable future):

• Constraints on attributes.  You can set ranges on allowable attribute values.
• Query Language. You can do selections (filters), ordering, counting.  Debugging facilities which allow you to see the query plan are also available.
• Joins.  While BDB has some join capabilities, Carbonado has a more comprehensive set.  Many-to-one, one-to-one, one-to-many, and nautral joins are available.  Also, facilities reminiscient of relational views are provided.
• Non-persistent triggers.  As in a relational database, these allow custom operations when various actions on persistent records occur.  Actions can be invoked before, after, and on failure of an insert, update, or delete.

I finally got around to doing a quick test of calling Berkeley DB Java Edition (JE) from JRuby (JRuby is a 100% pure-Java implementation of Ruby).

Before we get to JE and JRuby you probably want to know the answer to this question: "Why you would want to run Ruby on a JVM?"  The answer is threefold:

1. Ruby Performance.  A large amount of effort has
   been put into tuning contemporary JVMs (e.g. Hotspot, Java 6, etc.) and Ruby programmers (through JRuby) can benefit from these
   tuning efforts. The JRuby guys have set a goal to make JRuby the fastest Ruby implementation available and Sun is certainly
   throwing their weight behind that effort. 
   
2. Portability.  JRuby is a Ruby interpreter that runs anywhere a Java 5 JVM runs.  You download it as a single tar.gz and it will run pretty much anywhere.
   
3. Legacy Code.  JRuby makes legacy Java apps and libraries available to Ruby programmers (did you ever think you'd see the word "legacy" next to the word "Java"?).
   

JE interoperability with JRuby is important because it means that Ruby
programmers now have a simple, embeddable, ACID storage engine
(JE) available to them.


To test this interoperability, I cobbled together a simple Ruby test program which does the following:

• Opens an Environment, Database, and Transaction
  
• Creates 10 records with keys 1..10 and marshaled Ruby Time instances as the corresponding data.  This uses the Ruby Marshal package for the data binding and the JE Integer binding on the key side.  There's no reason why you couldn't use different marshaling packages or methods for keys and data.
• Commits the transaction,
  
• Performs a Cursor scan to read those 10 records and prints out the Time instances, and
  
• Searches for and reads the record with key 5 (an arbitrary key) and prints out the Time
  instance that is the corresponding data
  

By the way, hats off to the JRuby developers: all of this code "just worked", out of the box, and most of my two hour investment was spent learning enough basic Ruby to make it all work.  If you already know Ruby and JE, then demonstrating this interoperability would take you all of about 10 minutes.

This was all done at the "base API" level of JE and no modifications to JE were required.  I used Transactions in my code, but there's no reason that you need to.  Mark and I have been talking about how to integrate JE's Direct Persistence Layer (DPL) with JRuby and we think it can be done with some remodularization of some of the DPL code.  This is exciting because it would provide POJO ACID persistence to Ruby programmers.  [Update 1/24/07: the next post in this blog discusses a compromise solution.]

Linda and I have been talking about whether it makes sense to possibly use Ruby as a scripting platform for JE in the future.  Given how easy it was to bring up JE and JRuby, this certainly warrants some further thought.

The Ruby code and corresponding output is shown below.  By the way, if you see something that I didn't do "The Ruby Way", feel free to let me know.

I'd love to hear about your experiences with JE and JRuby.  Feel free to email me a charles.lamb at <theobviousdomain dot com>.

---------------------------------

require 'java'

module JESimple
  require 'date'

  # Include all the Java and JE classes that we need.

  include_class 'java.io.File'

  include_class 'com.sleepycat.je.Cursor'
  include_class 'com.sleepycat.je.Database'
  include_class 'com.sleepycat.je.DatabaseConfig'
  include_class 'com.sleepycat.je.DatabaseEntry'
  include_class 'com.sleepycat.je.Environment'
  include_class 'com.sleepycat.je.EnvironmentConfig'
  include_class 'com.sleepycat.je.OperationStatus'
  include_class 'com.sleepycat.je.Transaction'
  include_class 'com.sleepycat.bind.tuple.IntegerBinding'
  include_class 'com.sleepycat.bind.tuple.StringBinding'

  # Create a JE Environment and Database.  Make them transactional.

  envConf = EnvironmentConfig.new()
  envConf.setAllowCreate(true)
  envConf.setTransactional(true)

  f = File.new('/export/home/cwl/work-jruby/JE')
  env = Environment.new(f, envConf);

  dbConf = DatabaseConfig.new()
  dbConf.setAllowCreate(true)
  dbConf.setSortedDuplicates(true)
  dbConf.setTransactional(true)
  db = env.openDatabase(nil, "fooDB", dbConf)

  # Create JE DatabaseEntry's for the key and data.

  key = DatabaseEntry.new()
  data = DatabaseEntry.new()

  # Begin a transaction

  txn = env.beginTransaction(nil, nil)

  # Write some simple marshaled strings to the database.  Use Ruby
  # Time just to demonstrate marshaling a random instance into JE.

  for i in (1..10)

    # For demonstration purposes, use JE's Binding for the key and
    # Ruby's Marshal package for the data.  There's no reason you
    # couldn't use JE's bindings for key and data or visa versa or
    # some other completely different binding.

    IntegerBinding.intToEntry(i, key)
    StringBinding.stringToEntry(Marshal.dump(Time.at(i * 3600 * 24)),
                                data)

    status = db.put(txn, key, data)
    if (status != OperationStatus::SUCCESS)
      puts "Funky status on put #{status}"
    end
  end

  txn.commit()

  # Read back all of the records with a cursor scan.

  puts "Cursor Scan"

  c = db.openCursor(nil, nil)

  while (true) do
    status = c.getNext(key, data, nil)
    if (status != OperationStatus::SUCCESS)
      break
    end

    retKey = IntegerBinding.entryToInt(key)
    retData = Marshal.load(StringBinding.entryToString(data))
    dow =
    puts "#{retKey} => #{retData.strftime('%a %b %d')}"
  end

  c.close()

  # Read back the record with key 5.

  puts "\nSingle Record Retrieval"

  IntegerBinding.intToEntry(5, key)
  status = db.get(nil, key, data, nil)
  if (status != OperationStatus::SUCCESS)
    puts "Funky status on get #{status}"
  end

  retData = Marshal.load(StringBinding.entryToString(data))
  puts "5 => #{retData.strftime('%a %b %d')}"

  db.close
  env.close
end

--------------------------------

Cursor Scan
1 => Fri Jan 02
2 => Sat Jan 03
3 => Sun Jan 04
4 => Mon Jan 05
5 => Tue Jan 06
6 => Wed Jan 07
7 => Thu Jan 08
8 => Fri Jan 09
9 => Sat Jan 10
10 => Sun Jan 11

Single Record Retrieval
5 => Tue Jan 06

In my previous post (Berkeley DB Java Edition in JRuby), I showed an example of calling JE's base API layer and mentioned that Mark and I had been thinking about how to use the DPL from JRuby.  Our ideal is to be able to define classes in Ruby, annotate those class definitions with DPL-like annotations, and have the JE DPL store them.  There are a number of technical hurdles to overcome before we can do this.  For instance, Ruby classes defined in JRuby do not map directly to underlying Java classes; instead they all appear as generic RubyObjects to a Java method.  Granted, it would be possible for the DPL to fish out all of the fields from these classes using reflection, but presently it's just not set up to do that (hence the modification to the DPL that I spoke about in my previous blog entry).  Furthermore, unlike Java, Ruby allows classes to change on the fly (add/remote new fields and methods) causing more heartburn for the DPL unless we required that only frozen Ruby classes could be stored persistently.

On thinking about this some more, we realized that there may be a way to use the DPL from JRuby, albeit with some compromises.  The key to this is that in JRuby, if a Java instance is passed back to the "Ruby side" (e.g. through a return value or by calling the constructor for a Java class), it  remains a Java instance, even when passed around in JRuby (and eventually passed back into the "Java side").  So what if we require all persistent classes to be defined (i.e. annotated) on the Java side?  That buys us the standard DPL annotations (effectively the DDL), freezes the classes that the DPL sees, and still lets us benefit from the POJO persistence of the DPL.  All of this can be done without modification to JE or the DPL using the currently available release.  I cooked up a quick example that builds on the standard "Person" example in the DPL doc and included the code below.

require 'java'

module DPL
  require 'date'

  # Include all the Java and JE classes that we need.

  include_class 'java.io.File'

  include_class 'com.sleepycat.je.Environment'
  include_class 'com.sleepycat.je.EnvironmentConfig'
  include_class 'com.sleepycat.persist.EntityCursor'
  include_class 'com.sleepycat.persist.EntityIndex'
  include_class 'com.sleepycat.persist.EntityStore'
  include_class 'com.sleepycat.persist.PrimaryIndex'
  include_class 'com.sleepycat.persist.SecondaryIndex'
  include_class 'com.sleepycat.persist.StoreConfig'
  include_class 'com.sleepycat.persist.model.Entity'
  include_class 'com.sleepycat.persist.model.Persistent'
  include_class 'com.sleepycat.persist.model.PrimaryKey'
  include_class 'com.sleepycat.persist.model.SecondaryKey'
  include_class 'com.sleepycat.persist.model.DeleteAction'
  include_class 'persist.Person'
  include_class 'persist.PersonExample'

  # Create a JE Environment and Database.  Make them transactional.

  envConf = EnvironmentConfig.new()
  envConf.setAllowCreate(true)
  envConf.setTransactional(true)

  f = File.new('/export/home/cwl/work-jruby/JE')
  env = Environment.new(f, envConf);

  # Open a transactional entity store.
  storeConfig = StoreConfig.new();
  storeConfig.setAllowCreate(true);
  storeConfig.setTransactional(true);
  store = EntityStore.new(env, "PersonStore", storeConfig);

  class PersonAccessor
    attr_accessor :personBySsn, :personByParentSsn

    def init(store)
      stringClass = java.lang.Class.forName('java.lang.String')
      personClass = java.lang.Class.forName('persist.Person')
      @personBySsn = store.getPrimaryIndex(stringClass, personClass)

      @personByParentSsn =
        store.getSecondaryIndex(@personBySsn, stringClass, "parentSsn");
    end
  end

  dao = PersonAccessor.new(store)
  dao.init(store)

  personBySsn = dao.personBySsn

  person = Person.new('Bob Smith', '111-11-1111', nil)
  personBySsn.put(person);

  person = Person.new('Mary Smith', '333-33-3333', '111-11-1111')
  personBySsn.put(person);

  person = Person.new('Jack Smith', '222-22-2222', '111-11-1111')
  personBySsn.put(person);

  # Get Bob by primary key using the primary index.
  bob = personBySsn.get("111-11-1111")
  puts "Lookup of Bob => #{bob.name}, #{bob.ssn}"

  children = dao.personByParentSsn.subIndex(bob.ssn).entities()

  puts "\nRetrieving children of Bob"

  while (true) do
    child = children.next()
    break if child == nil
    puts "#{child.name}, #{child.ssn}"
  end

  children.close()

  store.close
  env.close
end