Charles Lamb's Blog

Here's the Berkeley DB Java Edition 3.3 change log as it stands now.  I can't give specifics of when it will be released other than "Calendar Year 2008" due to Oracle Revenue Recognition policies.  However, the change log will give you some idea of what is coming when it's available.

Berkeley DB Java Edition 3.3.46 Change Log

The changes between 3.2.76 and 3.3.46 are described
directly below.

Changes since 3.2.76

Log File On-Disk Format Changes:

The change is forward compatible in that JE files created with release
3.2.* and earlier can be read when opened with JE 3.3.46. The change
is not backward compatible in that files created with JE 3.3.46
cannot be read by earlier releases. Also read carefully the note above about
the incompatibility of this release with subsequent releases of 3.3.*. Note
that if an existing environment is opened read/write, a new log file is
written by JE 3.3.46 and the environment can no longer be read
by earlier releases.

A number of these on-disk format changes were made to reduce
disk-space usage. [#15399]

New Features:

1. 
   JE can now be configured to use a shared cache among multiple
   environments. By using the shared cache, multiple open environments in
   the same process will make better use of memory because the cache LRU
   algorithm is applied across all information in all environments
   sharing the cache. See EnvironmentConfig.setSharedCache() for more
   information. Two related stats were also added to EnvironmentStats:
   getNSharedCacheEnvironments and getSharedCacheTotalBytes. [#15267]
   


2. 
   JE now provides a "temporary" database mode, specified through
   DatabaseConfig.setTemporary(). Temporary databases operate internally in
   deferred-write mode to provide reduced disk I/O and increased concurrency. But
   unlike an ordinary deferred-write database, the information in a temporary
   database is not durable or persistent, and therefore temporary databases are
   lighter weight than deferred-write databases. More details can be found in the
   Getting Started Guide and in the javadoc for DatabaseConfig.
   

   
   In addition, two internal changes have been made to durable deferred-write
   database behavior. These changes are compatible with the existing use of
   deferred-write database, but may influence performance characteristics. These
   changes were necessary to ensure that the log cleaner operates properly for
   deferred-write databases.
   

   • The JE checkpointer now effectively performs a Database.sync for
     all open deferred-write databases. If you don't want this behavior
     you can a) use a Temporary DB instead, or b) disable the
     checkpointer. Disabling the checkpointer is commonly used to reduce
     logging during a bulk load.
   • JE now calls Database.sync for deferred-write databases when
     they are closed. If the data is not intended to be durable, we
     recommend using a Temporary DB instead.
   
   [#15365]
   


3. 
   Added a new EnvironmentConfig parameter: je.cleaner.upgradeToLogVersion. This
   parameter is used to perform background upgrading of earlier format JE log
   files. All log files having a log version prior to the specified version will
   be cleaned -- migrating the log records forward to the end of the log and
   updating their format -- at a time when no other log cleaning is necessary.
   This parameter is intended for use in upgrading earlier format log files
   forward to the current log format version, for example, to take advantage of
   disk usage improvements. Note that earlier format log files are supported by
   later JE releases and log upgrading is optional. [#15365]
   


4. 
   JE now supports key prefixing. When enabled, the keys stored on a
   given internal b-tree nodes may be represented as a set of
   prefix/suffix combinations where the prefix is the common prefix of
   all keys for the internal b-tree node. This can improve storage
   efficiency for some key sets, both on-disk and in-memory, at the cost
   of computational efficiency. The degree of storage improvement is a
   function of the application's key values and access pattern. Key
   prefixing can be enabled with the
   DatabaseConfig.setKeyPrefixing(boolean) method. [#15399]
   


5. 
   Added the ability to control caching policy on a per-operation basis.
   See Cursor.get/setCacheMode() and com.sleepycat.je.CacheMode to see
   how calls to Cursor "get" and "put" operations can specify LRU, MRU,
   or "pinned" cache hotness. [#15850]
   


6. 
   Four new stats have been added to meter random vs sequential disk io.
   

   
   EnvironmentStats.getNRandomReads()
   EnvironmentStats.getNRandomWrites()
   EnvironmentStats.getNRandomReadBytes()
   EnvironmentStats.getNRandomWriteBytes()
   EnvironmentStats.getNSequentialReads()
   EnvironmentStats.getNSequentialWrites()
   EnvironmentStats.getNSequentialReadBytes()
   EnvironmentStats.getNSequentialWriteBytes()
   
   
   

   
   return operation and byte counts for the number of random and
   sequential disk IOs. All values are approximate and may differ from
   the actual number of operations/byte-counts depending on the type of
   disks and file system, disk geometry, and file system cache size.
   [#16086]
   


7. 
   A new stat has been added to gauge the cost of log file opens, which
   can be a factor for applications with large data sets. A large value
   for EnvironmentStats.getNFileOpens() can indicate a need
   to tune the size of the file handle cache through increasing the
   je.log.fileCacheSize property. Also, general improvements
   have been made to improve the file handle cache, and to remove a
   bottleneck in concurrency in that area. [#16166]
   

General Environment Changes

1. 
   JE btree nodes now use shared latches for a portion of the btree
   access path. This used to be available through the
   je.env.sharedLatches configuration property, and is now the
   default. This results in better concurrency for read operations.
   The je.env.sharedLatches parameter has been deprecated. [#12136]
   


2. 
   In the past, JE's cache management pinned the internal metadata
   representing application databases. That made the use of large numbers
   of databases problematic, because significant portions of the cache
   could be consumed by metadata. That has now been resolved, and
   there are no cache or performance limitations on the number of
   databases in a JE environment. [#13415]
   


3. 
   The per-lock memory overhead has been reduced. [#15512]
   


4. 
   Various parameters have been adjusted so that JE will run under the Google
   Android platform. Refer to the Android HOWTO
   document for details.
   


5. 
   Fixed a bug in Transaction.setLockTimeout and setTxnTimeout so that the timeout
   is applied for all isolation modes. Before, the timeout was not applied for
   ReadCommitted and Serializable isolation under certain circumstances. The
   problem was initially reported on this JE Forum
   thread. [#16017]
   


6. 
   Fixed a bug where an internal data structure was being accessed sequentially
   during logging, while blocking other threads that attempt to write. For
   applications where the data set does NOT fit in cache, the data structure was
   small and the sequential access had little impact. But for applications where
   the entire data set DOES fit in cache, the sequential access was causing
   noticeable performance and concurrency problems during writing. [#16037]
   


7. 
   Improved concurrency when a large number of log files (more than fit in the
   open-file-cache) are being accessed. Before, the latch on the cache was held
   while opening the log file and reading the header. Now, only the latch on the
   individual file is held. Note that the size of the open-file-cache is
   configured with the je.log.fileCacheSize Environment configuration parameter.
   Normally the cache can be configured to a size large enough to hold all files
   being accessed. However, for very large data sets, this may not be practical.
   [#16166]
   


8. 
   The EnvironmentStats.getTotalLogSize statistic is now a slow stat (not returned
   when StatsConfig.setFast(true) is called) and a bug was fixed that caused the
   total to be incorrect and overflow in some cases. [#15985]
   


9. 
   Fixed a bug where lock timeouts (DeadlockException) would sometimes occur after
   twice the configured lock timeout interval. This fix may cause
   DeadlockException to occur more frequently in some applications because the
   configured lock timeout is now applied correctly. Note that the lock timeout
   is 0.5 seconds by default and may be configured via the setLockTimeout method
   in EnvironmentConfig and Transaction. [#16021]
   

Direct Persistence Layer (DPL)

1. 
   Several validation checks have been added or corrected having to do with entity
   subclasses, which are @Persistent classes that extend an @Entity class.
   
   • An IllegalArgumentException is now thrown when @PrimaryKey appears on an
     entity subclass. Before this was documented to be illegal but was not
     checked. [#15757]
   • The DPL now throws IllegalArgumentException if an instance of an entity
     subclass is embedded in another entity. It has always been illegal to embed
     entities within other entities, but the check for entity subclasses was not
     correct and has been fixed. [#16077]
   • In certain circumstances a NullPointerException was thrown before when
     attempting to embed an entity subclass, and this has also been corrected to
     throw IllegalArgumentException. [#16077]
   
   

API Changes:

1. 
   Deprecated the EnvironmentStats.getCacheDataBytes method. The new getDataBytes
   should be used instead. The getLockBytes method now includes the lock and
   transaction overhead. Before it returned only the lock overhead. [#15267]
   


2. 
   A number of fields in EnvironmentStats and TransactionsStats were
   changed from int to long to prevent overflows. [#15979]
   

Utility Changes:

1. 
   Database salvage mode is now available in the API through the
   com.sleepycat.je.util.DbScavenger class rather than solely from
   the DbDump -r command line. This helped to fix and clarify a mismatch
   between the Javadoc and the constructor parameters for DbDump in the
   JE 3.2.X code line. See the com.sleepycat.je.util.DbScavenger class.
   [#15385]
   

Configuration, Documentation, Test and Build Changes:

1. 
   JE 3.3.46 requires Java 1.5.0.12 or higher.
   


2. 
   Ant 1.7.0 is now required to build JE 3.3.46.
   


3. 
   In the past, the javadoc for JE's public classes was generated through
   an internal documentation process, and the actual content of the
   comments was not in the source itself. This was inconvenient for IDE
   users, because the javadoc contents would often not display in
   convenient places. This has been changed, and the javadoc content now
   resides in the source code in the normal fashion.
   


4. 
   Java developers often approach JE with past experience using SQL. To
   help illustrate the JE data model and API, we created a new example
   program which takes a list of common SQL queries and implements them
   using the Direct Persistent Layer API (DPL). Please refer to the
   Translating SQL Queries Example in JE Installation Notes for more
   information.
   

Here's one other entry that is probably interesting:

Improved performance for applications having a very large JE cache
and where
the active data set fits in the cache.  Before, in this scenario,
checkpoints
could take a long time to occur.  This in turn prevented log cleaning
from
being performed until after the end of the checkpoint.  Two changes
were made:

 

• Cleaner utilization information is now updated during the
  checkpoint.  The
  log cleaner will no longer wait until the end of the checkpoint to
  perform
  cleaning.  Note that although cleaning takes place during the
  checkpoint, log files are still not deleted until the end of the checkpoint.
  

• A new environment configuration parameter --
  je.checkpointer.highPriority
  -- can be used to reduce the time to perform a checkpoint.  If set to
  true, the
  checkpointer uses more resources in order to complete the checkpoint in a
  shorter time interval.  Btree latches are held and other threads are
  blocked
  for a longer period.  Log cleaner record migration is performed by
  cleaner
  threads instead of during checkpoints.  A shorter checkpoint is
  beneficial
  because a shorter recovery period will be required in the event of a
  crash.
  However, application response time may be longer during the
  checkpoint, and
  more cleaner threads may be required to maintain the configured je.cleaner.minUtilization.