Charles Lamb's Blog

Jonathan Gennick has a very good article in the latest edition of Oracle Magazine on Berkeley DB Java Edition.
The article is aimed at developers building apps that need object
persistence. It includes both a solid architectural overview of
Berkeley DB JE, and code that shows how to use the Direct Persistence
Layer in detail.

What do all the things in the title of have to do with each other?  It turns out that in some cases they are quite intertwined.  Here are some details:

A user reported a situation where their Berkeley DB Java Edition (JE) application would periodically cause their entire machine to freeze up for several seconds. In fact, the freeze was so severe that they couldn't type at an xterm window on the same machine and even running apps like vmstat (again in another window) would freeze.  This occurred on all of their test machines -- some more often and longer than others -- even though the machines were fairly large hunks of hardware (dual core Opteron boxes running a version of the Linux 2.6 kernel).  Our first reaction was that JE is just a Java library and if the machine freezes up, it's because of some bug in the JVM so it's not our problem.  But since it's a bad idea to blame bad behavior in a user's app built with JE on someone else's bug (i.e. the JVM), we set about to see if we could figure out what might be causing this with the hope of a workaround.

The user had given us a test case which had the following interesting characteristics:  it used multiple JE Environments, and there was a thread assigned to each Environment who's only job was to do inserts into that Environment.  Although each one of the Environments had a relatively small cache, the sizing was such that the total cache consumption was taking most of the JVM's heap (-Xmx).  The test case could reproduce the hanging within about 10 minutes on any of their machines and on one of our's.  The hangs only ever occurred on Linux platforms and never on Solaris and were generally during a Full GC (as indicated by the output from passing -verbose:gc to the JVM).  Running different versions of the JVM sometimes helped (i.e. later version of the Java 5 JVM or the Java 6 JVM would make improvements, but the problem was still there).  We tried any number of GC parameter combinations and some of them definitely caused different results (e.g. the test ran longer or shorter, or an OOME occurred, or the hangs were shorter), but nothing completely solved the problem.

After several days of poking at it, we had little to show for our efforts except that we knew that it had something to do with GC and using multiple JE Environments.  Going back to a higher level to think about the problem we asked ourselves why multiple Environments would cause this problem.  After all, we have plenty of users out there who have single Environment applications, benchmarks, tests, etc. that just sit there all day long doing inserts and they've never observed this problem.

Perhaps our cache eviction code wasn't working correctly?  Periodic output of the environment stats showed that the cache usage was not excessive and that "critical eviction" was being called (this is the type of eviction that occurs when a JE entrypoint is called and the cache is full -- we just call eviction right from the user's thread of control rather than wait for a background thread to deal with the problem).  But how often was each cache being evicted?  Adding annotations to the code which showed how often each of the individual caches was being evicted turned up an interesting anomaly.  Some of the threads (remember that each thread had its own Environment) were not being scheduled fairly.  In fact, some threads were being scheduled 40 or more times before some of the other threads were scheduled at all.

So what happens when certain threads are starved?  In a single environment case, nothing too bad happens (at least as it relates to this particular problem) because the threads that are still being scheduled are still inserting into the cache (and causing objects in that cache to be freed up so that the GC can reclaim the space).  But in a multi-environment case, the same cache-churn is not occurring in those caches that belong to the threads being starved.  Because of that, the objects in those caches age from the young space to the old space on the heap which is costly in terms of GC time (hence the large amount of time being spent doing Full GCs).

So what's the solution?  The scheduler couldn't be fixed -- that was out of our control.  Nor can we fix the system lockup -- we're working with Sun to help diagnose that as OS or JVM.  But reducing the number of Full GC's is important.  So we added a quick fix to JE which calls Thread.yield() whenever we check for cache overflow.  This causes somewhat fairer scheduling and therefore better cache churn across all Environments.  While it doesn't eliminate the problem, it does reduce the frequency of the hangs as well as their duration.  Ultimately, the answer is for us to implement a feature that we've talked about for sometime, shared caches for multiple environments, which will allow a single JE cache to be shared by multiple environments.

The forced yield fix is enabled with the je.evictor.forcedYield property and is available in JE 3.2.17 or later.

Updated 5/18/07:

The resolution to this issue is posted at http://forums.oracle.com/forums/thread.jspa?messageID=1849977

We'd like to thank Tony Printezis, an engineer in the Sun GC group, for his
help on this. Mark found Tony at the Sun booth at JavaOne last week and he
was kind enough to track down this problem and send us information
about the workaround. Thanks again Tony!