Charles Lamb's Blog

Often users experience performance problems related to improper cache sizing.  A useful tool, albeit an undocumented one, for helping to size caches is com.sleepycat.je.util.DbCacheSize.  The header comments in the source code provide a good description of how to invoke it, what it calculates, and what its limitations are.  Briefly, you provide (at a minimum) the key size and the number of records.  Optionally you can also provide data size, "overhead" percentage, btree fanout, node density, and a directory in which to perform a trial run.  The program then provides an estimate of the minimum and maximum cache size required for holding the database in memory.  This comes in two forms: (1) the cache size to hold the internal nodes, and (2) the cache size to hold the internal and leaf (i.e. data) nodes.  To account for various overheads, an "overhead" percentage is added.  So the output has both "Cache Size" and "Btree Size", the former including the overhead and being the larger.

Here is an example run (again, taken right from the source code class header comments):

$ java DbCacheSize -records 554719 -key 16 -data 100
Inputs: records=554719 keySize=16 dataSize=100 nodeMax=128 density=80%
overhead=10%

   Cache Size      Btree Size  Description
 ------------    ------------  -----------
   30,547,440      27,492,696  Minimum, internal nodes only
   41,460,720      37,314,648  Maximum, internal nodes only
  114,371,644     102,934,480  Minimum, internal nodes and leaf nodes
  125,284,924     112,756,432  Maximum, internal nodes and leaf nodes

Btree levels: 3

This says that the minimum cache size to hold only the internal nodes of the btree in cache is approximately 30MB. The maximum size to hold the entire database in cache, both internal nodes and datarecords, is 125Mb.