The plugin allows you to log statistics to a csv file at a specified interval. It also lets you graph stats in real time:
]]>
This release has High Availability and Replication (HA), improved IO performance (especially on Linux/ext3), and a jconsole plugin which should make performance tuning quite a bit easier for JE users.
You can download it at: http://www.oracle.com/technology/software/products/berkeley-db/je/index.html
At this point I am reasonably certain that the problem has to do with a write() call being initiated on a RandomAccessFile when an fsync() is already in progress in another thread (i.e. a concurrent fsync and write on the same file, but not with the same file descriptors). JE routinely performs concurrent IO operations on a given file. In the particular test case, it is by virtue of the checkpointer initiating an fsync while the user application thread is writing.
It turns out that in ext3 we previously encountered a performance slowdown because that file system takes an exclusive mutex on the inode for any IO operation, and therefore an fsync will block reads and writes. JE 4.0 has a "fix" to this problem which is described here. While the 4.0 "write queue" work improves performance on file systems like ext3 which take exclusive mutexes on the inode for any IO operation, it also has the added benefit that JE no longer performs concurrent fsync and write operations.
That said, there seems to be a true bug in Windows 7 IO, if for no other reason than I observe corruption on sector boundaries in the log files which are produced by the test case (JE does no operations on sector boundaries).
]]>Database.preload() since that would sort the LSNs of all of the records in the database and then load the cache by reading the records in LSN order rather than key order. The customer's program set the cache size to a fixed size of 1200 * 1024 * 1024 bytes. Interestingly enough, the call to preload() made the overall time longer than just doing the scan and taking the hit from the random IO.
The reason is that preload() will stop when it has filled the cache. In this case, 1.2GB was not large enough to hold all of the records in the database. Once preload() had filled the cache, it returned a status of PreloadStatus.CACHE_FILLED after which the scan commenced. Whereas the preload had read the records in LSN order, the scan was reading the records in key order (effectively random LSN order). Since the cache had already been filled by preload(), any cache miss by the scan would cause something to be evicted from the cache, and if the evicted record had not already been used by the scan, the work done by preload() to load the cache for that record was wasted. So with too small a cache, some IO done by preload() was inevitably wasted, thereby causing lower throughput.
Increasing the cache size to a level where preload() could fill the cache resulted in a significant speed-up.
Further, DPL now works on Android. That sure made life a lot easier when getting the demos for OOW ready.
All of this willmake the entire JE/Android experience a lot simpler.
The Boston "Big Data Summit" will be holding its first meeting on Thursday, October 22nd 2009 at 6pm at the Emerging Enterprise Center at Foley Hoag in Waltham, MA.
The Boston area is home to a large number of companies involved in the collection, storage, analysis, data integration, data quality, master data management, and archival of "Big Data". If you are involved in any of these, then the meeting of the Boston "Big Data Summit" is something you should plan to attend. Save the date!
More info here.
]]>----------------------------------------------------------------------
Where to find Berkeley DB people & sessions at Oracle Open World
----------------------------------------------------------------------
Hello and welcome to the upcoming Oracle Open World 2009 in San Francisco. We're less than 10 days away and the excitement is palpable. If you are interested in hearing about Berkeley DB, learning more about how the Berkeley DB products can be integrated into your applications/appliances/devices, seeing exciting Berkeley DB customer use cases or speaking directly with one of the Berkeley DB product development engineers, you can find us here:
* Session ID#: S311365, Title: Oracle Berkeley DB: Lightning-Fast Key Value Storage Just Got Faster, Date: Sun, Oct. 11th, Time: 15:45 - 16:45, Venue: Hilton Hotel, Room: Golden Gate 1, Track: Oracle Develop: Database, Speaker: David Segleau (Director Product Management), including customer presentations from Lucas Vogel, Managing Partner at EndPoint Systems and Madhu Bhimaraju, Database Architecture Engineer at Verizon Wireless. We will be covering several of the new features in Berkeley DB 4.8 and discussing how Verizon Wireless uses Berkeley DB to provide services to an ever-growing customer base.
* Session ID#: S311364, Title: Oracle Berkeley DB Java Edition High Availability: Java Persistence at Network Speeds, Date: Sun, Oct 11th, Time: 14:30 - 15:30, Venue: Hilton Hotel, Room: Golden Gate 1, Track: Oracle Develop: Database, Speaker: Sam Haradhvala (Senior Engineer on the Berkeley DB Java Edition product). We will be discussing the new High Availability/Replication functionality that will soon be available in Berkeley DB Java Edition -- how it works, how it can improve your application performance, reliability and throughput, as well as common use cases and configurations.
* Berkeley DB Applications Demo/Lunch. Date: Mon, Oct. 12th, Meeting Time: 11:00 - 13:00, Venue: Hilton Hotel - Union Square Room 6, 4th Fl. Greg Burd (Senior Product Manager) and several Berkeley DB engineers will be showing several Berkeley DB application demos, discussing how they were implemented and how similar functionality can be part of your application. Join us for lunch, some interesting demos and a open question and answer session.
* The Berkeley DB Product booth in the Exhibition Hall in Moscone West. We're workstation W-035, under the Database Track, in the Embedded Database sub-area just like last year. The booth is open Monday and Tuesday from 10:30-6:30 and Wednesday from 9:15-5:15. We're always delighted to talk with existing users, potential users and anyone who is curious about our NoSQL embedded database libraries.
]]>I have verified that the Android 1.5 (and presumably recently released 1.6) SDK supports JE and DPL. JE/DPL on Android adds a major chunk of infrastructure to the Android platform in that a pure Java transactional POJO datastore is now available.
You can expect to see stepped up support of JE on Android in the near future. For instance, we plan on making je-android.jar libraries available so that developers won't have to compile the JE src under Dalvik.
The plugin allows you to log statistics to a csv file at a specified interval. It also lets you graph stats in real time:
]]>
eWeek also has an article about this.
]]>We know of a workaround for the bug which we can give you if you run into this under Windows 7.
If you do run into this, we'd be grateful if you could give us your test case so that we could try to reproduce it on our own machines.
You can contact us on the OTN forum or via email (charles.lamb @ 0racle.com)
Thanks.
Charles Lamb
Here's the text of it:
What JVM parameters should I consider when tuning an application with a large cache size?
If your application has a large cache size, tuning the Java GC may be necessary. You will almost certainly be using a 64b JVM (i.e. -d64), the -server option, and setting your heap and stack sizes with -Xmx and -Xms. Be sure that you don't set the cache size too close to the heap size so that your application has plenty of room for its data and to avoided excessive full GC's. We have found that the Concurrent Mark Sweep GC is generally the best in this environment since it yields more predictable GC results. This can be enabled with -XX:+UseConcMarkSweepGC.
Best practices dictates that you disable System.gc() calls with -XX:-DisableExplicitGC.
Other JVM options which may prove useful are -XX:NewSize (start with 512m or 1024m as a value), -XX:MaxNewSize (try 1024m as a value), and -XX:CMSInitiatingOccupancyFraction=55. NewSize is typically tuned in relationship to the overall heap size so if you specify this parameter you will also need to provide a -Xmx value. A convenient way of specifying this in relative terms is to use -XX:NewRatio. The values we've suggested are only starting points. The actual values will vary depending on the runtime characteristics of the application.
You may also want to refer to the following articles:
* Java SE 6 HotSpot Virtual Machine Garbage Collection Tuning
* The most complete list of -XX options for Java 6 JVM
* My Favorite Hotspot JVM Flags
NullPointerException during eviction when an exception occurs during logging, for example, an IOException when the disk is full. The NullPointerException masked the original exception (e.g., IOException). We'd like to thank OTN Forum user tkram01 for reporting this on the BDB JE Forum. An example stack trace is below.
java.lang.NullPointerException
at com.sleepycat.je.tree.LN.log(LN.java:534)
at com.sleepycat.je.tree.LN.log(LN.java:373)
at com.sleepycat.je.tree.BIN.logDirtyLN(BIN.java:1016)
at com.sleepycat.je.tree.BIN.evictInternal(BIN.java:985)
at com.sleepycat.je.tree.BIN.evictLNs(BIN.java:940)
at com.sleepycat.je.evictor.Evictor.evictIN(Evictor.java:761)
at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:344)
at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:244)
at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:269)
at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:711)
at com.sleepycat.je.Cursor.close(Cursor.java:326)
at com.sleepycat.je.Database.putInternal(Database.java:999)
at com.sleepycat.je.Database.put(Database.java:885)