Tuesday Sep 22, 2009

Videos to get acquainted with Apache Derby and Java DB



Kristian Wagaan and Richard Hillegas, both Apache Derby committers who also work on Java DB, recently gave some great presentations about Apache Derby and Java DB.

Kristian Wagaan presented at the 2009 OpenSQLCamp, in conjunction with FrOSCon 2009 in Europe.

The presentation (slides) gives a brief overview of Apache Derby, its history and the community around it. Further, it goes into the details of features that have been added in the latest releases and show how they enable powerful ways to use a relational database.

Rick Hillegas gave an introduction of Java DB / Apache Derby at The San Francisco Java User Group's Sep 8th event in San Francisco.

Wednesday Aug 26, 2009

Apache Derby 10.5.3 Bug-Fix Release is now available!

The Apache Derby project is pleased to announce a new bug-fix release of Derby,

Apache Derby is a subproject of the Apache DB project. Derby is a pure Java relational database engine which conforms to the ISO/ANSI SQL and JDBC standards. Derby aims to be easy for developers and customers to use.

Derby can be obtained from the Apache download site:

Derby contains many bug fixes plus localizations for messages added in the previous feature release, replaces, which introduced a query-ordering regression.

Here is the list of bug fixes addressed in 10.5.3.

Friday May 29, 2009

Java DB 10.5.1 now available

The Java DB project is pleased to announce a new feature release of Java DB 10.5.1.

Java DB is Sun's branded distribution of the Apache Derby open source  database. Java DB is a pure Java relational database engine which conforms to the ISO/ANSI SQL and JDBC standards. Java DB aims to be easy for developers and end-users to work with.

Java DB 10.5.1 can be obtained from the Java DB download site:

Java DB 10.5.1 introduces the following new features and improvements:

  • In-memory databases for test rigs and transient data
  • Replication of Encrypted Data for failover of sensitive databases
  • OFFSET/FETCH FIRST for paging through query results efficiently
  • SQL Roles for administering fine-grained access controls
  • Generated Columns for speeding up queries by pre-computing results
  • LOB Improvements for speeding up access to large binary and text objects
  • Optimizer Statistics Improvements for better management of query plans

Java DB offers the following: 

  • Installers for various platforms
  • Support offering (more info)
  • Notification of new releases and updates via registration
  • Native Netbeans integration
  • Bundled in GlassFish and Sun JDK6
If you are interested in learning more or/and sharing feedback about Java DB, please come and visit us at our POD during CommunityOne and JavaOne 2009.

Friday Mar 27, 2009

Java DB Articles and Interviews with Masood Mortazavi, Engineering Manager

Amongst the recent updates to Java DB's portal, I recommend reading these 2 great interviews of Masood Mortazavi, engineering manager for Java DB.

You can also find a various selection of technical articles, tips and white-papers.

If you are using Netbeans, this article will show you how to use Java DB more effectively inside this IDE. 

Tuesday Dec 16, 2008

Java DB on Twitter - Become a follower

Java DB (based on Apache Derby) is now on Twitter. If you want to follow the latest about Java DB, then become a follower ;-)

Sunday Nov 09, 2008

Java DB and GlassFish v3 Prelude

GlassFish v3 Prelude launched recently and bundles Java DB, Sun's supported version of Apache Derby.

As you may already know, GlassFish v3's foundations are based on HK2, some Hundred Kilobytes Kernel with a module subsystem and component model that can run on top of OSGI.

Java DB modular architecture is similarly based on modules and services components and can also run as an OSGI bundle. Both GlassFish and Java DB can run embedded in some Java framework or as standalone applications in their own JVM process.

I thought I would highlight how you can get Java DB running in one command and start using it. Note that there is no change as far as how Java DB is run with previous versions of GlassFish.

To start a GlassFish server instance:
\*\* Hint: First make sure to set JAVA_HOME environment variable to point to Sun JDK6 root installation directory.
> cd glassfishv3-prelude (or wherever directory you have installed GlassFish)
> bin\\asadmin start-domain (to start the default domain instance)

Then all you have to do to start Java DB as a standalone server process is:
> bin\\asadmin start-database

Database started in Network Server mode on host and port 1527.
--------- Derby Network Server Information --------
Version: CSS10020/ - (538595)  Build: 538595  DRDA Product Id: CSS10020
-- listing properties --

Now, that GlassFish and Java DB have been started, you can test a ping connection via the GlassFish Administration console at:

Navigate to Resources> JDBC> Connection Pools> DerbyPool
which is a pre-defined sample JDBC connection pool.

You will be able to ping the running Java DB instance by clicking on the Ping button.

It will create some sun-appserv-samples GlassFish sample database by default under glassfishv3-prelude\\glassfish\\databases directory and derby.log will also be under that same location.

That's it. You can now access the sample database via the JDBC sample DerbyPool connection pool from your application.

You could also connect via IJ, Java DB's command-line tool as follows:
> set DERBY_INSTALL=C:\\glassfishv3-prelude\\javadb
> cd glassfishv3-prelude
> javadb\\frameworks\\NetworkServer\\bin\\ij
> ij version 10.2
   ij> connect 'jdbc:derby://localhost:1527/sun-appserv-samples;user=APP;password=APP';
   ij> select count(\*) COUNT from sys.systables;
   ij> exit;

To stop Java DB, simply run:
> bin\\asadmin stop-database
Connection obtained for host:, port number 1527.
Apache Derby Network Server - - (538595) shutdown at 2008-11-09 08:48:43.312 GMT
Command stop-database executed successfully.

More information about GlassFish's admin start-database and stop-database commands.

Wednesday Oct 22, 2008

Java DB / Apache Derby Table Functions White Paper

Rick Hillegas has posted a great white paper on Java DB / Apache Derby support of ANSI SQL 2003 Table Functions.

A table function is like a virtual table that allows you to access data externally as if it was a local table, via the full, expressive power of SQL.

Data returned by these special functions can come from anywhere:

  1. files - files and web resources
  2. collections - in-memory collections
  3. foreign data - other databases, including non-relational sources
  4. streams - transient information streams, including data feeds and device outputs

Table functions are easy to implement and Rick's white-paper has several examples of practical table function use cases.

Derby has itself several implementations of virtual tables that can be accessed directly. One that you might already know is LOCK_TABLE that shows all locks currently held in the database.
TRANSACTION_TABLE, STATEMENT_CACHE and SPACE_TABLE amongst others are additional ones that can be quite useful.

Documentation about Java DB / Derby table functions can also be found here.

Monday Aug 11, 2008

Apache Derby Case Study: Benefits of a Microkernel architecture

Software embracing a microkernel architecture has been happening for the last 20 years, yet the great aspects of this particular architecture are too often undermined and not considered when working on a new project. There are various reasons for this, history behind it with poor experiences, but there has also been a lot of lessons learned . So what can one benefit from a microkernel architecture?

Microkernel introduction

In the last two decades, a lot of changes happened in the world of software and Internet has revolutionized quite a few trends. Microkernel architecture used to be adopted for operating system type of implementations pretty much exclusively. I remember about ChorusOS, a Unix variant and first-generation microkernel OS developed in the early 90's which was targeted at high-availability applications in the telecommunications space. Chorus was truly innovative back then and like Mach, another first generation microkernel OS implementation, suffered performance issues due to message processing between the kernel and user spaces (no sharing of address spaces). Minix, another well-known unix-like OS based on a microkernel architecture was created by Andrew S. Tanenbaum in 1987 and influenced the development of Linux although Linus Torvalds went with a monolithic kernel architecture instead.

There has been quite a few debates regarding the benefits and tradeoffs of a microkernel architecture versus a monolithic one. Most Unix operating systems are based on the last one, with no messaging between parts of the system and an easier (and controlled) access between the kernel and user spaces. A microkernel-based implementation typically grants the software with added modularity and flexibility due to the abstraction and extra independency (clear boundaries) between the various modules / parts of a system.

I stated earlier that trends had changed in the last 20 years and although it may sound like stating the obvious, what I really meant to say is that software architectures do evolve due to other factors such as hardware, markets and  new requirements. Today one can see a lot more software implementations based on a microkernel architecture, due to an increase demand and new requirements motivating that choice. Hardware (e.g. memory, CPU) has gotten a lot faster and cheaper, there is a lot of hardware environments to port to, the internet and its fast moving pace has increased the demand for more dynamic and flexible software.

Multi-threaded server applications are a lot more popular these days and the overhead of a microkernel-based implementation is no longer an issue when it comes to interactions between various modules involved in a particular processing operation. It used to be true for the most part, when IPC mechanisms had to be used to relay messages between modules running as part of different processes.

Why Microkernel

A microkernel architecture could be summarized to a "highly modular collection of application-neutral abstractions upon which can be built robust, flexible and dynamic applications" and brings the following and non-exclusive benefits to software implementations:

  • Modularity
  • Flexibility
  • Extensibility
  • Better footprint management
  • Easier development, unit testing, maintenance and portability

The modularity, flexibility and extensibility aspects are what I would consider the most important ones of a microkernel-based implementation. A highly modular architecture  allows protocols to be separated from their implementations as well as maintaining a level of independence between the various modules and components / services (enforcement of boundaries) of the system. Flexibility grants you with dynamic loading and unloading of modules and extensibility gives the developers means to extend the current system with new protocol or schemes implementations.

Footprint management can be an important issue in certain environments and the ability to remove unnecessary components as part of your application can help reducing the footprint significantly.

Modules can be developed and unit tested independently of each others and portability is usually easier to handle as non portable logic can be placed at the kernel level directly and not mixed with non-kernel components.

As far as potential performance improvements, I think this is very debatable unless one is dealing with a single-threaded monolithic implementation and in this case it would be slower on SMP machines obviously. I'm a big fan of multi-threaded implementations versus multi-process ones and in this case both architectures would run better on multi-core systems.

Microkernel-based architecture has also reached the world of database systems and most recently, Drizzle, a fork of the MySQL core engine has been announced and is supposedly aimed at Web and Cloud applications. This confirms the need for more modular and flexible implementations at the database management system level.

Microkernels and the likes in Java land

The Java platform and more specifically the language, facilitates the development of microkernel-based implementations. The object-oriented aspect of Java allows abstractions via published interfaces to be defined for the various parts to be implemented. In other words, this enables a clear separation between the protocol various implementations. Java dynamic class loading also allows modules to be loaded on the fly as needed.

To name a few Java applications and frameworks which have adopted a microkernel-based architecture:

  • GlassFish V3 has adopted HK2, the Hundred Kilobytes Kernel, with a module subsystem and component model on top of OSGI

For an example of a Spring DM application leveraging OSGI dynamic module system and separation of application logic into modules, I recommend reading this article.
Apache Derby which can also run as an OSGI bundle (since 2005) is used in this example as well.

There is also growing interest in the cloud computing arena with having a more modular, flexible and extensible Apache Hadoop. Being able to plug-in a query language of choice on top of Hadoop's huge datasets is a very attractive value-proposition indeed.

Apache Derby Case Study: How to benefit from a Microkernel architecture

Apache Derby / Java DB has been supporting a highly modular architecture since its inception in 1997 with its first release as part of Cloudscape. Derby's system is composed of a monitor, acting as a kernel of its own, booting and shutting down services as well as discovering appropriate modules to load for a particular service. As an example, different modules can be loaded during boot-up based on the environment such as the JVM version. Modules themselves can also require other modules to be loaded in the environment.

Derby already supports pluggable authentication schemes as well as pluggable raw storages via its module-based subsystem. It promotes a clean separation between the SQL and Store layers which favors the implementation and pluggability of different types of storages. For instance, one could envision a GRAPH storage for Derby to enable a graph-based data model to be accessed via Derby's relational engine. There is already some implementation of an in-memory storage and even a store-less storage type to test Derby's layers separation and submit queries against non-database objects.

The extra benefit of having a clear separation between a protocol and its implementation allows other projects to re-use parts of the system and contribute back to the open source community. For instance, several project have expressed their interest in integrating Derby's raw B-Tree storage as it has proven to be robust over 10 years and has an active open source community.

There are already on-going efforts and proposals to make Derby more flexible and reduce dependencies between modules of the store layer and have a cleaner pluggable storage implementation. When this modular architecture got implemented 10 years ago, there was little demand for specialized and pluggable storage. This has changed with new types of data models and requirements for web applications and cloud computing, translating into innovations at the data management level.

The good news for Derby is that it has already gotten the proper foundations in place to go the extra mile and have a clean microkernel architecture. This would not only benefit the store layer but also other parts of the engine, such as the SQL compiler (parsing and execution plan generation) and the access layer, residing above the store. For instance, one could imagine having Derby's SQL layer plugged-in on top of Hadoop, as an extension to be able to run SQL queries against MapReduce datasets.

In Conclusion

Software implementations which are supporting a microkernel architecture inherit highly modular and flexible characteristics. The performance gap between a monolithic and microkernel architecture is no longer much of an issue, especially in a multi-threaded environment where messaging is done via local interfaces without requiring the use of IPC mechanisms.

The dynamic nature of web applications and requirements to constantly adapt to new standards and protocols will continue to promote this architecture. OSGI has already showed some path and additional web frameworks have become compliant.

Derby with its highly-modular architecture excels in an embedded server mode configuration and is a natural database storage of choice for the Java platform and microkernel-based implementations in the middle-tier.

Wednesday Jul 30, 2008

SNCF rides on GlassFish and Java DB

As Alexis Moussine-Pouchkine posted on this blog, the French railways company, SNCF uses GlassFish and Java DB (based on Apache Derby) to power a monitoring application which produces alerts with a real-time graphical representation of key system variables as well as PDF reports.

Don't miss this questionnaire with Franck Leprêtre, Software Architect at SNCF.

Java DB version is bundled in Glassfish V2.

Sunday Mar 04, 2007

Sun releases Java Enterprise System 5.0 with Java DB (based on Apache Derby)

"Java ES (Enterprise System) already includes Java DB, Sun's supported distribution of the open-source Apache Derby database. The middleware also supports the PostgreSQL open-source database.

Sun now lays claim to more than 1.3 million subscribers for Java ES compared to the nearly 1 million it declared in October 2005."

Monday Feb 26, 2007

Derby Embedded w/ Eclipse

To follow-up on the Java DB / Apache Derby with Netbeans blog entry that I recently posted, here is another user experience from Paweł Piotrowski who used Eclipse (IDE) to access Derby in embedded mode.

Wednesday Feb 14, 2007

Connecting a GUI to a Java DB Database with NetBeans IDE

This NetBeans tutorial contributed by Talley Mulligan guides you through the process of connecting the GUI of an application to a Java DB database. It shows how to add data-aware JDBC components to the GUI that will enable the program to interact with the database.

If you have not seen it already, there is also a "Working with the Java DB (Derby) Database in Netbeans 5.5" tutorial contributed by Brian Leonard.

Wednesday Feb 07, 2007

ActiveMQ graduates from Apache Incubator and uses Apache Derby

Apache ActiveMQ Open Source Message Broker has now graduated to become a top level project (TLP) at Apache! Congratulations to the ActiveMQ community!

To achieve high performance of durable messaging, ActiveMQ embeds and uses Apache Derby (configured by default), to allow for some very fast persistence via JDBC, coupled along with ActiveMQ's high performance journal.

Tuesday Jan 16, 2007

Java TV

Intriguing blog entry title eh? Well, no it's not a new product from Sun Microsystems, neither some TV that can run or deliver some Java...

It a is repository of video interviews around Java posted by Developpez.com, one of the biggest Java developers community in France.

Please note that a lot of the videos are in french but not all of them.

Most recently posted video interviews are of Marc Fleury (JBoss), Vincent Massol (XWiki) and myself (Java DB).

Open Source Report on Apache Derby

David Spector has published an interesting article about Apache Derby on the Open Source Report blog.

Tuesday Nov 21, 2006

Databases community at OpenSolaris.org

There is a new databases community at OpenSolaris.org.

This community is about all databases on Solaris. It is for both users of databases as well as those of us involved in creating and delivering database technology.

Topics will range from:

\* How to make your database run best on Solaris
\* Ideas for improving Solaris to better enable database technology
\* Comparison and contrast of various database technologies
\* Pointers to open source database add-ons, tools, and other code
\* Other topics related to databases on Solaris

Please feel free to join in and participate!

Wednesday Dec 14, 2005

ApacheCon demo - Web Browser Client Offline Storage with Java DB / Apache Derby

What I demo'ed at Apache during Tim Bray's keynotes was one particular use case of embedded Apache Derby (100% pure Java RDBMS) running within a web browser such as Mozilla Firefox...This allowed me to persist data automatically as HTML FORM fields got changed (at the field level or not) - It autosaved data entries locally without being paranoid about loosing my data in case of a browser crash or bad manipulation (fat fingers anyone?)...The store service can be called at anytime really from your client script...

Various database systems excel at different types of needs...My intent here was to use Apache Derby as the repository for an application managing some user's personal data (i.e. personal/private/financial taxes info). This is very different than managing browser's data/metadata (i.e. bookmarks, history, cache, etc) at the low-level and where indeed I would need a closer integration of the RDBMS with the browser engine...The demo highlighted how a web application could be enhanced (fast, embedded and secure local SQL storage) across web browsers, not just how it could enhance a particular web browser (i.e. Firefox).

As this particular demo manages personal and financial data, it is very likely that a production application would want to encrypt the data stored on the client storage - This is a possible and very likely an application requirement for this type of webapps. Derby supports strong encryption at the database level as well as strong authentication at the Derby engine or/and database level. Hence, I can have my local derby database(s) be secure, portable and platform independent without having to worry if I loose my usb drive or stick and my DB is on it...Derby also has some ACL-light support for embedded type of apps but Ansi SQL Grant & Revoke is being implemented as we speak - this is independent of the OS FS file permission scheme and is portable across OS's.

The demo I shown is one particular browser database integration scheme/topology - I did not have to \*natively\* integrate with the browser engine at all for this application. It was all driven from the application scripting logic interacting directly with a ClientStoreService object - This last one is implemented in Java and acts as a controller to the local client storage (Derby). It is seen as an object from Javascript (LiveConnect technology) which allows me to access some client storage service to issue queries and get data returned in XML, parse/process it with the native scripting engine XML Dom parser. The ClientStore Service can process requests asynchronously or not - I have another implementation where I added some HTTP request handling capability to Derby but this is NOT part of Apache Derby (yet) - this allows me to send a (Local) AJAX request (LAJAX) to Derby directly and get data returned as XML. The scripting engine does NOT have to be Javascript for the (L)AJAX mode...

For the Java engine I used Sun's JRE (browser Java plugin) which is a one-click install and is portable across and widely known web browsers.

Requirements for this Webapp demo were:
- A fast, embeddable, portable and secure (authentication/authorization/encryption) local client storage engine
- Portable across most widely known web browsers (at least 2 of them)
- Support for scripting standard asynchronous requests (Ajax mode) to the embedded client data store
- Various deployment options for the web application and database engine
- Support for direct access via a client scripting engine (non-Ajax mode, ala LiveConnect)

The deployment options for the ClientStorage engine and service would be:
- Direct HTTP download and archival of JARs,
- Direct copy of files,
- Firefox extension (w/ a dependence on the J2SE runtime environment Firefox plugin)
- Web shipping of already pre-populated databases, then made available locally to the web browser.

You can read all the details about this demo and get the source code at:

Thanks for reading...





« July 2016