IIOP Changes in GlassFish 3.1
By kcavanaugh on Feb 28, 2011
A number of changes were made in GlassFish 3.1 for IIOP issues. The major new feature for IIOP in GlassFish 3.1 is fail over and load balancing (FOLB). FOLB in GlassFish 3.1 has the same functionality as in GlassFish 2.1, with some adaptations for changes in Admin, Shoal, and OSGi.
Several ORB internal changes were made as well, significantly improving the quality of the ORB code, and reducing the ORB size at the same time. Quite a few bugs were fixed. As always, not every bug could be fixed in this release, but we made some significant progress.
The changes have also simplified the ORB build: the GF 3.0 ORB takes around 61 seconds to build (ant build make-bundles), while the GF 3.1 ORB takes around 42 seconds (ant build, which now includes make-bundles) on the same machine. Most of this difference is due to the elimination of Jscheme (see ORB Exceptions below).
The main ORB OSGi bundle (glassfish-corba-orb) went from 1.923 MB to 1.592 MB. This decrease is due mostly to the improvement in the ORB tracing and exception handling facilities. Unfortunately, OSGi issues (no split packages) required an increase in the size of glassfish-corba-omgapi from 306 KB to 605 KB, which basically wiped out the decrease in the ORB size. The total size of all ORB bundles (7 in 3.0, 8 in 3.1) went from 3.212 MB to 3.299 MB, a very modest overall increase.
- gf31-master, which contains the final ORB code for GlassFish 3.1 (tagged VERSION-3.1.0-b027)
- iiop-folb-dev-tests, which contains the IIOP FOLB dev tests (discussed below)
Note that there are 2 GlassFish ORB related projects: gf-corba-v3-mirror and glassfish-corba. We still need to merge these projects together as the current situation is quite confusing. What happened is we moved the CORBA work from an internal TeamWare based repository to kenai.com several years ago, when kenai.com was first released. Separately we created glassfish-corba simply to have a place to make the CORBA documentation publicly available. This happened because at the time, kenai.com did not have a good solution for project docs, and java.net did not support mercurial. Since then, kenai.com has added a lot of new features, and java.net has moved to the kenai infrastructure, and so java.net now supports mercurial.
Hopefully in a near future release we can eliminate one of these projects and consolidate the contents into a single project. There simply was not enough time to do this for GF 3.1.
You can see all of the resolved IIOP issues for GlassFish 3.1 in this JIRA query. We fixed around 50 issues. Many of these are bugs, but the issues also includes a number of new features:
- Quite a few changes were made to improve ORB performance in several areas:
- Lock contention:
- Object Copying:
- The ORB tracing facility was completely redesigned and reimplemented
- The ORB exception generation and logging facility was completely redesigned and reimplemented
- No issue; see ORBExceptions.
- IIOP fail over and load balancing was added from GF 2.1
A number of performance improvements were made, as noted above. The biggest area of improvement centered on lock contention, which was particularly an issue for a certain telecommunications customer as well as for some GlassFish benchmarks (notably Trade2). The basic pattern was the same in most cases: a common path through the ORB (usually related to request dispatching or marshaling) needed to lock a resource against an update for a relatively uncommon event (such as destroying a POA instance). But in most cases, no updates occur, so no lock is really needed. We couldn't easily move to a lock-free implementation, but relatively simple read-write locking was very effective, since the normal case allowed an arbitrary number of readers to acquire the read lock.
The copyobject issues were only small changes here, related to how linked lists are handled in the object copier. Long linked lists currently can cause stack overflows, so these require special handling. Some of the changes caused performance regressions, so some careful balancing of the various issues was required. We have plans to do some major rework in this area in a near-future release (see Issue 13996).
Some other issues were caused by a simple fact of life: making major changes often causes new performance issues to surface. This happened with both the new exception facility (Issue 14852) and the new tracing facility (Issue 15392).
One very interesting issue that showed up in 3.1 is related to JPA. We have some applications in which the same class appears in both the client and the server, and is marshaled as part of an RMI-IIOP request. However, the class is one that is instrumented by the JPA bytecode weaver on the server side. New JPA behavior introduced in GlassFish 3.1 added a new field to the modified class, and this in turn changed the RMI-IIOP hashcode for the class. The result of this is that (as far as Java Serialization or RMI-IIOP marshaling semantics are concerned) the old class and the new class are now different versions of the same class, and special handling is needed. RMI-IIOP handles this case using a remote object called the full value descriptor (FVD), which is sent in an IIOP message the first time a value type is marshaled on a newly opened ORB connection. When the receiver of an object detects that the version of the object received is not the same as the version of the object in the receiving VM, a method named "meta" is invoked on the FVD. This introduces an extra round trip at the network level, considerably slowing the request processing. This problem affected issues 15816 and 16078. There is no easy ORB-level solution for this (caching meta data from FVDs may be possible, but would be a bit tricky). The solutions are either to change the behavior of the JPA weaver, or to make both sides (client and server) weave (or not) the same way.
The old ORB tracing facility was very intrusive in the source code, requiring try-finally blocks in many significant methods. The new version avoids this by annotating methods that need tracing, and then using bytecode instrumentation at build time to add the appropriate try-finally blocks in traced methods. This is a simple example of aspect-oriented programming. We did not want to deal with either the complexity of existing AOP solutions, or the legal complications of adding another third-party tool and library to the ORB project, so we build a custom AOP library and tool for this purpose. The tool is also customized for annotation-driven processing in ways that do not seem to be easy to do with other tools. For details, see issue 11943, which references the details at the CorbaTracing page on the CORBA wiki. Some additional work is under consideration for a near future release: see issues 15404 and 15567.
ORB exceptions used to be handled using a wrapper class which took care of a number of details:
- Internationalization through the standard logger resource bundle mechanisms
- Logging using java.util.logger
- Generating an appropriate CORBA exception, which contains a GlassFish message catalog
- Passing exception specific parameters into the log message (again, using the standard logger mechanisms)
All of this is done with a single call to a method in a wrapper class. The old version generated the wrapper class at build time using a Scheme script (written using the JScheme java extensions) that generated Java classes form Scheme source files. This mostly worked well, but had some significant limitations:
- Most developers hate anything that looks like Lisp (I don't mind: syntax just isn't that important for the most part)
- It added a third-party dependency to the ORB build
- The old system was relatively inflexible: it only supported CORBA system exceptions
- Adding new exceptions required editing a scheme source file (see the first bullet)
- The source code generated was quite large
The new system stores the logging information in annotations on a wrapper interface, so no code generation is needed. A dynamic proxy is used to implement the interface at runtime. We wrote more scheme code to do the initial conversion from the original scheme source code to the new annotated java interfaces, which were then cleaned up as needed. All of the scheme-related code was then removed from the build. This also preserved most of the ORB code that called wrapper methods: perhaps 5% of the ~700 wrapper calls in the ORB needed some change (usually to deal with completion status on CORBA system exceptions).
As noted in the Issues section, all of this is described in more detail in ORBExceptions.
The most important change we made for CORBA in GF 3.1 was the addition of IIOP FOLB (fail over and load balancing). This is a feature from GlassFish 2.1 that could not be brought forward into GF 3.0 because we did not have cluster support in 3.0. The addition of cluster support in GF 3.1 provided the foundation for implementing IIOP FOLB.
The semantics of FOLB in GF 3.1 are the same as in the latest GF 2.1.1:
- InitialContext load balancing makes a load balancing decision when new InitialContext is called. All subsequent requests coming from the InitialContext are handled in the same GlassFish instance.
- per-request load balancing makes a load balancing decision on every invocation. This is only allowed for stateless session beans. This feature was added for a customer in GlassFish 2.1.1. It was not present in the original plan for GF 2.1.
- fail over, which causes a request that fails due to a recoverable exception to fail over to another instance in the same cluster. As in GF 2.1, the fail over is sticky: all requests to the failed instance failover to the same new instance in the cluster.
The implementation work for FOLB is covered in issues 11944 and 11945 as noted above. The dev tests are available in the CORBA project. The dev tests cover basic fail over and load balancing tests, as well as a number of failure scenarios from FOLB issues filed by SQE.