Wednesday May 13, 2009

Monitoring direct buffers

One of the smaller features that JSR-203/NIO2 brings to JDK 7 is a management interface for monitoring the resources associated with pools of buffers. "Buffers" here means direct buffers that are allocated outside of the garbage-collected heap (by ByteBuffer.allocateDirect), and mapped buffers, created by mapping a region of a file into memory (using The management interfaces are registered with the platform MBeanServer and exposed to tools that connect to the JMX agent. This means that tools such as jconsole and VisualVM can effectively monitor the usage of these buffers. The following is a partial screen-shot of VisualVM doing just that:

The management interface is java.nio.BufferPoolMXBean and the screen-shot is of the MBean browser plugin looking at the attributes of a buffer pool named "direct", the pool for direct buffers. In this case there are 31831 direct buffers with a total capacity of 4074368 bytes. The application in this example is a rogue test called DirectBufferGCKiller2 that continuously allocates, and discards, tiny direct buffers. Due to page alignment, the memory usage is significant higher than the total capacity of the buffers; this application could be a lot more efficient if it allocated its tiny buffers by slicing a large buffer. As an aside, page alignment is something we hope to eliminate in JDK 7 (this is a topic for another day).

For those that prefer command-line tools then it is easy to hack up a tool to monitor the usage over time. is one example. It uses the Attach API to attach to a running VM and print usage information like this:

$ java -classpath .:$JAVA_HOME/lib/tools.jar MonBuffers 18051
           direct                        mapped
 Count   Capacity     Memory   Count   Capacity     Memory
 26346    3372288  219198720       0          0          0
 29296    3749888  243742720       0          0          0
 32382    4144896  269418240       0          0          0
 36957    4730496  307482240       0          0          0
 40032    5124096  333066240       0          0          0
 44848    5742336  373251840       0          0          0
 51664    6612992  429844480       0          0          0
 58698    7513344  488367360       0          0          0

The columns with "0" relate to "mapped" buffers which aren't used by this test. With a bit of effort it would be possible to develop a much slicker tool and maybe a plugin for VisualVM (hint hint).

In addition to monitoring buffer usage in a running system it is also important to be able to examine the usage postmortem (say after a crash due to exhaustion of the native heap). In this case it is easy to hack up a quick tool that uses the HotSpot Serviceability Agent to obtain this information. It might be useful to add an option to jmap to do just that.

Update (June 23, 2009). Tomas Hurka, has announced a VisualVM plugin to monitor buffer pools. It's available now at the plugin center.

Monday May 04, 2009

Copy that is one of several new samples included in JDK 7 to demonstrate its new API to the file system. If you have any recent snapshot installed then you'll find these samples in the $JAVA_HOME/sample/nio/file directory. works like the Unix cp program to copy a file or file tree to a target location. It supports the -r, -p and -i options to do a recursive copy, preserve attributes, or work interactively to prompt whenever an existing file would be overwritten. The sample code is relatively simple and demonstrates many aspects of the API that are worth looking at.

At its heart, the copyTo method is used to do the copy. The Copy sample uses the COPY_ATTRIBUTES option when invoked with -p to preserve attributes. This is useful when you want the file attributes/meta-data copied to the target file. This means the file timestamps, permissions, Access Control List, extended attributes, etc. The other option used in the sample code is REPLACE_EXISTING; this is used to replace the target file if it exists. One thing to point out is that the copyTo method doesn't copy the contents of a directory. If you invoke it on a directory then it creates an empty directory. On the surface this might appearing limiting but work through the sample and it should becomes clear.

Another interesting thing to point out is that you'll see code like:


The relativize method used to compute a relative path between source and file. This is then resolved against target. For example, suppose we are doing a recusive copy from /users/gus to /users/joe/backup_of_gus. As we walk the tree we will need to copy /users/gus/stamps/rare/black_penny.html. In this sample source.relativize(file) yields the relative path stamps/rare/black_penny.html. Resolving this against the target directory (/users/joe/backup_of_gus) yields us the path in the target tree: /users/joe/backup_of_gus/stamps/rare/black_penny.html.

The other important API used in the sample is the Files.walkFileTree method. This method is used to implement recursive file operations, starting at a given starting point. The method is passed a FileVisitor that is invoked for each file or directory in the file tree. The Copy sample has an inner class TreeCopier that implements FileVisitor . TreeCopier's preVisitDirectory method is invoked for each directory before the entries in the directory are visited. It simply invokes the copyTo method to copy the directory (creating an empty directory as I mentioned above). When the copy succeeds or the directory already exists the preVisitDirectory returns CONTINUE to instruct the iterator to continue and visit the files in the directory. If the copy fails, then preVisitDirectory returns SKIP_SUBTREE to tell the iterator to skip the directory. You'll see that postVisitDirectory is also implemented. This is invoked after all of the entries in the directory have been visited. In this code it is implemented so that the directory's last-modified-time can be fixed up after we are done with the directory. Most recursive operations will only need to implement one of preVisitDirectory or postVisitDirectory. The visitFile method is very simple and just copies the file to the corresponding file in the target tree. One interesting thing to point out is that visitFile is invoked if a cycle is detected (a cycle being a symbolic link to a parent directory). Copying is one of the few cases where you want to follow symbolic links and so you need to be concerned with the possibility of a cycle.

Hopefully this sample will be useful when learning the new API. The $JAVA_HOME/sample/nio/file directory has a number of other examples that demonstrate other parts of the API.

Thursday Apr 23, 2009

Sockets Direct Protocol

Solaris has support for the InfiniBandTM (IB) Sockets Direct Protocol (SDP) since Solaris 10 5/08. SDP is a wire protocol to support stream sockets networking over IB. It utilizes features such as RDMA for high performance data transfers and provides higher performance and lower latency compared to IP encapsulation over IB (IPoIB). A long standing request has been to add implementation-specific support to the JDK to make it easy to use SDP in Java Language applications. As SDP provides a streams sockets interfaces it can be made transparent to applications using and of course the stream-oriented SelectableChannels and AsynchronousChannels in the channels package.

The simple solution in jdk7 uses a configuration file with rules to indicate the endpoints that are connected to the IB fabric for when SDP should be used. When a or java.nio.channels.SocketChannel binds or connects to an address that is an endpoint on the IB fabric then the socket is converted to use the SDP protocol. The solution can be thought of a kind of built-in interposer library and using an interposer library is an alternative way of doing this. In the future it might be interesting to support factory methods that allow the SDP protocol be selected when creating channels and sockets.

The $JAVA_HOME/lib/sdp directory contains a template configuration file. The format is relatively simple with each non-comment line a rule to indicate when the SDP transport protocol should be used. The configuration file is specified via the system property com.sun.sdp.conf. A second property (com.sun.sdp.debug) can be set to enable debug messages to be printed to System.out, useful to check the configuration. The property value can also be set to the location of a file so that the debug messages are directed to a log file instead. SDP needs to be enabled on Solaris via the sdpadm(1M) command and of course the interfaces need to be plumbed with IP addresses.

Our lab machines have the InfiniBand adapters plumbed with addresses on the 192.168.1.\* network so a simple configuration may be:

$ cat << EOF > sdp.conf
bind \*
bind  5000
connect  1024-\*
connect   1521

This basically says that SDP should be used for applications that bind to or bind to the wildcard address on port 5000. All connections to application services on 192.168.1.\* should use SDP and all Net8 connections to the Oracle database on should use SDP.

Once the configuration file is created we simply specify it when running the application, eg:

$ java -Dcom.sun.sdp.conf=sdp.conf MyApplication

Note that this example also sets the property. The Java Runtime always uses IPv6 sockets if IPv6 is enabled. The Solaris implementation of SDP supports IPv6 addresses but currently doesn't support IPv4-mapped IPv6 addresses (::ffff: for example). For now this means setting the property so that all sockets are IPv4 sockets.

One final thing to mention is that although SDP has been available since Solaris 10 5/08 there are a couple of bugs that the JDK runs into. Thanks to Lida Horn, all these issues have been fixed for Solaris 10 Update 8. They should also be in next OpenSolaris release or build 113 for those downloading the Solaris Express Community Edition.




Top Tags
« April 2014

No bookmarks in folder