Sockets Direct Protocol
By user12820862 on Apr 23, 2009
Solaris has support for the InfiniBandTM (IB) Sockets Direct Protocol (SDP) since Solaris 10 5/08. SDP is a wire protocol to support stream sockets networking over IB. It utilizes features such as RDMA for high performance data transfers and provides higher performance and lower latency compared to IP encapsulation over IB (IPoIB). A long standing request has been to add implementation-specific support to the JDK to make it easy to use SDP in Java Language applications. As SDP provides a streams sockets interfaces it can be made transparent to applications using java.net.Socket/ServerSocket and of course the stream-oriented SelectableChannels and AsynchronousChannels in the channels package.
The simple solution in jdk7 uses a configuration file with rules to indicate the endpoints that are connected to the IB fabric for when SDP should be used. When a java.net.Socket or java.nio.channels.SocketChannel binds or connects to an address that is an endpoint on the IB fabric then the socket is converted to use the SDP protocol. The solution can be thought of a kind of built-in interposer library and using an interposer library is an alternative way of doing this. In the future it might be interesting to support factory methods that allow the SDP protocol be selected when creating channels and sockets.
The $JAVA_HOME/lib/sdp directory contains a template configuration file. The format is relatively simple with each non-comment line a rule to indicate when the SDP transport protocol should be used. The configuration file is specified via the system property com.sun.sdp.conf. A second property (com.sun.sdp.debug) can be set to enable debug messages to be printed to System.out, useful to check the configuration. The property value can also be set to the location of a file so that the debug messages are directed to a log file instead. SDP needs to be enabled on Solaris via the sdpadm(1M) command and of course the interfaces need to be plumbed with IP addresses.
Our lab machines have the InfiniBand adapters plumbed with addresses on the 192.168.1.\* network so a simple configuration may be:
$ cat << EOF > sdp.conf bind 192.168.1.1 \* bind 0.0.0.0 5000 connect 192.168.1.0/24 1024-\* connect dbcluster.foo.com 1521 \^D
This basically says that SDP should be used for applications that bind to 192.168.1.1 or bind to the wildcard address on port 5000. All connections to application services on 192.168.1.\* should use SDP and all Net8 connections to the Oracle database on dbcluster.foo.com should use SDP.
Once the configuration file is created we simply specify it when running the application, eg:
$ java -Dcom.sun.sdp.conf=sdp.conf -Djava.net.preferIPv4Stack=true MyApplication
Note that this example also sets the java.net.preferIPv4Stack property. The Java Runtime always uses IPv6 sockets if IPv6 is enabled. The Solaris implementation of SDP supports IPv6 addresses but currently doesn't support IPv4-mapped IPv6 addresses (::ffff:192.168.1.1 for example). For now this means setting the property so that all sockets are IPv4 sockets.
One final thing to mention is that although SDP has been available since Solaris 10 5/08 there are a couple of bugs that the JDK runs into. Thanks to Lida Horn, all these issues have been fixed for Solaris 10 Update 8. They should also be in next OpenSolaris release or build 113 for those downloading the Solaris Express Community Edition.