Tuesday Jan 29, 2008

JXTA Micro Edition MIDP-2.0

Last Wednesday (January 23rd), JXTA Micro Edition MIDP-2.0 was open sourced under http://jxta-jxme.dev.java.net.  Up until now JXTA on mobile devices had been pretty limited in functionality, however with the introduction of this new binding, a mobile device can participate fully as an edge device on the network without the need of a proxy.   So what does this mean you ask?

 With this new binding, a mobile device can :

  • Discover other devices and services
  • Publish advertisements about it's own resources
  • Establish direct connections to any other node
  • Create/Join private virtual domains
  • Exchange/Access content within the domain without the need of a central server(s)/service(s)

Imagine audio streaming from a home machine to a phone, or contact and calendar syncing within a virtual domain (home, office, and mobile), or true mobile-2-mobile gaming.   Imagine no further, the networking foundation is now available to enable such capabilities.

Thursday Nov 29, 2007

Shoal (A Dynamic Scalable Clustering) white paper posted

In my previous post I gave an overview of the feature set provided by Shoal. The blog did not provide enough details about the clustering foundation.  I recently committed a white paper about Shoal's clustering, which should provide more details.

Saturday Oct 06, 2007

Dynamic Scalable Clustering

In today's information technology world, fault tolerance has become an expected system characteristic, as demands on such systems, not only requires the availability of data, but also the efficiency of such systems.

By clustering a set of servers, with minimal, or no configuration,  through a dynamic discovery protocol, a compute cluster can be formed to increase compute power, availability, security, and geographical distribution.  

As a fault tolerance strategy, the Shoal clustering framework devises a self organization protocol, allowing nodes to autonomously elect a master for the cluster (based on discovery views), as well as candidates in the case of master failure.  This allows for dynamic deployment of clusters as well as expanding/shrinking an existing cluster.

Shoal, the cluster management framework, provides the foundation for network configuration and dynamic and autonomous cluster formation. The goals of the framework are:
  • Near Zero-configuration
  • Multi cluster support
    • Application specific clustering, eg. load balancer/HA cluster
    • Cluster traffic isolation
       Multi Cluster Support
    • Added security
    • improved resource utilization
  • Dynamic clustering
    • Hot pluggable
  • Simplified addressing
    • By cluster, node, and channel name
  • Adaptive communication channels
    • Fault tolerance
    • Transparent multi protocol support  (tcp, udp, http)

Shoal 1.0 was release August 30th, and is utilized in GlassFish V2, which was released September 17th, for clustering and http session HA. 

Currently the following features are being worked on :

  • Enabling multi cluster support
  • Enabling cross sub-net and cross firewall  deployment
  • Solidifying lightweight cluster multicast
We encourage community participation in enabling and identifying missing features by joining and participating in the shoal community.

Wednesday May 23, 2007

Recent optimization in propagated pipes

There has been many performance enhancements introduced in the upcoming release of JXTA for JSE 2.5, which I will try to cover in a series of blogs detailing the enhancements and the expected benefits. Starting with the most recent, the propagated pipe.  To give a little background on how propagated pipes work, a propagated pipe utilizes a pipe service named WirePipe which deals with duplicate message filtering and TTL on the network.  The WirePipe service then relies on the rendezvous service for network topology and message distribution.   A less frequently publicized feature of propagated pipes, is the ability to specify a set of recipients, which eventually end up as individual messengers (1-1 connections) to recipients.  The performance enhancement was introduced as a new class, BlockingWireOuputPipe, in 2.5, which is automatically constructed when the specified set of nodes contains a singleton. BlockingWireOuputPipe then binds directly the destinations propagated pipe endpoint bypassing two layers of indirection (WirePipe and the RendezvousService), thus eliminating two message elements (wire, and propagation header),  resulting in smaller message overhead, and faster delivery, also since it is bound directly to the endpoint, it provides reliable feedback on message delivery.

It is highly recommended that such object is cached and reused, for an example, see the PropagatedPipeServer tutorial. 



Wednesday Oct 25, 2006

Java logging Filter Interface

Recently, I switched from using log4j to java logging, and I needed a way to selectively constrain logging to a specific set of classes, or methods.  So I came up with the following example. Since there are a few examples on the java logging Filter interface, I am sharing this example.

Hopefully it's useful to others.
SelectiveLogFilter filter = new SelectiveLogFilter();

 \* A simple logging filter to allow multi per class logging
public class SelectiveLogFilter implements Filter {
    private HashSet classLogSet = new HashSet();
    private HashSet methodLogSet = new HashSet();

     \* Adds a name to the set of loggable classes
     \* @param  className  canonical class name e.g. App.class.getName()
    public void add(String className) {

     \* Removes a name from the set loggable classes
     \* @param  className  canonical class name e.g. App.class.getName()
    public void remove(String className) {

     \* Adds a method name to the set of loggable classes
     \* @param  methodName  canonical method name e.g.
     \* App.class.getName()+"methodName"
    public void addMethod(String methodName) {

     \* Removes a name from the set loggable classes
     \* @param  methodName  canonical method name e.g.
     \* App.class.getName()+"methodName"
    public void removeMethod(String methodName) {

     \* Clears the loggable Class Set (effectively turns off logging)
    public void reset() {

     \* Clears the loggable Method Set
    public void resetMethodSet() {

     \* {@inheritDoc}
    public boolean isLoggable(LogRecord record) {
        return classLogSet.contains(record.getSourceClassName()) || 

Saturday Jun 24, 2006

NetworkConfigurator. A programmatic interface to JXTA configuration

If you are familiar with JXTA configuration, you will surely appreciate the new bug fixes, and features added to the latest JXTA platform release (2.4).

The latest release included significant fixes and features to deal with node configuration:
  1. ext:config included several bug fixes, and improved stability, it also included additional configuration profiles.
  2. A programmatic interface to node configuration (see NetworkConfigurator), by design, NetworkConfigurator is a JavaBean, allowing for declarative configuration deployment (see jxta-spring).

    In addition NetworkConfigurator exposes API's (see load(uri), getPlatformConfig(), etc.), which would facilitate secure network based deployment services, as well as automate certificate, ID, and rendezvous/relay seed provisioning.

    Special thanks to the community members who contributed and helped drive these new API's (especially vwilliams, ae6rt, bondolo, fdimglio, ibirrer)
Network Based Configuration Provisioning Network Based Configuration Provisioning
   NetworkConfigurator config = new NetworkConfigurator();
if (!config.exists()) {
// Create a new configuration with a new name, principal, and pass
config.setName("New Name");
try {
//persist it
} catch (IOException io) {
// deal with the io error
} else {
// Load the pre-existing configuration
File pc = new File(config.getHome(), "PlatformConfig");
try {
// make changes if so desired
// store the PlatformConfig under the default home
} catch (CertificateException ce) {
// In case the root cert is invalid, this creates a new one
try {
//password to encrypt private key with
} catch (Exception e) {

Thursday May 25, 2006

JxtaSocket vs. Java Socket

If you have wondered about JxtaSocket performance vs. Java Socket then this maybe of interest. I have always wondered about how JxtaSockets stack up to Java Sockets (there has some research done on this topic, but it's not the same unless you can see for yourself). So, I have decided to write a benchmark comparing the two. The benchmark included two machines connected through a 100Mbit switch, and JSE5.0. Each benchmark run included a data transfer of 65MB, which was repeated 3 times. The chart shows the lowest and highest rates measured. You should also note, that the transfer unit size, and the VM min heap size had an effect on both benchmarks.JxtaSocket vs. Java Socket Benchmarks.



Wednesday Jan 04, 2006

JXTA's newest member of the socket family "JxtaDatagramSocket"

In the spirit of providing JXTA interfaces a la java.net.Socket, I would like to introduce you to the newest member to the net.jxta.socket family "JxtaDataDramSocket", which can be found under http://platform.jxta.org/issues/show_bug.cgi?id=1470 Please provide comments and feedback under the issue described above.

Thursday Oct 20, 2005

Are you interested in a UDP transport for JXTA?

If you are interested in a JXTA UDP transport. an early copy of the transport is available under http://platform.jxta.org/issues/show_bug.cgi?id=1448 It would be interesting to benchmark JXTA's tcp transport against the new UDP transport.

Friday Oct 14, 2005

CDC 1.1 RI is now available

See: http://java.sun.com/products/cdc/ For detailed information on how to download the binaries

Thursday Sep 15, 2005

JXTA for J2ME JXME 2.1 (Tantà) Screen Shots

I captured the following images during the JXME 2.1 functional test against the JXTA public network.
        JXME 2.1 HelloWorld JXME 2.1 HelloWorld

        JXME 2.1 group chat session within MyJXTA's sub-group JXME 2.1 group chat session within MyJXTA's sub-group

Wednesday Sep 14, 2005

JXME 2.1 (Tantà) on an iPAQ 3600

I had promised to provide detailed step-by-step instructions on installing and running JXME on an iPAQ. So here goes. Hardware
  • iPAQ 3600
    • 64MB RAM
    • 32MB flash
    • Orinoco 802.11B
build and install JXME 2.1
  • set optimize on, and debug off
  • build against log4j_stub
  • copy jxme-cdc.jar, and log4j_stub.jar to the device
Installing handhelds.org familiar
  • Download and install handhelds familiar opie release v0.8.2 image for your device
  • Follow the excellent installation instructions on handhelds.org
  • Once booted
  • Install libfloat
  • ipkg install libfloat_1.0_arm.ipkg
  • cd /usr/lib
  • ln -s /home/QtPalmtop/lib/libfoat.so .
  • install cvm 1.1
    • cp -rf cvm /tmp (or any directory with available space)
Running the demos
    • cvm -classpath \\ jxme-cdc.jar:log4j_stub.jar GroupDemo
    • cvm -classpath \\ jxme-cdc.jar:log4j_stub.jar ChatDemo

Tuesday Aug 30, 2005

JXTASink. Peer emulation, and load generation

JxtaSink and companion utilities are some of the least publicized tools available underneath the JXTA platform hood, however don't underestimate their power. JxtaSink emulates the JXTA endpoint protocol, and provides API for message callback, or it can act as a message drain. Such features are beneficial when trying simulate a network of 1000's nodes on a single machine, and in determining breaking, or tuning points. Using JxtaSink requires knowledge of the JXTA protocols and behavior. One test developed which makes use of these utilities is the RendezvousLoad. RendezvousLoad emulates the rendezvous connection protocol and can scale to 1000's peers (on certain platforms). This load test was beneficial in exposing unforeseen OS limitation, as well as bottlenecks. Similar load tests can be written using the forementioned utilities, provided good understanding of protocol of interest. JxtaSink resides in the platform project under platform/binding/java/test/src/net/jxta/test/util

Wednesday Aug 24, 2005

Demystifying Pipes, JxtaSockets, JxtaMulticastSocket, and JxtaBiDiPipes

Lately there has been several inquiries about JXTA's PipeService, and companion utilities (JxtaSocket, JxtaMulticastSocket, and JxtaBiDiPipe) on JXTA's discussion lists, hence this blog to shed more light on the PipeService and utilities provided, and their inherit features.
The initial goal of the PipeService was to provide primitive (unidirectional, and unreliable) message based communication channels upon which complex and sophisticated communication channels can be built. The PipeService provides three types of pipes described as follows:
  • Unidirectional
  • Unreliable
  • Many to one
  • Limited to 64KB messages
  • Unidirectional
  • Limited reliability (no direct notification of failures other than channel failure)
  • Many to one
  • Limited to 64KB messages
  • Multidirectional (many to many)
  • Unreliable
  • Limited to 64KB messages in infrastructure mode, and limited by physical transport limitation where applicable
JXTA JSE platform software stack JXTA JSE platform software stack
Unicast pipes (clear, and secure) are bound by the PipeResolver, which provides two modes of bindings, a static (where a set of peers is specified), and a dynamic (no peers specified) binding. Static binding is beneficial in constraining communication to a limited set of peers, while dynamic is beneficial in implementing unconstrained mobile communication channels. Both modes can be utilized in implementing fault tolerant communication channels.
Propagated pipes employ a mechanism similar to that of IGMP, where a creation of an input pipe results in propagated pipe channel join message (a pipe SRDI message to a the peer's rendezvous). In infrastructure mode propagated pipe messages are sent to the rendezvous for propagation to the subscribers of the propagated pipe channel, while in ad-hoc mode messages are propagated over the endpoint (currently only the TCP transport supports such function over multicast). A pipe closure results in a channel resign message (a pipe SRDI message to the rendezvous expiring the previous join). It's worth noting that the PipeResolver plays no part in binding of such pipe type
Pipe Binding An InputPipe creation results in the creation of an endpoint incoming messenger, and an SRDI message with the pipe ID with an infinite expiration time. Once the InputPipe is closed another SRDI message is emitted with the pipe ID, and expiration time of 0, invalidating any reference for the pipe and peer. note: This behavior applies to all pipe types. An output pipe creation results in a PipeResolver query message emitted querying for instances of the pipe. A match results in a PipeResolver response to the querying peer, where an endpoint messenger to responding peer is created. Note that such messenger may not be fully resolved until the EndpointRouter has determined a valid route. During such event few messages maybe queued (3 to be exact) until the messenger has been resolved. In addition, as part of the pipe resolver protocol the pipe resolver periodically reattempts pipe resolutions in the background to validate resolved pipe endpoints. It is through this mechanisms pipes can migrate from one peer to another. note: this behavior is limited to unicast pipe types.
SRDI message path within the JXTA network SRDI message path within the JXTA network
Pipe Endpoint A pipe endpoint is composed of a peer, group, and pipe ID, through this definition there are two inherit modes of pipe endpoint migration:
  1. Peer endpoint migration, an inherit feature of the Endpoint-Router, where a peer may change physical, or virtual addresses.
  2. Pipe endpoint migration, where a pipe endpoint ceases to exists on a peer, where the PipeResolver, attempts to rebind to the pipe (which I always refer to as a virtual channel)
The first mode is transparent the application, while the second may result in an IOException. Reliability A standalone message and stream based reliability layer over non reliable messengers, is provided in the platform to facilitate reliable communication channels. Reliability Library
  • Ensures message sequencing
  • Interfaces to pipes or messengers
  • Ensures delivery
  • Exposes message, and stream interfaces
  • Spawns a single thread to deal with retransmission
  • Does not ensure message integrity
Bidirectional communication channels JxtaServerSocket, and JxtaServerPipe expose a input pipe to process connection requests and negotiate communication parameters, whereby a JxtaSocket, or JxtaBiDipipe bind to respectively to establish private dedicated pipes independent of the connection request pipe. JxtaSocket, and JxtaBiDiPipe utilize a messenger instead of an output pipe for the back channel to reduce connection latency (avoids 4 messages), therefore only peer endpoint migration is inherited, and one should always expect an IOException if a PipeEndpoint migrates, or ceases to exist.
JxtaSocket, JxtaServerSocket
  • Subclass java.net.Socket, and java.net.ServerSocket respectively
  • Built on top of pipes, endpoint messengers, and the reliability library
  • Provides bidirectional and reliable communication channels
  • Exposes stream based interface a la Socket
  • Provides configurable internal buffering, and message chunking
  • Does not implement the Nagels algorithm, therefore streams must be flushed as needed (a common oversight when wrapping object and data streams over the socket stream)
JxtaBiDiPipe, JxtaServerPipe
  • Built on top of pipes, endpoint messengers, and the reliability library
  • Provides bidirectional and reliable communication channels
  • Exposes message based interface
  • Requires no heart beat to maintain an open connection
  • Provides no message chunking
Multidirectional communication utility JxtaMulticastSocket extends java.net.MutlicastSocket and provides mutlicast functionality over JXTA propagated pipes. JxtaMulticastSocket
  • Subclass java.net.MulticastSocket
  • Built on top of propagated pipes
  • Exposes datagram based interface a la MulticastSocket
  • Provides unicasting through datagram addressing
Message size and limiting factors
  1. The JXTA platform imposes a soft MTU of 64KB (which becomes a hard limit under load), messages exceeding MTU are simply dropped. While nodes maybe able to transmit and receive messages larger than 64KB, it is strongly advised this limit is not exceeded in reliable mode, as it will result in failed or broken channels.
  2. When utilizing relay peers, such limit must not be exceeded in either mode (reliable or otherwise) as the relay nodes are likely to be loaded and/or the fairness message queue size quotas are enforced, thus leading to message loss and consequently failed or broken channels.
Common pitfalls
  • Pipe advertisement mismatch. Typically occurs when dynamically creating and discovering pipe advertisements using non unique identifiers, or deterministic method. Normally overcome through hardcoded pipe advertisements (or pipe ID's), encoded pipe ID's.
  • Un-synchronized pipe binding. Typically caused by network islands, which are joined at a later point in time. Easily overcome through rendezvous event listener prior to any binding attempt.
  • Failed Secure pipe connections. Typically due to failure of establishing NetPeerGroup credentials (TLS resides in the NetPeerGroup, and inherited by all sub-groups), or missing a trusted certificate or certificate chain in PSE. This is overcome by stablishing NetPeerGroup credential, and ensuring the perspective trusted certificate or certificate chain is stored into PSE (See the shell command pse.importcert).

Saturday Jul 30, 2005

JxtaNetMap gets a new face-lift

JxtaNetMap is a pretty useful network debugging tool, however you would probably agree that the color choices and arrow representation were kind of dated (reminiscent of the 70's). Since I use the tool often I decided to give it a little face-lift. Some of the changes were simple, such as choosing a color scheme, and node text spacing, others were a bit more involved and entailed changing the arrow drawing algorithm. I hope you like the new look. New look 





« April 2014