Wednesday Aug 08, 2007

File transfer to an MUC

We recently submitted a proposal to XSF to standardize what was a proprietary implementation for message moderation ... which we have had for a couple of years now. The other effort which we should probably kick off soon is file transfer to multi-user chat room - this is another custom implementation which we have been carrying on for a few releases now ...
This has come up time and again in JIG as part of other threads, and yet we never got much farther that 'yep, we should document it'. I know of atleast one other server who have their own implementation to support this .... so maybe it is time we standardized this !
To give a basic idea of what we do:
Currently, our client supports IBB - so does the server, and the muc (which is part of the server). So idea is simple - essentially stream the data from the client to the server, store it locally (if content filtering, etc is enabled) post-process it (if enabled), and then the component streams it to the recipients.
As all the participants who can understand this are Sun IM clients - we just use IBB for now.
The caveats are obvious - potentially very heavy server traffic due to the IBB. But the main advantages are:
  1. The sender transmits only a single copy of the file - in comparison to a p2p model (imagine streaming a 512 KB file to a muc with 100 participants from a moderately low b/w connection).
  2. It will always work for all cases - irrespective of firewall, nat, etc. IBB is the lowest common denominator.
  3. Ability to analyze the content before sending it on to the recipients - including the possibility of archiving it (for participants who might join the room 'later').
There are ofcourse other means of achieving the same while satisfying the requirements above - including other means of file transfer which are yet to be standardized (custom, future additions).
File transfer in general was unfortunately not a very strong point in xmpp, and we are still working on the details ... so I am not very sure if we have to wait for the churn to subside, before proceeding - or start work on this in parallel, so that the requirements for multicasting file(s) to a room are kept in mind while tackling the file transfer problem.

Saturday Jul 28, 2007

Message moderation specs

We recently submitted two proposals for message moderation - Message Moderated Conferences and Managing message moderators in MUC rooms - both of which are extensions to Multi-User Chat (muc) spec.
Taken together, they allow a message moderation system to be put in place. It has been a conscious decision not to introduce another set of acl's for this, but to reuse the affiliations and roles specified in muc. I will try to go through the basic intent and design decisions behind the specs.

Message Moderated Conferences

This proposal specifies how a conference which has message moderation enabled would interact with participants - particularly, participants who dont have 'voice' or the ability to post message to the room.
If message moderation is enabled, these occupants would be able to request the room for approving the messages they want to post - and on approval, room would publish them to all occupants of the room. Simple scenarios would be a celebrity chat, moderated webcast/presentation, etc.
The spec does not deal with how the message moderation happens - just the interaction between a occupant and the room - how they interact, state changes, notifications, etc. So in effect, you could have any number of backend moderation implementations - but the interface between the occupant and the room will remain the same.
The basic flow is as follows :
  1. Occupant sends message to room for approval.
  2. Room assigns a message id to the submitted message (which is then used to identify the submitted message for all further interaction, in this and other moderation related specs) and returns that to the user with message in pending state.
  3. After the backend moderation system decides on the message, room will inform the submitted about the 'decision' - approved, rejected (or error).
  4. If approved, room will then multicast it to all the occupants.
That is it !
The actual moderation might be quite involved, with multiple moderators/moderation modules chained, clustered/distributed rooms, etc in complex deployments - but the occupant will have a simple and clean interface to communicate with the room (room's bare jid actually).

The way we designed it, hopefully this will someday get rolled into XEP 45 itself :-)

Managing message moderators in MUC rooms

This spec defines one possible way in which message moderation can actually be implemented - where the moderation is done by participants who have sufficient rights (owner's and moderators of the room).
It defines how a room moderator/owner can become a message moderator (and other state changes), how they notify room of the moderation decision, and how the room is expected to manage these moderators and submitted messages for moderation.

Both of these specs are based on our implementation - though what we support currently is quite different from what we have submitted.

As a sidenote : It should be noted that, in both the specs we have tried to make sure that the interface between the various entities in question remains as simple as possible - while not precluding more complicated scenarios .. like usecases which are not specifically related to users chatting over a moderated conference (for example an approval workflow) - there could be custom configurations with extensions to room configurations which allow for more complicated scenarios while keeping the actual client interfaces (at both submitter and moderator side) constant.

Monday Mar 26, 2007

S2S connection availability & state recovery

some thoughts on xmpp s2s availability and failover : and lack of protocol support for it.[Read More]

Saturday Dec 16, 2006

In order delivery in xmpp servers

Discusses some rationale behind thoughts on why strict in order delivery maynot be a 'good thing'.[Read More]

Tuesday Nov 28, 2006

server pool and expirimental stuff

One of the things which we introduced with the previous interim release was the concept of server pool.The admin can have a combination of disparate set of boxes - solaris sparc/x86, linux and logically combine them to create a single XMPP deployment.Ofcourse, one deployment could support multiple hosted domains - so typically it is a single deployment : not really a single domain which is hosted on a pool.

For the purpose of minimising internode communication, we have also introduced to concept of an xmpp aware load balancer called redirect server.The actual redirection mechanism is an spi and can be customized, but there are a bunch of out of the box methods - including ones which allow for roster based logical grouping of contacts to nodes.The server pool will redistribute any new load across itself in the face of failures ... So, introducing redundency into the pool not only helps in performance of a single node under normal operation, but allows the pool to operate smoothly when you are faced with network, power, etc outages.

Taken together this results in some interesting usecases.
There is minimal network overhead and yet high degree of failover and scalability can be achieved - we have not really tested to reach the limits of how 'large' a server pool can be .... This essentially means that a service provider can not only support a large number of hosted domains, but also provide high availability for all of them with a single deployment.
As far as the end user is concerned, there is no difference whether he is talking to a server pool or to a single server: the behaviour is uniform and spec compliant.
If you consider the distributed nature of features like pubsub, muc, etc in the server pool - it essentially means that you do not have the concept of failure: unless every node in the pool goes down that is :-)
So essentially, you can use the server as a near-realtime messaging middleware with very high availability and scalability.

And yes, the next release we have does include expirimental support for caps and pep at the server and in client api ... might not be the latest version of the spec though. But developers wishing to hack at it are welcome : this impl is not very rigourously tested - so I am not going to popularise it :-), but we have used it for avataar and seen it working beautifully !
The api which is hosted at collab project of netbeans has the client side support for this (:pserver:anoncvs-AT-cvs.netbeans-DOT-org:/cvs , collab/service/src/org/netbeans)
Btw, did I mention that we have always supported privacy lists ?

Friday Nov 17, 2006

File transfer to a multi user chat

File transfer to a Multi user chat is not defined in the current set of xmpp extensions. Though this does not really describe what our server/client does today, this post discusses a simple means to achieve this required functionality.[Read More]

Wednesday Nov 15, 2006

Corner cases of presence probes in xmpp

Some random thoughts on presence probes in rfc 3921 and the xmpp bis specs ... just random rambling, I could be grossly wrong ![Read More]

Sunday Nov 12, 2006

Security improvements in upcoming Sun IM server release

A small summary of primary features to expect in security and authentication in the upcoming release of Sun's IM server[Read More]

Thursday Nov 09, 2006

Minimising S2S traffic in XMPP - elaboration , clarifications and corrections

Elaborated the idea posted in the previous post along with some clarifications and corrections.[Read More]

Wednesday Nov 08, 2006

Minimising S2S traffic in XMPP

Discusses a potential proposal we are considering to implement which will minimise the interserver traffic tremendously without compromising on security, responsibility and accountability.[Read More]

Monday Aug 28, 2006

AJAX and HTTP - transport management : JEP 124 ?

As more and more websites and applications start moving to using the AJAX paradigm , people are going to hit this problem of how to use HTTP as a transport 'reliably'.
What I mean is , HTTP is essentially similar to a one way datagram approach to communication.
You send request A , wait for response A - move on.
Since this need not be a direct connection from client to server (even if it was) , the request could fail to be delivered , response could be 'lost' , connection errors , etc could intervene.
If they are idempotent methods being made , then you can reissue them in case the request fails for whatever reason.
If not , things are slightly trickier - but there is some sort of support in most frameworks on how to handle this currently.
Add HTTP/1.1 persistent connections to this mix and things get really interesting - one socket fails : and a whole bunch of requests are lost.
Also, AJAX typically issues POST in most cases - a non-idempotent method.
Add to it the requirement to 'poll' the backend for data or updates and you have a interesting problem : you are trying to simulate a HTTP client into being something similar to a bidirectional stream : simulate is a key word here , mind you !

So you have requirements to 'manage HTTP as a transport' coming both at the client side and at the server side.
In the XMPP world , when support for HTTP was initially added the first approach was a pure polling solution.
As can be guessed , they later developed the HTTP Binding solution under JEP 124.

This is a very very interesting extension and I believe is the key to solving most of the problems above.
Though defined for XMPP , it is a solution to essentially act as a transport for any xml based protocol : only requirement, one or more "full xml stanza's" are exchanged - not partial xml data.
It solves the problems of retransmission , session management (no need for cookie's , url rewriting , etc) , support against replay attacks , etc.
Heck , we have XMPP - a purely xml based streaming protocol working on top of this successfully !

How does this help AJAX ?
All the browser based XMPP clients which support this JEP have some variants or other javascript library in place to actually do all this.
So , potentially you could have :
<your application> --> <jep 124 client side js library>   --- HTTP ---> <HTTP Bind gateway> <-- Custom protocol --> <Your server>
With suitable modification (IF your app is not XMPP that is :-) ... why not move to it? Read prev post on some advantages) , you can use the ideas from JEP 124 to solve a whole bunch of problems with using HTTP as a reliable transport.
Add to it the fact that you already have server side and client side opensource component's available to support this - and you have the perfect solution just waiting to be used.

Happy coding !

Thursday Aug 24, 2006

XMPP as an infrastructure

  In this post , I briefly try to push how XMPP can be used as a infrastructure - and not just as 'yet another chat protocol'.
It is not as descriptive or authoritative as I would like it to be ... but here goes nothing.

The abstract from RFC 3920 says :
"This memo defines the core features of the Extensible Messaging and Presence Protocol (XMPP), a protocol for streaming Extensible Markup Language (XML) elements in order to exchange structured information in close to real time between any two network endpoints. While XMPP provides a generalized, extensible framework for exchanging XML data, it is used mainly for the purpose of building instant messaging and presence applications that meet the requirements of RFC 2779."

For the purpose of this discussion , let us take a small subset of what XMPP provides which allows us this :
  1. As specified in section 4 of 3920 , an xml stream between two entities which allows for exchange of structured xml stanza's at realtime.
  2. A binding to tcp and defines how to secure and authenticate the stream.
  3. Defines the top level stanzas allowed in this xml stream. These model request-response , broadcast and one way meps.
  4. Server federation , routing rules , multiple resources per node built into the protocol.
Let us consider the implications of these.
We have an authenticated and secure stream between two entities - so this asserts the client identity.
This explictly identifies an entity within , not only its server - but if federated , across the XMPP universe of federated servers ! This guarantees that any stanza orginating anywhere in this grand federation of XMPP servers will be unambigously routed to this specific client endpoint
Defines an xml stream ... most data transfer protocols have already moved , or are moving towards standardizing on usage of xml : so XMPP provides a native transport mechanism for these.
The MEP's supported allows us both synchronous communication and asynchronous notification - in either direction: hence both push and pull models are natively supported.
The protocol itself allows for presence notifications through a publish subscribe model.

Now , the XMPP community has come up with a variety of protocol extensions which have been standardized as JEP's.
A few notable ones which are relevent for this discussion are :
  • A flexible discovery mechanism : allowing various clients/servers to interoperate. It helps either side to discover the supported subset and allows protocol extension without breaking compatibility.
  • Multi User Chat - though it defines how to set up a 'chat room' : it can be used or modeled to multicast message's to a subset of interested entities.
  • A powerful publish subscribe extension for XMPP.
  • Advanced message processing allows specifying additional delivery semantics for messages.
  • HTTP Binding - allowing XMPP to be bound to HTTP. (More on this later)
All this taken together allows for providing solutions for some challenging problems.
We could use xmpp as a middleware for realtime communication and use it for both synchronous and asynchronous notifications. Some examples :
  1. Pushing information to interested entities - like stock quotes to customers or brokers , etc.
  2. A whole bunch of publish subscribe usecases can be solved.
  3. Multicast stanza's as messages to interested entities.
  4. Allows us to write presence aware applications - the entity in question need not really be another 'person' ! Also , use of CAPS and PeP will definitely make this world of applications much more interesting !
Now , add JEP 124 (httpbind) into this picture and things become really interesting.
Httpbind essentially can be considered as way to tunnel a xml stream based protocol on top of http : no , not the old CONNECT hack that most implementations use , nor a dumb frequest poll mechanism.
As an analogy , you can consider it to be similar to how you would simulate a persistent stream on top of UDP - only it is HTTP here instead of UDP.
The extension takes care of retransmissions , packet loss , request/response id , etc and makes allowances for HTTP characteristics .

Hence , all the power of XMPP is now available over HTTP.
So be it a rich client application or a rich internet application - both can talk XMPP and harness its power !
And with it , allows all usecases of using XMPP as a realtime messaging middleware available to all developers/users on most platforms - be it a RIA or a thick client.

XMPP - coming to a browser near you :-)

Thursday Jul 27, 2006

XMPP Interop event

Just got back yesterday from Portland after the XMPP interop event ... it was a really cool event.
We had 6 completely different servers interoperating without too much of a problem - including when some funky multi-byte testcases were tried in S2S which stpeter dreamed up :-) There were some hiccups when TLS was enabled with SASL External / dialback ... but nothing too serious which could not be fixed.
The achievments of the event are pretty incredible - there is no other protocol on instant messaging which can possibly boast of achieving something like this - most dont even have an implementation of size more than 1 !

While the testing was going on , in parallel we had some very productive discussions , including to clarify on aspects of the RFC's and JEP's : what was debated upon would get translated into changes or discussions at standards-jig pretty soon !
Portland looked "pretty" - from what little I saw , and the event venue itself allowed for a real nice view of the downtown.
I could not attend OSCON unfortunately ...

Tuesday May 09, 2006

No updates

Unfortunately , caught up with beta build for JES5 ... so that means not much time to blog and even less to work on xmpp-im-client :-(
I checked in a barebones structure there - no code , just a build file which pulls the required dependency.

Will get to posting some actual code this weekend ... so will keep all posted !
In the meantime , feel free to join the project like others have done already :-)

Monday May 01, 2006

Matisse for eclipse

Have you seen Matisse ? Most probably you have ! It is an award winning GUI builder which comes with netbeans.
It is one kickass GUI builder ... and even a person like me can create a UI using it :-)

Now , eclipse folks have long since been clamouring for an equivalent ... and looks like they are finally getting their wish !
Matisse4MyEclipse is out ... hope it is as good as the netbeans version.




« July 2016

No bookmarks in folder