Tuesday Nov 11, 2008

REST APIs must be hypertext driven

Roy Fielding recently wrote in "REST APIs must be hypertext-driven"

I am getting frustrated by the number of people calling any HTTP-based interface a REST API. Today's example is the SocialSite REST API. That is RPC. It screams RPC. There is so much coupling on display that it should be given an X rating.

That was pretty much my thought when I saw that spec. In a comment to his post he continues.

The OpenSocial RESTful protocol is not RESTful. It could be made so with some relatively small changes, but right now it is just wrapping RPC results in common Web media types.

Clarification of Roy's points

Roy then goes on to list some key criteria for what makes an application RESTful.

  • REST API should not be dependent on any single communication protocol, though its successful mapping to a given protocol may be dependent on the availability of metadata, choice of methods, etc. In general, any protocol element that uses a URI for identification must allow any URI scheme to be used for the sake of that identification.

    In section 2.2 of the O.S. protocol we have the following JSON representation for a Person.

        "id" : "example.org:34KJDCSKJN2HHF0DW20394",
        "displayName" : "Janey",
        "name" : {"unstructured" : "Jane Doe"},
        "gender" : "female"

    Note that the id is not a URI. Further down in the XML version of the above JSON, it is made clear that by appending "urn:guid:" you can turn this string into a URI. By doing this the protocol has in essence tied itself to a URI scheme, since there is no way of expressing another URI type in the JSON - the JSON being the key representation in this Javascript specific API by the way, the aim of the exercise being to make the writing of social network widgets interoperable. Furthermore this scheme has some serious limitations such as for example that it limits one to 1 social network per internet domain, is tied to a quite controversial XRI spec that has been rejected by OASIS, and does not provide a clear mechanism for retrieving information about it. But that is not the point. The definition of the format is tying itself unnecessarily to a URI scheme, and moreover one that ties one to what is clearly a client/server model.

  • A REST API should not contain any changes to the communication protocols aside from filling-out or fixing the details of underspecified bits of standard protocols, such as HTTP's PATCH method or Link header field.
  • A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types. Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type (and, in most cases, already defined by existing media types). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]

    Most of these so called RESTful APIs spend a huge amount of time specifying what response a certain resource should give to a certain message. Note for example section 2.1 entitled Responses

  • A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations. [Failure here implies that clients are assuming a resource structure due to out-of band information, such as a domain-specific standard, which is the data-oriented equivalent to RPC's functional coupling].

    In section 6.3 one sees this example:

    /activities/{guid}/@self                -- Collection of activities generated by given user
    /activities/{guid}/@self/{appid}        -- Collection of activities generated by an app for a given user
    /activities/{guid}/@friends             -- Collection of activities for friends of the given user {guid}
    /activities/{guid}/@friends/{appid}     -- Collection of activities generated by an app for friends of the given user {guid}
    /activities/{guid}/{groupid}            -- Collection of activities for people in group {groupid} belonging to given user {uid}
    /activities/{guid}/{groupid}/{appid}    -- Collection of activities generated by an app for people in group {groupid} belonging to given user {uid}
    /activities/{guid}/@self/{appid}/{activityid}   -- Individual activity resource; usually discovered from collection
    /activities/@supportedFields            -- Returns all of the fields that the container supports on activity objects as an array in json and a repeated list in atom.

    For some reason it seems that this protocol does require a very precise lay out of the patterns of URLs. Now it is true that this is then meant to be specified in an XRDS document. But this document is not linked to from any of the representations as far as I can see. So there is some "out of band" information exchange that has happened and on which the rest of the protocol relies. Furthermore it ties the whole service again to one server. How open is a service which ties you to one server?

  • A REST API should never have "typed" resources that are significant to the client. Specification authors may use resource types for describing server implementation behind the interface, but those types must be irrelevant and invisible to the client. The only types that are significant to a client are the current representation's media type and standardized relation names. [ditto]

    Now clearly one does want to have URIs name resources, things, and these things have types. I think Roy is here warning against the danger that expectations are placed on types that depend on the resources themselves. This seems to be tied to the previous point that one should not have fixed resource names or hierarchies as we saw above. To see how this is possible check out my foaf file:

    $ cwm http://bblfish.net/people/henry/card --ntriples | grep knows | head
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://axel.deri.ie/~axepol/foaf.rdf#me> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://b4mad.net/FOAF/goern.rdf#goern> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://bigasterisk.com/foaf.rdf#drewp> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://crschmidt.net/foaf.rdf#crschmidt> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://danbri.org/foaf.rdf#danbri> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://data.boab.info/david/foaf.rdf#me> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://davelevy.info/foaf.rdf#me> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://dblp.l3s.de/d2r/page/authors/Christian_Bizer> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://dbpedia.org/resource/James_Gosling> .
        <http://bblfish.net/people/henry/card#me>     <http://xmlns.com/foaf/0.1/knows> <http://dbpedia.org/resource/Roy_Fielding> .

    Notice that there is no pattern in the URIs to the right. (As it happens there are no ftp URLs there, but it would work just as well if there were). Yet the Tabulator extension for Firefox knows from the relations above alone that (if it believes my foaf file of course) the URIs to the right refer to people. This is because the foaf:knows relation is defined as

    @prefix foaf: <http://xmlns.com/foaf/0.1/> .
    foaf:knows  a rdf:Property, owl:ObjectProperty;
             :comment "A person known by this person (indicating some level of reciprocated interaction between the parties).";
             :domain <http://xmlns.com/foaf/0.1/Person>;
             :isDefinedBy <http://xmlns.com/foaf/0.1/>;
             :label "knows";
             :range foaf:Person .

    This information can then be used by a reasoner (such as the javascript one in the tabulator) to deduce that the resources pointed to by the URIs to the right and to the left of the foaf:knows relation are members of the foaf:Person class.

    Note also that there is no knowledge as to how those resources are served. In many cases they may be served by simple web servers sending resources back. In other cases the RDF may be generated by a script. Perhaps the resources could be generated by java objects served up by Jersey. The point is that the Tabulator does not need to know.

    Furthermore, the ontology information above is not out of band. It is GETable at the foaf:knows URIs itself. The name of the relation links to the information about the relations, which gives us enough to be able to deduce further facts. This is hypertext - hyperdata in this case - at its best. Compare that with the JSON example given above. There is no way to tell what that JSON means outside of the context of the totally misnamed 'Open Social RESTful API'. This is a limitation of JSON, or at least this name space less version. One would have to add a mime type to the JSON to make it clear that the JSON had to be interpreted in a particular manner for this application, but I doubt most JSON tools would know what to do with mime typed JSON versions. And do you really want to go through a mime type registration process every time a social networking application wants to add a new feature or interact with new types of data?

    as Roy summarizes in one one of the replies to this blog post:

    When representations are provided in hypertext form with typed relations (using microformats of HTML, RDF in N3 or XML, or even SVG), then automated agents can traverse these applications almost as well as any human. There are plenty of examples in the linked data communities. More important to me is that the same design reflects good human-Web design, and thus we can design the protocols to support both machine and human-driven applications by following the same architectural style.

    To get a feel of this it really helps to play with other hyperdata applications, other than ones residing in web browsers The semantic address book is one such, that I spent some time writing.

  • A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user‚Äôs manipulation of those representations. The transitions may be determined (or limited by) the client's knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]

    That is the out of band point made previously, and confirms the point made about the danger of protocols that depend on URI patterns or resources that are somehow typed at the protocol level. You should be able to pick up a URI and just go from there. With the tabulator plugin you can in fact do just that on any of the URLs listen in my foaf file, or in other RDF.

What's the point?

Engineers under the spell of the client/server architecture, will find some of this very counter intuitive. This is indeed why Roy's thesis, and the work done by the people who engineered the web before that and whose wisdom is distilled in various writings by the Technical Architecture Group did something that was exceedingly original. These very simple principles that can feel unintuitive to someone who is not used to thinking at a global information scale, make a lot of sense when you do come to think at that level. When you do write such an Open system, that can allow people to access information globally, you want it to be such that you can send people a URI to any resource you are working with, so that both of you can speak about the same resource. Understanding what the resource that URL is about should be found by GETting the meaning of the URL. If the meaning of that URL depends on the way you accessed it, then you will no longer be able to just send a URL, but you will have to send 8 or 9 URLs with explanations on how to jump from one representation to the other. If some out of band information is needed to understand that one has to inspect the URL itself to understand what it is about, then you are not setting up an Open protocol, but a secret one. Secret protocols may indeed be very useful in some circumstances, and so as Roy points out may non RESTful ones be:

That doesn’t mean that I think everyone should design their own systems according to the REST architectural style. REST is intended for long-lived network-based applications that span multiple organizations. If you don’t see a need for the constraints, then don’t use them. That’s fine with me as long as you don’t call the result a REST API. I have no problem with systems that are true to their own architectural style.
but note: it is much more difficult for them to make use of the network effect: the value of information grows exponentially with its ability to be linked to other information. In another reply to a comment Roy puts this very succinctly:
encoding knowledge within clients and servers of the other side’s implementation mechanism is what we are trying to avoid.

Monday Nov 10, 2008

Possible Worlds and the Web

Tim Berner's Lee pressed to define his creation said recently (from memory): "...my short definition is that the web is a mapping from URI's onto meaning".

Meaning is defined in terms of possible interpretations of sentences, also known as possible worlds. Possible Worlds under the guise of the 5th and higher dimensions are fundamental components of contemporary physics. When logic and physics meet we are in the realm of metaphysics. To find these two meet the basic architecture of the web should give anyone pause for thought.

The following extract from RDF Semantics spec is a good starting point:

The basic intuition of model-theoretic semantics is that asserting a sentence makes a claim about the world: it is another way of saying that the world is, in fact, so arranged as to be an interpretation which makes the sentence true. In other words, an assertion amounts to stating a constraint on the possible ways the world might be. Notice that there is no presumption here that any assertion contains enough information to specify a single unique interpretation. It is usually impossible to assert enough in any language to completely constrain the interpretations to a single possible world, so there is no such thing as 'the' unique interpretation of an RDF graph. In general, the larger an RDF graph is - the more it says about the world - then the smaller the set of interpretations that an assertion of the graph allows to be true - the fewer the ways the world could be, while making the asserted graph true of it.

A few examples may help here. Take the sentence "Barack Obama is the 44th president of the U.S.A". There are many many ways the world/universe/complete 4 dimensional space time continuum from the beginning of the universe to the end if there is one, yes, there are many ways the world could be and that sentence be true. For example I could not have bothered to write this article now, I could have written it just a little later, or perhaps even not at all. There is a world in which you did not read it. There is a world in which I went out this morning to get a baguette from one of the many delicious local french bakeries. The world could be all these ways and yet still Barack Obama be the 44th president of the United States.

In N3 we speak about the meaning of a sentence by quoting it with '{' '}'. So for our example we can write:

@prefix dbpedia: <http://dbpedia.org/resource/> .
{ dbpedia:Barack_Obama a dbpedia:President_of_the_United_States . } = :g1 .

:g1 is the set of all possible worlds in which Obama is president of the USA. The only worlds that are not part of that set, are the worlds where Obama is not President, but say McCain or Sarah Palin is. That McCain might have become president of the United States is quite conceivable. Both those meanings are understandable, and we can speak about both of them

@prefix dbpedia: <http://dbpedia.org/resource/> .
{ dbpedia:Barack_Obama a dbpedia:President_of_the_United_States . } = :g1 .
{ dbpedia:John_McCain a dbpedia:President_of_the_United_States . } = :g2 .
:g1 hopedBy :george .
:g2 feardedBy :george .
:g1 fearedBy :jane .

Ie. we can say that George hopes Barack Obama to be the 44th president of the United States, but that Jane fears it.

Assume wikipedia had a resource for each member of the list of presidents of the USA, and that we were pointing to the 44th element above. Then even though we can speak about :g1 and :g2, there is no world that fits them both: The intersection of both :g1 and :g2 is { } , the empty set, whose extension according to David Lewis' book on Mereology is the fusion of absolutely all possibilities. The thing that is everything and everywhere and around at all times. Ie. you don't make any distinction when you say that: you don't say anything.

The definition of meaning in terms of possible worlds, make a few things very simple to explain. Implication being one of them. If every president has to be human, then

@prefix log: <http://www.w3.org/2000/10/swap/log#> .
{ dbpedia:Barack_Obama a dbpedia:President_of_the_United_States . } log:implies { dbpedia:Barack_Obama a dbpedia:Human . }

Ie the set of possible worlds in which Obama is a president of the United States is a subset of the set of worlds in which he is Human. There are worlds after all where Barack is just living a normal Lawyer's life.

So what is this mapping from URIs to meaning that Tim Berners Lee is talking about? I interpret him as speaking of the log:semantics relation.

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
log:semantics a rdf:Property;
         :label "semantics";
         :comment """The log:semantics of a document is the formula.
         achieved by parsing representation of the document.
          For a document in Notation3, log:semantics is the
          log:parsedAsN3 of the log:contents of the document.
          For a document in RDF/XML, it is parsed according to the
          RDF/XML specification to yield an RDF formula [snip]""";
         :domain foaf:Document;
         :range log:Formula .

Of course it is easier to automate the mapping from resources that return RDF based representations, but log:semantics can be applied to any document. Any web page, even those written in natural languages, have some semantics. It is just that they currently require very advanced wetware processors to interpret them. These can indeed be very specialised wetware processors, as for example those that one meets at air ports.

Wednesday Nov 05, 2008

The coming postmodern era

Kevin Kelly argued convincingly that the growth in technology is creating a new world wide super organism, something Nova Spivack likes to call One Mind (OM). I argue here that this One Mind will have to be a postmodern mind: it will have to take points of view as a fundamental given. In other words it is a world of Many Many Minds (MMM) that is being born.

Concepts can take a long time from their birth to their acceptance by society. Democritus reasoned in 400BC that the earth was round, that there were other stars, and that they had planets which had life. It took a 2400 years, a trip to the moon, satellite television, mass air travel to turn these deep insights into common sense.

I think one can make the case that the massive intrusion of the Personal Computer in the 1980ies into huge numbers of household and businesses led to the strengthening of the concepts of 'efficiency' and the self. The metaphor of the brain as computer took hold silencing previous behaviorist intuitions. The computer could be programmed. It could think. Some could think faster than others. Every year they became more efficient. The PC was the icon of the age. It was alone and did not communicate. It was the era of selfish competition: the 'me' generation. As Margaret Thatcher, prime minister of Britain at the time said: "There is no such thing as society".

In the 1990ies the internet entered public consciousness, and with it the realization of how the network was overtaking the PC in importance. Information moved from being mostly on a computer to being mostly in the network cloud. The network was slow, so the experience people had was primarily of being connected to information and commerce. The experience of globalization of commerce and information blended with a modernistic view of the future unity of humanity moving towards one end: the end of history.

Behind the growth of the web and the internet, hidden to many, lay the strength of community. Unix, Linux, Apache, Open Source software, that had been the cause of the huge growth of the Network became more apparent, and became visible to the majority in the form of the read/write web under the banner of blogging. The last 8 years have been the discovery of the web as a platform for each individual voice to be heard, of community and mostly protected social networks. The end of the 20th century was also the end of the read only society as Lawrence Lessig argues so well. Millions of different points of view came to express themselves on innumerable topics.

Where next? What will happen as we move from a human readable read/write web to a machine readable one? What happens when we manage to break through the autism of current tools? What happens when software becomes widely available that can ask you if you want to reason over data you believe, or if you'd rather look at what your parents believe, or what republicans tend to believe, or what your children believe? This is as I argued recently the fundamentally new thing the semantic web is making possible; something unlike anything that humanity has ever witnessed before. The first tools that can make the step out of autism.

Of course, we mostly all come to understand around the age of 4 that other people believe different things from us, and that different people may think incompatible things about the world. But what happens when this everyday intuition becomes mechanized, objectified in tools that each year become more efficient? Most people always knew that society was very important, but the growth of the PC in the 1980ies created a strong icon in public discourse around which concepts of the self could cluster. In a similar way the growth of software that can point out contradictions between different points of views expressed in a distributed way around the web, would by doing this place a huge emphasis on the notion of points of views. If it were to make exploring these views easy, easier than it is for a normal human being living a normal life nowadays, then we can imagine that people may start exploring points of views much more often, more easily, in more detail, without thinking too much of it. Just like people now may drive 35 miles to work because they can, we can imagine people thinking more about others because some of the hard work has now been automated for them. Discovering conflicts in belief before they lead to conflicting actions could remove a lot of problems before they occur. ( Hopefully it won't lead us into some crazy world such as that described in the movie Being John Malkovitch ).

So how does this fit in with Post Modernism? Well, post modernism is a fuzzy concept, possibly even fuzzier than Web 2.0 or for that matter Web 3.0. It arose out of the disillusionment with all deterministic explanations of the future given by many of the western schools of thought, from christian evangelism to Marxism, Futurism, Consumerism ... Weary of all totalitarian explanations of everything, baffled by their sheer number, thinkers came to look at the different theories not from the inside, but from the outside, Instead of looking for a theory in which to believe trying to find a theory that would subsume all others, postmodernism, as I understand it, accepted the multiplicity of viewpoints, and found it more interesting to understand their differences. By putting more emphasis on understanding than on Truth, it was possible to look at the multiplicity of different points of view in the world. The pygmy in his tribe was no longer someone in need of conversion to the Truth, but someone one should try to understand in his context. This was felt by many to lead to a dangerous relativism, where the notion of truth itself seemed like it was loosing its meaning. In fact truth has never been better and more precisely defined: It is at its core a disquotation mechanism. According to Tarski's definition of truth:

"Snow is white" is true, in English if and only if snow is white .
Or in N3
  @prefix log: <http://www.w3.org/2000/10/swap/log#> .

  { { ?s ?r ?o } a log:Truth } <=> { ?s ?r ?o } .
or in SPARQL
PREFIX log: <http://www.w3.org/2000/10/swap/log#>
CONSTRUCT { ?subject ?relation ?object }
   GRAPH ?g { ?subject ?relation ?object }
   ?g a log:Truth .

Ie, if you hear someone say something, and you believe what they said to be true then you believe what they said. That is so simple it is self evident. So what has it got us? Well believing something is not neutral. Because we infer things from what we believe, and because we act on what we believe, to believe something is also to act and to be predisposed to act. And that is where the contact with reality ends up being felt at some point or another. If someone shouts "Une voiture arrive a ta gauche" in French and you understand it then you might add the following to your database:

{ _:c a :Car; 
      :moving [ :closeProximityTo :you ] .
  _:c positionleftOf :you .
} saidBy :joe .

At that point you just believe that Joe believes this. It makes a big difference when you come to believe the same content, namely

[] a :Car;
   :moving [ :closeProximityTo :you ] .
  _:c positionleftOf :you .

The disquotation mechanism (In N3 the removing of the '{' '}' ) is therefore an essential part of communication. One should not believe everything one hears - one may after all have misunderstood what was said. To remember who said what, and when one heard it is essential to good thinking. And sometimes who is right is really not that important anyway. Sometimes understanding is more important still. And that means putting oneself into other person's shoes, trying to look at things from their point of view - in essence, realizing that there are many many minds (MMM). So again what will happen when all tools we use every day make it as easy for us to explore points of view as it is to look at a web page, or take the car to work?

And where does this leave the absolute conception of Truth? Metcalf's Law gives a good explanation of the value of such a conception. Remember that this law states that the value of a network grows exponentially with the size of the network. The search for the Truth was always the search for an explanation that could explain as many things as possible: i.e. to create the biggest possible network, to predict as much as possible, to englobe all points of view, to create a framework that could link all of them together.

But what if the largest possible network has to take into account points of views as basic constitutive elements of the network?

Wednesday Sep 24, 2008

Serialising Java Objects to RDF with Jersey

Jersey is the reference implementation of JSR311 (JAX-RS) the Java API for RESTful Web Services. In short JSR311 makes it easy to publish graphs of Java Objects to the web, and implement update and POST semantics - all this using simple java annotations. It makes it easy for Java developers to do the right thing when writing data to the web.

JAX-RS deals with the mapping of java objects to a representation format - any format. Writing a good format, though, is at least as tricky as building RESTful services. So JAX-RS solves only half the problem. What is needed is to make it easy to serialize any object in a manner that scales to the web. For this we have RDF, the Resource Description Framework. By combining JAX-RS and the So(m)mer's @rdf annotation one can remove remove the hurdle of having to create yet another format, and do this in a way that should be really easy to understand.

I have been wanting to demonstrate how this could be done, since the JavaOne 2007 presentation on Jersey. Last week I finally got down to writing up some initial code with the help of Paul Sandoz whilst in Grenoble. It turned out to be really easy to do. Here is a description of where this is heading.


The code to do this available from the so(m)mer subversion repository, in the misc/Jersey directory. I will refer and link to the online code in my explanations here.

Annotate one's classes

First one needs to annotate one's model classes with @rdf annotations on the classes and fields. This is a way to give them global identifiers - URIs. After all if you are publishing to the web, you are publishing to a global context so global names are the only way to remove ambiguity. So for example our Person class can be written out like this:

public class Person extends Agent {
    static final String foaf = "http://xmlns.com/foaf/0.1/";

    @rdf(foaf+"surname") private Collection<String> surnames = new HashSet<String>();
    @rdf(foaf+"family_name") private Collection<String> familynames = new HashSet<String>();
    @rdf(foaf+"plan") private Collection<String> plans = new HashSet<String>();
    @rdf(foaf+"img") private Collection<URL> images = new HashSet<URL>();
    @rdf(foaf+"myersBriggs") private Collection<String> myersBriggss = new HashSet<String>();
    @rdf(foaf+"workplaceHomepage") private Collection<FoafDocument> workplaceHomePages = new HashSet<FoafDocument>();

This just requires one to find existing ontologies for the classes one has, or to publish new ones. (Perhaps this framework could be extended so as to automate the publishing of ontologies as deduced somehow form Java classes? - Probably a framework that could be built on top of this)

Map the web resources to the model

Next one has to find a mapping for web resources to objects. This is done by subclassing the RdfResource<T> template class, as we do three times in the Main class. Here is a sample:

   public static class PersonResource extends RdfResource<Employee> {
      public PersonResource(@PathParam("id") String id) {
          t = DB.getPersonWithId(id);

This just tells Jersey to publish any Employee object on the server at the local /person/{id} url. When a request for some resource say /person/155492 is made, a PersonResource object will be created whose model object can be found by querying the DB for the person with id 155492. For this of course one has to somehow link the model objects ( Person, Office,... in our example ) to some database. This could be done by loading flat files, querying an ldap server, or an SQL server, or whatever... In our example we just created a simple hard coded java class that acts as a DB.

Map the Model to the resource

An object can contain pointers to other objects. In order for the serialiser to know what the URL of objects are one has to map model objects to web resources. This is done simply with the static code in the same Main class [ looking for improovements here too ]

   static {
      RdfResource.register(Employee.class, PersonResource.class);
      RdfResource.register(Room.class, OfficeResource.class);
      RdfResource.register(Building.class, BuildingResource.class);              

Given an object the serialiser (RdfMessageWriter) can then look up the resource URL pattern, and so determine the object's URL. So to take an example, consider an instance of the Room class. From the above map, the serialiser can find that it is linked to the OfficeResource class from which it can find the /building/{buildingName}/{roomId} URI pattern. Using that it can then call the two getters on that Room object, namely getBuildingName() and getRoomId() to build the URL referring to that object. Knowing the URL of an object means that the serialiser can stop its serialisation at that point if the object is not the primary topic of the representation. So when serialising /person/155492 the serialiser does not need to walk through the properties of /building/SCA22/3181. The client may already have that information and if not, the info is just a further GET request away.

Running it on the command line

If you have downloaded the whole repository you can just run

$ ant run
from the command line. This will build the classes, recompile the @rdf annotated classes, and start the simple web server. You can then just curl for a few of the published resources like this:
hjs@bblfish:0$ curl -i http://localhost:9998/person/155492
HTTP/1.1 200 OK
Date: Wed, 24 Sep 2008 14:37:38 GMT
Content-type: text/rdf+n3
Transfer-encoding: chunked

<> <http://xmlns.com/foaf/0.1/primaryTopic> </person/155492#HS> .
</person/155492#HS> <http://xmlns.com/foaf/0.1/knows> <http://www.w3.org/People/Berners-Lee/card#i> .
</person/155492#HS> <http://xmlns.com/foaf/0.1/knows> </person/528#JG> .
</person/155492#HS> <http://xmlns.com/foaf/0.1/birthday> "29_07" .
</person/155492#HS> <http://xmlns.com/foaf/0.1/name> "Henry Story" .

The representation returned not a very elegant serialisation of the Turtle subset of N3. This makes the triple structure of RDF clear- subject relation object - and it uses relative URLs to refer to local resources. Other serialisers could be added, such as for rdf/xml. See the todo list at the end of this article.

The represenation says simple that this resource <> has as primary topic the entity named by #HS in the document. That entity's name is "Henry Story" and knows a few people, one of which is refered to via a global URL http://www.w3.org/People/Berners-Lee/card#i, and the other via a local URL /person/528#JG.

We can find out more about the /person/528#JG thing by making the following request:

hjs@bblfish:0$ curl -i http://localhost:9998/person/528#JG
HTTP/1.1 200 OK
Date: Wed, 24 Sep 2008 14:38:10 GMT
Content-type: text/rdf+n3
Transfer-encoding: chunked

<> <http://xmlns.com/foaf/0.1/primaryTopic> </person/528#JG> .
</person/528#JG> <http://xmlns.com/foaf/0.1/knows> </person/155492#HS> .
</person/528#JG> <http://xmlns.com/foaf/0.1/knows> </person/29604#BT> .
</person/528#JG> <http://xmlns.com/foaf/0.1/knows> <http://www.w3.org/People/Berners-Lee/card#i> .
</person/528#JG> <http://www.w3.org/2000/10/swap/pim/contact#office> </building/SCA22/3181#it> .
</person/528#JG> <http://xmlns.com/foaf/0.1/birthday> "19-05" .
</person/528#JG> <http://xmlns.com/foaf/0.1/name> "James Gosling" .

... where we find out that the resource named by that URL is James Gosling. We find that James has an office named by a further URL, which we can discover more about with yet another request

hjs@bblfish:0$ curl -i http://localhost:9998/building/SCA22/3181#it
HTTP/1.1 200 OK
Date: Wed, 24 Sep 2008 14:38:38 GMT
Content-type: text/rdf+n3
Transfer-encoding: chunked

<> <http://xmlns.com/foaf/0.1/primaryTopic> </building/SCA22/3181#it> .
</building/SCA22/3181#it> <http://www.w3.org/2000/10/swap/pim/contact#address> _:2828781 .
_:2828781 a <http://www.w3.org/2000/10/swap/pim/contact#Address> .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#officeName> "3181" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#street> "4220 Network Circle" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#stateOrProvince> "CA" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#city> "Santa Clara" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#country> "USA" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#postalCode> "95054" .

Here we have a Location that has an Address. The address does not have a global name, so we give it a document local name, _:2828781 and serialise it in the same representation, as shown above.

Because every resource has a clear hyperlinked representation we don't need to serialise the whole virtual machine in one go. We just publish something close to the Concise Bounded Description of the graph of objects.

Browsing the results

Viewing the data through a command line interface is nice, but it's not as fun as when viewing it through a web interface. For that it is best currently to install the Tabulator Firefox plugin. Once you have that you can simply click on our first URL http://localhost:9998/person/155492. This will show up something like this:

picture of tabualator on loading /person/155492

If you then click on JG you will see something like this:

picture tabulator showing James Gosling

This it turns out is a resource naming James Gosling. James knows a few people including a BT. The button next to BT is in blue, because that resource has not yet been loaded, whereas the resource for "Henry Story" has. Load BT by clicking on it, and you get

picture of tabulator showing Henry Story knowning Bernard Traversat

This reveals the information about Bernard Traversat we placed in our little Database class. Click now on the i and we get

Tabulator with info about Tim Berners Lee

Now we suddenly have a whole bunch of information about Tim Berners Lee, including his picture, some of the people he has listed as knowing, where he works, his home page, etc... This is information we did not put in our Database! It's on the web of data.

One of the people Tim Berner's Lee knows is our very own Tim Bray.

tabulator showing info from dbpedia on Tim Bray

And you can go on exploring this data for an eternity. All you did was put a little bit of data on a web server using Jersey, and you can participate in the global web of data.


There are of course a lot of things that can be done to improove this Jersey/so(m)mer mapper. Here are just a few I can think of now:

  • Improove the N3 output. The code works with the examples but it does not deal well with all the literal types, nor does it yet deal with relations to collections. The output could also be more human readable by avoiding repetiations.
  • Refine the linking between model and resources. The use of getters sounds right, but it could be a bit fragile if methods are renamed....
  • Build serialisers for rdf/xml and other RDF formats.
  • Deal with publication of non information resources, such as http:// xmlns.com/foaf/0.1/Person which names the class of Persons. When you GET it, it redirects you to an information resources:
    hjs@bblfish:1$ curl -i http://xmlns.com/foaf/0.1/Person
    HTTP/1.1 303 See Other
    Date: Wed, 24 Sep 2008 15:30:14 GMT
    Server: Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.61 OpenSSL/0.9.7e mod_fastcgi/2.4.2 Phusion_Passenger/2.0.2 DAV/2 SVN/1.4.2
    Location: http://xmlns.com/foaf/spec/
    Vary: Accept-Encoding
    Content-Length: 234
    Content-Type: text/html; charset=iso-8859-1
    <title>303 See Other</title>
    <h1>See Other</h1>
    <p>The answer to your request is located <a href="http://xmlns.com/foaf/spec/">here</a>.</p>
    This should also be made easy and foolproof for Java developers.
  • make it industrial strength...
  • my current implementation is tied much too strongly to so(m)mer. The @rdf annotated classes get rewritten to create getters and setters for every field, that links to the sommer mappers. This is not needed here. All we need is to automatically add the RdfSerialiser interface to make it easy to access the private fields.
  • One may want to add support for serialising @rdf annotated getters
  • Add some basic functionality for POSTing to collections or PUTing resources. This will require some thought.

Bookmarks: digg+, reddit+, del.icio.us+, dzone+, facebook+

Wednesday Sep 17, 2008

Are OO languages Autistic?

illustration of a simple play

One important criterion of Autism is the failure to develop a proper theory of mind.

A standard test to demonstrate mentalizing ability requires the child to track a character's false belief. This test can be done using stories, cartoons, people, or, as illustrated in the figure, a puppet play, which the child watches. In this play, one puppet, called, Sally, leaves her ball in her basket, then goes out to play. While she is out, naughty Anne moves the ball to her own box. Sally returns and wants to play with her ball. The child watching the puppet play is asked where Sally will look for her ball (where does Sally think it is?). Young children aged around 4 and above recognize that Sally will look in the basket, where she (wrongly) thinks the ball is.
Children with autism will tend to answer that Sally will look for the ball in the box.

Here are two further descriptions of autism from today's version of the Wikipedia article:

The main characteristics are of Autism are impairments in social interaction, impairments in communication, restricted interests and repetitive behavior.
Sample symptoms include lack of social or emotional reciprocity, stereotyped and repetitive use of language or idiosyncratic language, and persistent preoccupation with parts of objects.

In order to be able to have a mental theory one needs to be able to understand that other people may have a different view of the world. On a narrow three dimensional understanding of 'view', this reveals itself in that people at different locations in a room will see different things. One person may be able to see a cat behind a tree that will be hidden to another. In some sense though these two views can easily be merged into a coherent description. They are not contradictory. But we can do the same in higher dimensions. We can think of people as believing themselves to be in one of a number of possible worlds. Sally believes she is in a world where the ball is in the basket, whereas Ann believes she is in a world where the ball is in the box. Here the worlds are contradictory. They cannot both be true of the actual world.

To be able to make this type of statement one has to be able to do at least the following things:

  • Speak of ways the world could be
  • Refer to objects across these worlds
  • Compare these worlds
The ability to do this is present in none of the well known Object Oriented (OO) languages by default. One can add it, just as one can add garbage collection to C, but it requires a lot of discipline and care. It does not come naturally. Perhaps a bit like a person with Asperger's syndrome can learn to interact socially with others, but in a reflective awkward way.

Let us illustrate this with a simple example. Let us see how one could naively program the puppet play in Java. Let us first create the objects we will need:

Person sally = new Person("Sally");
Person ann = new Person("Ann");
Container basket = new Container("Basket");
Container box = new Container("Box");
Ball ball = new Ball("b1");
Container room = new Container("Room");
So far so good. We have all the objects. We can easily imagine code like the following to add the ball into the basket, and the basket into the room.
Perhaps we have methods whereby the objects can ask what their container is. This would be useful for writing code to make sure that a thing could not be in two different places at once - in the basket and in the box, unless the basket was in the box.
Container c = ball.getImmediateContainer();
Assert.true(c == basket);
try {
} catch (InTwoPlacesException e) {
All that is going to be tedious coding, full of complicated issues of their own, but it's the usual stuff. Now what about the beliefs of Sally and Ann? How do we specify those? Perhaps we can think of sally and ann as being small databases of objects they are conscious of. Then one could just add them like this:
But the problem should be obvious now. If we move the ball from the basket to the box, the state of the objects in sally and ann's database will be exactly the same! After all they are the same objects!
Ball sb = sally.get(Ball.class,"b1");
//that is because
Ball ab = ann.get(Ball.class,"b1");
There is really no way to change the state of the ball for one person and not for the other,... unless perhaps we give both people different objects. This means that for each person we would have to make a copy of all the objects that they could think of. But then we would have a completely different problem: namely deciding when these two objects were the same. For it is usually understood that the equality of two objects depends on their state. So one usually would not think that an physical object could be the same if it was in two different physical places. Certainly if we had a ball b1 in a box, and another ball b2 in a basket, then what on earth would allow us to say we were speaking of the same ball? Perhaps their name, if it we could guarantee that we had unique names for things. But we would still have some pretty odd things going on then, we would have objects that would somehow be equal, but would be in completely different states! And this is just the beginning of our problems. Just think of the dangers involved here in taking an object from ann's belief database, and how easy it would be to by mistake allow it to be added to sally's belief store.

These are not minor problems. These are problems that have dogged logicians for the last century or more. To solve it properly then one should look for languages that were inspired by the work of such logicians. The most serious such project is now knows as the Semantic Web.

Using N3 notation we can write the state of affairs described by our puppet show, and illustrated by the above graph, out like this:

@prefix : <http://test.org/> .

:Ann :believes { :ball :in :box . } .
:Sally :believes { :ball in :basket } .

N3 comes with a special notation for grouping statements by placing them inside of { }. We could then easily ask who believes the ball is in the basket using SPARQL

PREFIX : <http://test.org/>
     GRAPH ?g1 { :ball :in :basket }
     ?who :believes ?g1 .

The answer would bind ?who to :Sally, but not to :Ann.

RDF therefore gives us the basic tools to escape from the autism of simpler languages:

  • One can easily refer to the same objects across contexts, as URIs are the basic building block of RDF
  • The basic unit of meaning are sets of relations - graphs - and these are formally described.
The above allows query for objects across contexts and so to compare, merge and work with contexts.

It is quite surprising once one realizes this, to think how many languages claim to be web languages, and yet fail to have any default space for the basic building blocks of the web: URIs and the notion of different points of views. When one fetches information from a remote server one just has to take into account the fact that the server's view of the world may be different and incompatible in some respects with one's own. One cannot in an open world just assume that every body agrees with everything. One is forced to develop languages that enable a theory of mind. A lot of failures in distributed programming can probably be traced down to working with tools that don't.

Of course tools can be written in OO languages to work with RDF. Very good ones have been written in Java, such as Sesame, making it possible to query repositories for beliefs across contexts (see this example). But they bring to bear concepts that don't sit naturally with Java, and one should be aware of this. OO languages are good for building objects such as browsers, editors, simple web servers, transformation tools, etc... But they don't make it easy to develop tools that require just the most basic elements of a theory of mind, and so most things to do with communication. For that one will have to use the work done in the semantic web space and familiarize oneself with the languages and tools developed for working with them.

Finally the semantic web also has its OO style with the Web Ontology Language (OWL). This is just a set of relations to describe classes and relations. Notice though that it is designed for intra context inference, ie all inferences that you can make within a world. So in that sense thinking in OO terms does even at the Semantic Web layer seem to not touch on thinking across contexts, or mentally. Mind you, since people deal with objects, it is also important to think about objects to understand people. But it is just one part of the problem.

vote on reddit and follow the discussion
vote on dzone

Friday Sep 12, 2008

RDF: Reality Distortion Field

Here is Kevin Kelly's presentation on the next 5000 days on the web, in clear easy English that every member of the family can watch and understand. It explains what the semantic web, also known as Web 3.0, is about and how it will affect technology and life on earth. Where is the web going? I can find no fault in this presentation.

This is a great introduction. He explains how Metcalf's law brought us to the web of documents and is leading us inexorably to a web of things, in which we will be the eyes and the hands of this machine called the internet that never stops running.
For those with a more technical mind, who want to see how this is possible, follow this up with a look at the introductory material to RDF.

Warning: This may change the way you think. Don't Panic! Things will seem normal after a while.

Thursday Sep 04, 2008

Building Secure, Open and Distributed Social Network Applications

Current Social Networks don't allow you to have friends outside their network. When on Facebook, you can't point to your friend on LinkedIn. They are data silos. This audio enhanced slide show explains how a distributed decentralized social network is being built, how it works, and how to make is secure using the foaf+ssl protocol (a list of pointers on the esw wiki).

It is licenced under a CC Attribution ShareAlike Licence.
My voice is a bit odd on the first slide, but it gets better I think as I go along.

Building Secure Open & Distributed Social Networks( Viewing this slide show requires a flash plugin. Sorry I only remembered this limitation after having put it online. If you know of a good Java substitute let me know. The other solution would have been to use Slidy. PDF and Annotated Open Document Format versions of this presentation are available below. (why is this text visible in Firefox even when the plugin works?) )

This is the presentation I gave at JavaOne 2008 and at numerous other venues in the past four months.

The slidecast works a lot better as a presentation format, than my previous semantic web video RDF: Connecting Software and People which I published as a h.264 video over a couple of years ago, and which takes close to 64MB of disk space. The problem with that format is that it is not easy to skip through the slides to the ones that interest you, or to go back and listen to a passage carefully again. Or at least it feels very clunky. My mp3 sound file only takes 17MB of space in comparison, and the graphics are much better quality in this slide show.

It is hosted by the excellent slideshare service, which translated my OpenOffice odp document ( once they were cleaned up a little: I had to make sure it had no pointers to local files remaining accessible from the Edit>Links menu (which otherwise choked their service)). I used the Audacity sound editor to create the mp3 file which I then place on my bblfish.net server. Syncing the sound and the slides was then very easy using SlideShare's SlideCast application. I found that the quality of the slides was a lot better once I had created an account on their servers. The only thing missing would be a button in addition to the forward and backward button that would allow one to show the text of the audio, for people with hearing problems - something equivalent to the Notes view in Open Office.

You can download the OpenOffice Presentation which contains my notes for each slide and the PDF created from it too. These are all published under a Creative Commons Attribution, Share Alike license. If you would like some of the base material for the slides, please contact me. If you would like to present them in my absence feel free to.

Tuesday Sep 02, 2008

exploring a web 2.0 app

I was trying today to understand what communication was going in the social site project today, which contains an implementation of the Open Social protocol. As the application uses a lot of asynchronous javascript, and as it is difficult to find the URLs for the resources pointed to (they get constructed at run time), one has to listen in on the wire to find the structure of the project. But that is not at all as easy as one could hope. Here is how I got going:

  • Usually I use the command line tcpflow that just captures all the communication on the wire. The problem is that this does not decrypt the ssl connections, and social site has a lot of requests going over https.
  • There was a Firefox plugin called Slogger that would keep all the communication going back and forth and store it to the hard disk. But it is no longer maintained, and no longer seems to be working for Firefox 3.0.1 . (I even tried the workaround suggested on the plugin page)
  • Firebug is another, and very useful, such Firefox plugin. It works and was good enough for what I needed to do. The only problem is that it seems to have to make the requests again to show you the content, which is a problem for POST requests as these could have side effects. Here is a picture: Firebug HTTP console
    (one can immediately see some oddities here: the last visible request asks for xml content types, but the response is json).
  • Francois Le Droff pointed me to Charles Web Debugging Proxy (written in Java) which is a proprietary proxy, that seems to be able to do it all. I have not tried it yet, as Firebug did the job for me. But I may need to get it at some later time, when state on the server starts getting to be important.

Getting this type of information is really important when debugging an application. Hopefully bug 430155 will be fixed soon, allowing Firebug to do the right thing. I wonder if Google's Chrome comes with this functionality built in?

Thursday Aug 28, 2008

picture of my blog

picture from wordle.net

Wordle is a fun little Java applet that analyses your blog and builds any number of beautiful word clouds from the content. It has started a movement I think.

Just a pity that one cannot embed the applet in one's own web page. That would make it dynamic and a lot more interesting. Perhaps that will come. At present it seems the author is not sure what the IBM lawyers have decided.

This is somewhat similar to the graphic app I mentioned previously though with somewhat more content, clearly. Long term readers of my blog may find the picture of my blog to not be such a good reflection of my interests though. Statistic analysis of words does not get you that far.

Wednesday Aug 27, 2008

my blog is worth $49,114.98

According to Dan Carlson's program my blog is worth 49thousand dollars. That's nothing next to Tim Bray's who's coming up for 700thousand dollars. Of course Dan Carlson is a big winner in all this, because everybody is linking to his site, so his just keeps growing. He is now at 2 million by his own estimate which has something to do with with some AOL deal a few years ago.

My blog is worth $49,114.98.
How much is your blog worth?

Now would I sell my blog for a new car?

As with a new car you'd have to be a bit careful how you sell your site. Tim Brays http://tbray.org/ongoing is worth $693,819.66 . But http://tbray.org/ongoing/ [notice the trailing slash] is worth $375,983.64 . So is Tim's site worth the sum of both, 1 million?

Tuesday May 06, 2008

BOF-5911: Building a Web 3.0 Address Book

To give everyone a chance to try out the So(m)mer Address Book, I have made it available via Java Web Start: just click on the picture to the right, and try it out.

The Address Book is currently demoware: it shows how one can build virally an open distributed social network client that solves the social network data silo problem (video). No need to have an account on every social networking site on which you have friends, and so maintain your data on each one. You can simply belong to one network and link to all your friends wherever they are. With one click of a button you can publish your social network to your own web server, using ftp, scp, WebDAV, or even Atom. You can then link to other people who have (or not in fact), a foaf file. By pressing the space bar when selecting a friend, the Address Book with then GET their file. So you can browse your social network.

To get going you can explore my social network by dragging my foaf file icon onto the first pane of the application.

In BOF-5911 which I will be presenting on Thursday at 7:30pm I will be presenting the social networking problem, demonstrating how the So(m)mer Address Book solves it, and showing in detail how it is build, what the problems are, and what work remains. I will also discuss how this can be used to create global single sign on based on a network of trust.


An improved version of the presentation I gave is now available online with audio as Building Secure, Open and Distributed Social Network Applications

Monday Apr 21, 2008

FOAF & SSL: creating a global decentralised authentication protocol

Following on my previous post RDFAuth: sketch of a buzzword compliant authentication protocol, Toby Inkster came up with a brilliantly simple scheme that builds very neatly on top of the Secure Sockets Layer of https. I describe the protocol shortly here, and will describe an implementation of it in my next post.

Simple global ( passwordless if using a device such as the Aladdin USB e-Token ) authentication around the web would be extremely valuable. I am currently crumbling under the number of sites asking me for authentication information, and for each site I need to remember a new id and password combination. I am not the only one with this problem as the data portability video demonstrates. OpenId solves the problem but the protocol consumes a lot of ssl connections. For hyperdata user agents this could be painfully slow. This is because they may need access to just a couple of resources per server as they jump from service to service.

As before we have a very simple scenario to consider. Romeo wants to find out where Juliette is. Juliette's hyperdata Address Book updates her location on a regular basis by PUTing information to a protected resource which she only wants her friends and their friends to have access to. Her server knows from her foaf:PersonalProfileDocument who her friends are. She identifies them via dereferenceable URLs, as I do, which themselves usually (the web is flexible) return more foaf:PersonalProfileDocuments describing them, and pointing to further such documents. In this way the list of people able to find out her location can be specified in a flexible and distributed manner. So let us imagine that Romeo is a friend of a friend of Juliette's and he wishes to talk to her. The following sequence diagram continues the story...

sequence diagram of RDF+SSL

The stages of the diagram are listed below:

  1. First Romeo's User Agent HTTP GETs Juliette's public foaf file located at http://juliette.net/. The server returns a representation ( in RDFa perhaps ) with the same semantics as the following N3:

    @prefix : <#> . 
    @prefix foaf: <http://xmlns.com/foaf/0.1/> .
    @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
    @prefix todo: <http://eg.org/todo#> .
    @prefix openid: <http://eg.org/openid/todo#> .
    <> a foaf:PersonalProfileDocument;
       foaf:primaryTopic :juliette ;
       openid:server <https://aol.com/openid/service>; # see The Openid Sequence Diagram .
    :juliette a foaf:Person;
       foaf:name "Juliette";
       foaf:openid <>;
       foaf:blog </blog>;    
       rdfs:seeAlso <https://juliette.net/protected/location>; 
       foaf:knows <http://bblfish.net/people/henry/card#me>,
                  <http://www.w3.org/People/Berners-Lee/card#i> .
    <https://juliette.net/protected/location> a todo:LocationDocument .

    Romeo's user agent receives this representation and decides to follow the https protected resource because it is a todo:LocationDocument.

  2. The todo:LocationDocument is at an https URL, so Romeo's User Agent connects to it via a secure socket. Juliette's server, who wishes to know the identity of the requestor, sends out a Certificate Request, to which Romeo's user agent responds with an X.509 certificate. This is all part of the SSL protocol.

    In the communication in stage 2, Romeo's user agent also passes along his foaf id. This can be done either by:

    • Sending in the HTTP header of the request an Agent-Id header pointing to the foaf Id of the user. Like this:
      Agent-Id: http://romeo.net/#romeo
      This would be similar to the current From: header, but instead of requiring an email address, a direct name of the agent would be required. (An email address is only an indirect identifier of an agent).
    • The Certificate could itself contain the Foaf ID of the Agent in the X509v3 extensions section:
              X509v3 extensions:
                 X509v3 Subject Alternative Name: 

      I am not sure if it would be correct use of the X509 Alternative names field. So this would require more standardization work with the X509 community. But it shows a way where the two communities could meet. The advantage of having the id as part of the certificate is that this could add extra weight to the id, depending on the trust one gives the Certificate Authority that signed the Certificate.

  3. At this point Juliette's web server knows of the requestor (Romeo in this case):
    • his alleged foaf Id
    • his Certificate ( verified during the ssl session )

    If the Certificate is signed by a CA that Juliette trusts and the foaf id is part of the certificate, then she will trust that the owner of the User Agent is the entity named by that id. She can then jump straight to step 6 if she knows enough about Romeo that she trusts him.

    Having Certificates signed by CA's is expensive though. The protocol described here will work just as well with self signed certificates, which are easy to generate.

  4. Juliette's hyperdata server then GETs the foaf document associated with the foaf id, namely <http://romeo.net/> . Romeo's foaf server returns a document containing a graph of relations similar to the graph described by the following N3:
    @prefix : <#> . 
    @prefix foaf: <http://xmlns.com/foaf/0.1/> .
    @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
    @prefix wot: <http://xmlns.com/wot/0.1/> .
    @prefix wotodo: <http://eg.org/todo#> .
    <> a foaf:PersonalProfileDocument;
        foaf:primaryTopic :romeo .
    :romeo a foaf:Person;
        foaf:name "Romeo";
        is wot:identity of [ a wotodo:X509Certificate;
                             wotodo:dsaWithSha1Sig """30:2c:02:14:78:69:1e:4f:7d:37:36:a5:8f:37:30:58:18:5a:
                                                 eb:8c:11:08:1c:aa:93:7d:71:01""" ;
                           ] ;
        foaf:knows <http://bblfish.net/people/henry/card#me> .
  5. By querying the semantics of the returned document with a SPARQL query such as
    PREFIX wot: <http://xmlns.com/wot/0.1/> 
    PREFIX wotodo: <http://eg.org/todo#> 
    SELECT { ?sig }
    WHERE {
        [] a wotodo:X509Certificate;
          wotodo:signature ?sig;
          wot:identity <http://romeo.net/#romeo> .

    Juliette's web server can discover the certificate signature and compare it with the one sent by Romeo's user agent. If the two are identical, then Juliette's server knows that the User Agent who has access to the private key of the certificate sent to it, and who claims to be the person identified by the URI http://romeo.net/#romeo, is in agreement as to the identity of the certificate with the person who has write access to the foaf file http://romeo.net/. So by proving that it has access to the private key of the certificate sent to the server, the User Agent has also proven that it is the person described by the foaf file.

  6. Finally, now that Juliette's server knows an identity of the User Agent making the request on the protected resource, it can decide whether or not to return the representation. In this case we can imagine that my foaf file says that
     @prefix foaf: <http://xmlns.com/foaf/0.1/> .
     <http://bblfish.net/people/henry/card#me> foaf:knows <http://romeo.net/#romeo> .  
    As a result of the policy of allowing all friends of Juliette's friends to be able to read the location document, the server sends out a document containing relations such as the following:
    @prefix contact: <http://www.w3.org/2000/10/swap/pim/contact#> .
    @prefix : <http://juliette.org/#> .
        contact:location [ 
              contact:address [ contact:city "Paris";
                                contact:country "France";
                                contact:street "1 Champs Elysees" ]
                         ] .


  • Create an ontology for X509 certificates.
  • test this. Currently there is some implementation work going on in the so(m)mer repository in the misc/FoafServer directory.
  • Can one use the Subject Alternative name of an X509 certificate as described here?
  • For self signed certificates, what should the X509 Distinguished Name (DN) be? The DN is really being replaced here by the foaf id, since that is where the key information about the user is going to be located. Can one ignore the DN in a X509 cert, as one can in RDF with blank nodes? One could I imagine create a dummy DN where one of the elements is the foaf id. These would at least, as opposed to DN, be guaranteed to be unique.
  • what standardization work would be needed to make this

Discussion on the Web

Friday Apr 18, 2008

The OpenId Sequence Diagram

OpenId very neatly solves the global identity problem within the constraints of working with legacy browsers. It is a complex protocol though as the following sequence diagram illustrates, and this may be a problem for automated agents that need to jump around the web from hyperlink to hyperlink, as hyperdata agents tend to do.

The diagram illustrates the following scenario. Romeo wants to find the current location of Juliette. So his semantic web user agent GET's her current foaf file. But Juliette wants to protect information about her current whereabouts and reveal it only to people she trusts, so she configures her server to require the user agent to authenticate itself in order to get more information. If the user agent can prove that is is owned by one of her trusted friends, and Romeo in particular, she will deliver the information to it (and so to him).

The steps numbered in the sequence diagram are as follows:

  1. A User Agent fetches a web page that requires authentication. OpenId was designed with legacy web browsers in mind, for which it would return a page containing an OpenId login box such as the one to the right. openid login box In the case of a hyperdata agent as in our use case, the agent would GET a public foaf file, which might contain a link to an OpenId authentication endpoint. Perhaps with some rdf such as the following N3:
    <> openid:login </openidAuth.cgi> .
    Perhaps some more information would indicate which resources were protected.
  2. In current practice a human user notices the login box and types his identifying URL in it, such as http://openid.sun.com/bblfish This is the brilliant invention of OpenId: getting hundreds of millions of people to find it natural to identify themselves via a URL, instead of an email. The user then clicks the "Login button".
    In our semantic use case the hyperdata agent would notice the above openid link and would deduce that it needs to login to the site to get more information. Romeo's Id ( http://romeo.net/ perhaps ) would then be POSTed to the /openidAuth.cgi authentication endpoint.
  3. The OpenId authentication endpoint then fetches the web page by GETing Romeo's url http://romeo.net/. This returned representation contains a link in the header of the page pointing Romeo's OpenId server url. If the representation returned is html then this would contain the following in the header
     <link rel="openid.server" href="https://openid.sun.com/openid/service" />
  4. The representation returned in step 3, could contain a lot of other information too. A link to a foaf file may not be a bad idea as I described in foaf and openid. The returned representation in step 3 could even be RDFa extended html, in which case this step may not even be necessary. For a hyperdata server the information may be useful, as it may suggest a connection Romeo could have to some other people that would allow it to decide whether it wishes to continue the login process.
  5. Juliette's OpenId authentication endpoint then sends a redirect to Romeo's user agent, directing it towards his OpenId Identity Provider. The redirect also contains the URL of the OpenId authentication cgi, so that in step 8 below the Identity Provider can redirect a message back.
  6. Romeo user agent dutifully redirects romeo to the identity provider, which then returns a form with a username and password entry box.
  7. Romeo's user agent could learn to fill the user name password pair in automatically and even skip the previous step 6 . In any case given the user name and password, the Identity Provider then sends back some cryptographic tokens to the User Agent to have it redirect to the OpenId Authentication cgi at http://juliette.net/openidAuth.cgi.
  8. Romeo's Hyperdata user agent then dutifully redirects back to the OpenId authentication endpoint
  9. The authentication endpoint sends a request to the Openid Identity provider to verify that the cryptographic token is authentic. If it is, a conventional answer is sent back.
  10. The OpenId authentication endpoint finally sends a response back with a session cookie, giving access to various resources on Juliette's web site. Perhaps it even knows to redirect the user agent to a protected resource, though that would have required some information concerning this to have been sent in stage 2.
  11. Finally Romeo's user agent can GET Juliette's protected information if Juliette's hyperdata web server permits it. In this case it will, because Juliette loves Romeo.

All of the steps above could be automatized, so from the user's point of view they may not be complicated. The user agent could even learn to fill in the user name and password required by the Identity Provider. But there are still a very large number of connections between the User Agent and the different services. If these connections are to be secure they would need to protected by SSL (as hinted at by the double line arrows). And SSL connections are not cheap. So the above may be unacceptably slow. On the other hand it would work with a protocol that is growing fast in acceptance.

It is is certainly worth comparing this sequence diagram with the very light weight one presented in "FOAF & SLL: creating a global decentralised authentication protocol".

Thanks again to Benjamin Nowack for bringing the discussion on RDFAuth to thinking about using the OpenId protocol directly as described above. See his post on the semantic web mailing list. Benjamin also pointed to the HTTP OpenID Authentication proposal, which shows how some of the above can be simplified if certain assumptions about the capabilities of the client are made. It would be worth making a sequence diagram of that proposal too.

Thursday Apr 17, 2008

KiWi: Knowledge in a Wiki

KiWi logo

Last month I attended the European Union KiWi project startup meeting in Salzburg, to which Sun Microsystems Prague is contributing some key use cases.

KiWi is a project to build an Open Source Semantic Wiki. It is based on the IkeWiki [don't follow this link if you have Safari 3.1] Java wiki, which uses the Jena Semantic Web frameworks, the Dojo toolkit for the Web 2.0 functionality, and any one of the Databases Jena can connect to, such as PostgreSQL. KiWi is in many ways similar to Freebase in its hefty use of JavaScript, and its emphasis on structured data. But instead of being a closed source platform, KiWi is open source, and builds upon the Semantic Web standards. In my opinion it currently overuses JavaScript features, to the extent that all clicks lead to dynamic page rewrites that do not change the URL of the browser page. This I feel unRESTful, and the permalink link in the socialise toolbar to the right does not completely remove my qualms. Hopefully this can be fixed in this project. It would be great also if KIWI could participate fully in the Linked Data movement.

The meeting was very well organized by Sebastian Schaffert and his team. It was 4 long days of meetings that made sure that everyone was on the same page, understood the rules of the EU game, and most of all got to know each other. (see kiwiknows tagged pictures on flickr ). Many thanks also to Peter Reiser for moving and shaking the various Sun decision makers to sign the appropriate papers, and dedicate the resources for us to be part of this project.

You can follow the evolution of the project on the Planet Kiwi page.

Anyway, here is a video that shows the resourceful kiwi mascot in action:

Friday Mar 28, 2008

RDFAuth: sketch of a buzzword compliant authentication protocol

Here is a proposal for an authentication scheme that is even simpler than OpenId ( see sequence diagram ), more secure, more RESTful, with fewer points of failure and fewer points of control, that is needed in order to make Open Distributed Social Networks with privacy controls possible.


The following sketch led to the even simpler protocol described in Foaf and SSL creating a global decentralized authentication protocol. It is very close to what is proposed here but builds very closely on SSL, so as to reduce what is new down to nearly nothing.


Ok, so now I have your attention, I would like to first mention that I am a great fan of OpenId. I have blogged about it numerous times and enthusiastically in this space. I came across the idea I will develop below, not because I thought OpenId needed improving, but because I have chosen to follow some very strict architectural guidelines: it had to satisfy RESTful, Resource oriented hyperdata constraints. With the Beatnik Address Book I have proven - to myself at least - that the creation of an Open Distributed Social Network (a hot topic at the moment, see the Economist's recent article on Online social network) is feasible and easy to do. What was missing is a way for people to keep some privacy, clearly a big selling point for the large Social Network Providers such as Facebook. So I went on the search of a solution to create a Open Distributed Social Network with privacy controls. And initially I had thought of using OpenId.

OpenId Limitations

But OpenId has a few problems:

  • First it is really designed to work with the limitations of current web browsers. It is partly because of this that there is a lot of hopping around from the service to the Identity Provider with HTTP redirects. As the Tabulator, Knowee or Beatnik.
  • Parts of OpenId 2, and especially the Attribute Exchange spec really don't feel very RESTful. There is a method for PUTing new property values in a database and a way to remove them that does not use either the HTTP PUT method or the DELETE method.
  • The OpenId Attribute Exchange is nice but not very flexible. It can keep some basic information about a person, but it does not make use of hyperdata. And the way it is set up, it would only be able to do so with great difficulty. A RESTfully published foaf file can give the same information, is a lot more flexible and extensible, whilst also making use of Linked Data, and as it happens also solves the Social Network Data Silo problems. Just that!
  • OpenId requires an Identity Server. There are a couple of problems with this:
    • This server provides a Dynamic service but not a RESTful one. Ie. the representations sent back and forth to it, cannot be cached.
    • The service is a control point. Anyone owning such a service will know which sites you authenticate onto. True, you can set up your own service, but that is clearly not what is happening. The big players are offering their customers OpenIds tied to particular authentication servers, and that is what most people will accept.
As I found out by developing what I am here calling RDFAuth, for want of a better name, none of these restrictions are necessary.

RDFAuth, a sketch

So following my strict architectural guidelines, I came across what I am just calling RDFAuth, but like everything else here this is a sketch and open to change. I am not a security specialist nor an HTTP specialist. I am like someone who comes to an architect in order to build a house on some land he has, with some sketch of what he would like the house to look like, some ideas of what functionality he needs and what the price he is willing to pay is. What I want here is something very simple, that can be made to work with a few perl scripts.

Let me first present the actors and the resources they wish to act upon.

  • Romeo has a Semantic Web Address Book, his User Agent (UA). He is looking for the whereabouts of Juliette.
  • Juliette has a URL identifier ( as I do ) which returns a public foaf representation and links to a protected resource.
  • The protected resource contains information she only wants some people to know, in this instance Romeo. It contains information as to her current whereabouts.
  • Romeo also has a public foaf file. He may have a protected one too, but it does not make an entrance in this scene of the play. His public foaf file links to a public PGP key. I described how that is done in Cryptographic Web of Trust.
  • Romeo's Public key is RESTfully stored on a server somewhere, accessible by URL.

So Romeo wants to find out where Juliette is, but Juliette only wants to reveal this to Romeo. Juliette has told her server to only allow Romeo, identified by his URL, to view the site. She could have also have had a more open policy, allowing any of her or Romeo's friends to have access to this site, as specified by their foaf file. The server could then crawl their respective foaf files at regular intervals to see if it needed to add anyone to the list of people having access to the site. This is what the DIG group did in conjunction with OpenId. Juliette could also have a policy that decides Just In Time, as the person presents herself, whether or not to grant them access. She could use the information in that person's foaf file and relating it to some trust metric to make her decision. How Juliette specifies who gets access to the protected resource here is not part of this protocol. This is completely up to Juliette and the policies she chooses her agent to follow.

So here is the sketch of the sequence of requests and responses.

  1. First Romeo's user Agent knows that Juliette's foaf name is http://juliette.org/#juliette so it sends an HTTP GET request to Juliette's foaf file located of course at http://juliette.org/
    The server responds with a public foaf file containing a link to the protected resource perhaps with the N3
      <> rdfs:seeAlso <protected/juliette> .
    Perhaps this could also contain some relations describing that resource as protected, which groups may access it, etc... but that is not necessary.
  2. Romeo's User Agent then decides it wants to check out protected/juliette. It sends a GET request to that resource but this time receives a variation of the Basic Authentication Scheme, perhaps something like:
    Server: Knowee/0.4
    Date: Sat, 1 Apr 2008 10:18:15 GMT
    WWW-Authenticate: RdfAuth realm="http://juliette.org/protected/\*" nonce="ILoveYouToo"
    The idea is that Juliette's server returns a nonce (in order to avoid replay attacks), and a realm over which this protection will be valid. But I am really making this up here. Better ideas are welcome.
  3. Romeo's web agent then encrypts some string (the realm?) and the nonce with Romeo's private key. Only an agent trusted by Romeo can do this.
  4. The User Agent then sends a new GET request with the encrypted string, and his identifier, perhaps something like this
    GET /protected/juliette HTTP/1.0
    Host: juliette.org
    Authorization: RdfAuth id="http://romeo.name/#romeo" key="THE_REALM_AND_NONCE_ENCRYPTED"
    Content-Type: application/rdf+xml, text/rdf+n3
    Since we need an identifier, why not just use Romeos' foaf name? It happens to also point to his foaf file. All the better.
  5. Because Juliette's web server can then use Romeo's foaf name to GET his public foaf file, which contains a link to his public key, as explained in "Cryptographic Web of Trust".
  6. Juliette's web server can then query the returned representation, perhaps meshed with some other information in its database, with something equivalent to the following SPARQL query
    PREFIX wot: <http://xmlns.com/wot/0.1/>
    SELECT ?pgp
    WHERE {
         [] wot:identity <http://romeo.name/#romeo>;
            wot:pubkeyAddress ?pgp .
    The nice thing about working at the semantic layer, is that it decouples the spec a lot from the representation returned. Of course as usage grows those representations that are understood by the most servers will create a de facto convention. Intially I suggest using RDF/XML of course. But it could just as well be N3, RDFa, perhaps even some microformat dialect, or even some GRDDLable XML, as the POWDER working group is proposing to do.
  7. Having found the URL of the PGP key, Juliette's server, can GET it - and as with much else in this protocol cache it for future use.
  8. Having the PGP key, Juliette's server can now decrypt the encrypted string sent to her by Romeo's User Agent. If the decrypted string matches the expected string, Juliette will know that the User Agent has access to Romeo's private key. So she decides this is enough to trust it.
  9. As a result Juliette's server returns the protected representation.
Now Romeo's User Agent knows where Juliette is, displays it, and Romeo rushes off to see her.


It should be clear from the sketch what the numerous advantages of this system are over OpenId. (I can't speak of other authentication services as I am not a security expert).

  • The User Agent has no redirects to follow. In the above example it needs to request one resource http://juliette.org/ twice (2 and 4) but that may only be necessary the first time it accesses this resource. The second time the UA can immediately jump to step 3. [but see problem with replay attacks raised in the comments by Ed Davies, and my reply] Furthermore it may be possible - this is a question to HTTP specialists - to merge step 1 and 2. Would it be possible for a request 1. to return a 20x code with the public representation, plus a WWWAuthenticate header, suggesting that the UA can get a more detailed representation of the same resource if authenticated? In any case the redirect rigmarole of OpenId, which is really there to overcome the limitations of current web browsers, in not needed.
  • There is no need for an Attribute Exchange type service. Foaf deals with that in a clear and extensible RESTful manner. This simplifies the spec dramatically.
  • There is no need for an identity server, so one less point of failure, and one less point of control in the system. The public key plays that role in a clean and simple manner
  • The whole protocol is RESTful. This means that all representations can be cached, meaning that steps 5 and 7 need only occur once per individual.
  • As RDF is built for extensibility, and we are being architecturally very clean, the system should be able to grow cleanly.


I have been quietly exploring these ideas on the foaf and semantic web mailing lists, where I received a lot of excellent suggestions and feedback.


So I suppose I am now looking for feedback from a wider community. PGP experts, security experts, REST and HTTP experts, semantic web and linked data experts, only you can help this get somewhere. I will never have the time to learn these fields in enough detail by myself. In any case all this is absolutely obviously simple, and so completely unpatentable :-)

Thanks for taking the time to read this

Thursday Mar 20, 2008

how binary relations beat tuples

Last week I was handed a puzzle by Francois Bry: "Why does RDF limit itself to binary relations? Why this deliberate lack of expressivity?".

Logical Equivalence Reply

My initial answer was that all tuples could be reduced to binary relations. So take a simple table like this:

User IDnameaddressbirthdaycoursehomepage
1234Henry Story21 rue Saint Honoré
29 Julyphilosophyhttp://bblfish.net/
1235Danny AyersLoc. Mozzanella, 7
Castiglione di Garfagnana
14 Jansemwebhttp://dannyayers.com

The first row in the above column can be expressed as a set of binary relations as shown in this graph:

The same can clearly be done for the second row.

Since the two models express equivalent information I would opt aesthetically for the graph over the tuples, since it requires less primitives, which tends to make things simpler and clearer. Perhaps that can already be seen in the way the above table is screaming out for refactoring: a person may easily have more than one homepage. Adding a new homepage relation is easy, doing this in a table is a lot less so.

But this line of argument will not convince a battle worn database administrator. Both systems do the same thing. One is widely deployed, the other not. So that is the end of the conversation. Furthermore it seems clear that retrieving a row in a table is quick and easy. If you need chunks of information to be together that beats the join that seems to be required in the graph version above. Pragmatics beats aesthetics hands down it seems.

Global Distributed Open Data

The database engineer might have won the battle, but he will not win the war [1]. Wars are fought at a much higher level, on a global scale. The problem the Semantic Web is attacking is global data, not local data. On the Semantic Web, the web is the database and data is distributed and linked together. On the Semantic Web use case the data won't all be managed in one database by a few resource constrained superusers but distributed in different places and managed by the stake holder of that information. In our example we can imagine three stake holders of different pieces of information: Danny Ayers for his personal information, Me for mine, and the university for its course information. This information will then be available as resources on the web, returning different representations, which in one way or another may encode graphs such as the ones below. Note that duplication of information is a good thing in a distributed network.

By working with the most simple binary relations, it is easy to cut information up down to their most atomic unit, publish them anywhere on the web, distributing the responsibility to different owners. This atomic nature of relations also makes it easy to merge information again. Doing this with tuples would be unnecessarily complex. Binary relations are a consequence of taking the open world assumption seriously in a global space. By using Universal Resource Identifiers (URIs), it is possible for different documents to co-refer to the same entitities, and to link together entities in a global manner.

The Verbosity critique

Another line of attack similar to the first could be that rdf is just too verbose. Imagine the relation children which would relate a person to a list of their children. If one sticks just with binary relations this is going to be very awkward to write out. In a graph it would look like this.

image of a simple list as a graph

Which in Turtle would give something like this:

:Adam :children 
     [ a rdf:List;
       rdf:first :joe;
       rdf:rest [ a rdf:List;
            rdf:first :jane;
            rdf:rest rdf:nil ];
     ] .

which clearly is a bit unnecessarily verbose. But that is not really a problem. One can, and Turtle has, developed a notation for writing out lists. So that one can write much more simply:

:Adam :children ( :joe :jane ) .

This is clearly much easier to read and write than the previous way (not to speak about the equivalent in rdf/xml). RDF is a structure developed at the semantic level. Different notations can be developed to express the same content. The reason it works is because it uses URIs to name things.

Efficiency Considerations

So what about the implementation question: with tables oft accessed data is closely gathered together. This it seems to me is an implementation issue. One can easily imagine RDF databases that would optimize the layout in memory of their data at run time in a Just in Time manner, depending on the queries received. Just as the Java JIT mechanism ends up in a overwhelming number of cases to be faster than hand crafted C, because the JIT can take advantage of local factors such as the memory available on the machine, the type of cpu, and other issues, which a statically compiled C binary cannot do. So in the case of the list structure shown above there is no reason why the database could not just place the :joe and jane in an array of pointers.

In any case, if one wants distributed decentralised data, there is no other way to do it. Pragamatism does have the last word.


  1. Don't take the battle/war analogy too far please. Both DB technologies and Semantic Web ones can easily work together as demonstrated by tools such as D2RQ.

Wednesday Mar 19, 2008

Semantic Web for the Working Ontologist

I am really excited to see that Dean Allemang and Jim Hendler's book "Semantic Web for the Working Ontologist" is now available for pre-order on Amazon's web site. When I met Dean at Jazoon 2007 he let me have a peek at an early copy of this book[1]: it was exactly what I had been waiting a long time for. A very easy introduction to the Semantic Web and reasoning that does not start with the unnecessarily complex RDF/XML [2] but with the one-cannot-be-simpler triple structure of RDF, and through a series of practical examples brings the reader step by step to a full view of all of the tools in the Semantic Web stack, without a hitch, without a problem, fluidly. I was really impressed. Getting going in the Semantic Web is going to be a lot easier when this book is out. It should remove the serious problem current students are facing of having to find a way through a huge number of excellent but detailed specs, some of which are no longer relevant. One does not learn Java by reading the Java Virtual Machine specification or even the Java Language Specification. Those are excellent tools to use once one has read many of the excellent introductory books such as the unavoidable Java Tutorial or Bruce Eckel's Thinking in Java. Dean Allemang and Jim Hendler's books are going to play the same role for the Semantic Web. Help get millions of people introduced to what has to be the most revolutionary development in computer science since the development of the web itself. Go and pre-order it. I am going to do this right now.


  1. the draft I looked at 9 months ago had introductions to ntriples, turtle, OWL explained via rules, SPARQL, some simple well known ontologies such as skos and foaf, and a lot more.
  2. The W3C has recently published a new RDF Primer in Turtle in recognition of the difficulty of getting going when the first step requires understanding RDF/XML.
  3. Dean gave a talk at JavaOne that is now available online, which goes over the first chapters of the book. While you are waiting for the book, you can learn a lot by following this.

Friday Feb 15, 2008

Proof: Data Portability requires Linked Data

Data Portability requires Linked Data. To show this let me take a concrete and topical example that is the core use case of the Data Portability movement: Jane wants to move her account from social network A to social network B. And she wants to do this in a way that entails the minimal loss of information.

Let us suppose Jane wants to make a rich copy, and that she wants to do this without hyperdata. Ideally she would like to have exactly the same information in the new space as she had in the old space. So if Jane had a network of friends in social network A she would like to have the same network of friends in B. But this implies moving all the information about all her friends from A to B, including their social network too. For after all the great thing about one's friends is how they can help us make new friends. But then would one not want to move all the social network of one's friends too? Where does it stop? As William Blake said so well in Auguries of Innocence

        To see a world in a grain of sand,
	And a heaven in a wild flower,
	Hold infinity in the palm of your hand,
	And eternity in an hour.
the problem is that everything is linked in some way, and so it is impossible to move one thing and all its relations from one place to another using just copy by value, without moving everything. A full and rich copy is therefore impossible.

So what about pragmatically limiting ourselves to some subset of the information? We have to reduce our ambitions. So let us limit the data Jane can move to just her personal data and closest social network. So she copies some subset of the information about her friends over to network B. Nice, but who is going to keep that information up to date? When Jane's friend Jack moves house, how is Jane going to know about this in her new social network? Would Jack not have to keep his information on social Network B up to date too? And now if every one of Jack's 1000 friends moves to a different social network, won't he have to now keep 1000 identities up to date on each of those networks? Making it easy for Jane to move social network is going to make life hell for Jack it seems. Well of course not: Jack is never going to keep the information about himself up to date on these other social networks, however limited it is going to be. And so if Jane moves social network she is going to have to leave her friends behind.

The solution of course is not to try to copy the information about one's friends from one social network to another, but rather to move one's own information over and then link back to one's friends in their preferred social network. By linking by reference to one's friends identity one reduces to a minimum the information that needs to be ported whilst maintaining all the relationships that existed previously. Thus one can move one's identity without loss.

The rest follows nearly immediately from these observations. Since the only way to refer to resources in a global namespace is via URIs ( and the most practical way currently is to do this with URLs ), URI's will play the role of pointers in our space. This is the key architectural decision of the semantic web. So by giving people URLs as names we can point to our friends wherever they are, and even move our data without loss. All we need to do when we move our foaf file is to have the web server serve up a HTTP redirect message at the old URL, and all links to our old file will be redirected to our new home.


Tuesday Jan 15, 2008

Data Portability: The Video

Here is an excellent video to explain the problem faced by Web 2.0 companies and what Data Portability means. It is amazing how a good video can express something so much more powerfully, so much more directly than words can. Sit back and watch.

DataPortability - Connect, Control, Share, Remix from Smashcut Media on Vimeo.

Feeling better? You are gripped by the problem? Good. You should now find that my previous years posts start making a lot more sense :-)

Will the Data Portability group get the best solution together? I don't know. The problem with the name they have chosen is that it is so general, one wonders whether XML is not the solution to their problem. Won't XML make data portability possible, if everyone agrees on what they want to port? Of course getting that agreement on all the topics in the world is a never ending process.... Had they retained the name of the original group this stemmed from, Social Network Portability then one could see how to tackle this particular issue. And this particular issue seems to be the one this video is looking at.

But the question is also whether portability is the right issue. Well in some ways it is. Currently each web site has information locked up in html formats, in natural language (or even sometimes in jpegs (see the previous story of Scoble and Facebook), in order to make it difficult to export the data, which each service wants to hold onto as if it was theirs to own.

Another way of looking at this is that the Data Portability group cannot so much be about technology as policy. The general questions it has to address are question of who should see what data, who should be able to copy that data, and what they should be able to do with it. This does indeed involve identity technology insofar as all of the above questions turn around questions of identity ("who?"). Now if every site requires one to create a new identity in order to access one's data one has the nightmare scenario depicted in the video, where one has to maintain one's identity across innumerable sites. As a result the policy issue of Data Portability does require one to solve the technical problem of distributed identity: how can people maintain the minimum number of identities on the web? (ie not one per site) Another issue that follows right upon the first is that if one wants information to only be visible to a select group of people - the "who sees what" part of the question - then one also needs a distributed way to be able to specify group membership, be it friendship based or other. The video again makes that point very clearly why having to recreate one's social network on every site is impractical.

What may be misleading about the term Data Portability is that it may lead one to think that what one wants is to copy one's social information from one social service to another. That would just automate the job of what the video illustrates people having to do by hand currently. But that is not a satisfactory solution. Because one cannot extract a graph of information from one space to another without loss. If I extract my friends from LinkedIn into FaceBook, it is quite certain that Facebook will not recognise a large number of the people I know on LinkedIn. Furthermore the ported information on FaceBook would soon be out of date, as people updated their network and profiles on LinkedIn. Unless of course Facebook were able to make a constant copy of the information on LinkedIn. But that's impossible right? Wrong! That is the difference between copy by value and copy by reference. If FaceBook can refer to people on LinkedIn, then the data will always be as up to date as it can be. So this is how one moves from DataPortability to Linked Data, also known as hyper data.

Sunday Jan 06, 2008

2008: The Rise of Linked Data

Here is my one prediction for 2008. Social Networking's breakdown will lead to the rise of Linked Data. Here is the logic:

  1. Social Networking sites have grown tremendously over the last few years fuelled by huge profits from advertising dollars. When I worked at AltaVista it was well known that the more you knew about your users the more valuable an ad became. If you know all the friends, interests, habits of someone, and you know what they are doing right now, you can suggest exactly the right product at the right time to them. The cost of a simple add on AltaVista was $5 per thousand page views. If you knew a lot about what someone was looking for the value could go up to $50.
  2. The allure of profit is leading to an ever increasing number of players in this space. See the Social Networking 3.0 talk at Stanford earlier in 2007.
  3. This in turn leads to a fracturing of the Social Networking space. As more players enter the space, each ends up with a smaller and partial view of the whole graph or social relations.
  4. Which is leading to the need for Social Network Portability, and more generally Data Portability. Users such as Scoble want to use their data on their own computer and link it together. Social Network Providers such as Plaxo or Facebook have a financial interest in helping their users move with their social network to their service. Facebook helps users extract all the information from GMail. Plaxo wants to help users extract all the information from every other social network.
  5. Privacy concerns will mount tremendously as a result. Each social network will increase in their users the fear of giving their data over to other "spamming" services, to defend their position. But to do this they will make it more and more difficult to extract the data from their service, annoying and so going against their users desires for linking their information. This will seem more and more like an issue for anti trust involvement as the ire of more and more people mount.

The force of the above logic will release the energy needed for an investment in Linked Data tools such as Beatnik, since it solves all the problems mentioned above - at the expense of killing the dream some investors may have had of a world where they own Nineteen Eighty Four like, the world.




« July 2016