Wednesday Oct 07, 2009

Sketch of a RESTful photo Printing service with foaf+ssl

Let us imagine a future where you own your data. It's all on a server you control, under a domain name you own, hosted at home, in your garage, or on some cloud somewhere. Just as your OS gets updates, so all your server software will be updated, and patched automatically. The user interface for installing applications may be as easy as installing an app on the iPhone ( as La Distribution is doing).

A few years back, with one click, you installed a myPhoto service, a distributed version of fotopedia. You have been uploading all your work, social, and personal photos there. These services have become really popular and all your friends are working the same way too. When your friends visit you, they are automatically and seamlessly recognized using foaf+ssl in one click. They can browse the photos you made with them, share interesting tidbits, and more... When you organize a party, you can put up a wiki where friends of your friends can have write access, leave notes as to what they are going to bring, and whether or not they are coming. Similarly your colleagues have access to your calendar schedule, your work documents and your business related photos. Your extended family, defined through a linked data of family relationship (every member of your family just needs to describe their relation to their close family network) can see photos of your family, see the videos of your new born baby, and organize Christmas reunions, as well as tag photos.

One day you wish to print a few photos. So you go to web site we will provisionally call print.com. Print.com is neither a friend of yours, nor a colleague, nor family. It is just a company, and so it gets minimal access to the content on your web server. It can't see your photos, and all it may know of you is a nickname you like to use, and perhaps an icon you like. So how are you going to allow print.com access to the photos you wish to print? This is what I would like to try to sketch a solution for here. It should be very simple, RESTful, and work in a distributed and decentralized environment, where everyone owns and controls their data, and is security conscious.

Before looking at the details of the interactions detailed in the UML Sequence diagram below, let me describe the user experience at a general level.

  1. You go to print.com site after clicking on a link a friend of your suggested on a blog. On the home web page is a button you can click to add your photos.
  2. You click it, and your browser asks you which WebID you wish to use to Identify yourself. You choose your personal ID, as you wish to print some personal photos of yours. Having done that, your are authenticated, and print.com welcomes you using your nicknames and displays your icon on the resulting page.
  3. When you click a button that says "Give Print.com access to the pictures you wish us to print", a new frame is opened on your web site
  4. This frame displays a page from your server, where you are already logged in. The page recognized you and asks if you want to give print.com access to some of your content. It gives you information about print.com's current stock value on NASDAQ, and recent news stories about the company. There is a link to more information, which you don't bother exploring right now.
  5. You agree to give Print.com access, but only for 1 hour.
  6. When your web site asks you which content you want to give it access to, you select the pictures you would like it to have. Your server knows how to do content negotiation, so even though copying each one of the pictures over is feasible, you'd rather give print.com access to the photos directly, and let the two servers negotiate the best representation to use.
  7. Having done that you drag and drop an icon representing the set of photos you chose from this frame to a printing icon on the print.com frame.
  8. Print.com thanks you, shows you icons of the pictures you wish to print, and tells you that the photos will be on their way to your the address of your choosing within 2 hours.

In more detail then we have the following interactions:

  1. Your browser GETs print.com's home page, which returns a page with a "publish my photos" button.
  2. You click the button, which starts the foaf+ssl handshake. The initial ssl connection requests a client certificate, which leads your browser to ask for your WebID in a nice popup as the iPhone can currently do. Print.com then dereferences your WebId in (2a) to verify that the public key in the certificate is indeed correct. Your WebId (Joe's foaf file) contains information about you, your public keys, and a relation to your contact addition service. Perhaps something like the following:
    :me xxx:contactRegistration </addContact> .
    Print.com uses this information when it creates the resulting html page to point you to your server.
  3. When you click the "Give Print.com access to the pictures you wish us to print" you are sending a POST form to the <addContact> resource on your server, with the WebId of Print.com <https://nasdaq.com/co/PRNT#co> in the body of the POST. The results of this POST are displayed in a new frame.
  4. Your web server dereferences Print.com, where it gets some information about it from the NASDAQ URL. Your server puts this information together (4a) in the html it returns to you, asking what kind of access you want to give this company, and for how long you wish to give it.
  5. You give print.com access for 1 hour by filling in the forms.
  6. You give access rights to Print.com to your individual pictures using the excellent user interface available to you on your server.
  7. When you drag and drop the resulting icon depicting the collection of the photos accessible to Print.com, onto its "Print" icon in the other frame - which is possible with html5 - your browser sends off a request to the printing server with that URL.
  8. Print.com dereferences that URL which is a collection of photos it now has access to, and which it downloads one by one. Print.com had access to the photos on your server after having been authenticated with its WebId using foaf+ssl. (note: your server did not need to GET print.com's foaf file, as it still had a fresh version in its cache). Print.com builds small icons of your photos, which it puts up on its server, and then links to in the resulting html before showing you the result. You can click on those previews to get an idea what you will get printed.

So all the above requires very little in addition to foaf+ssl. Just one relation, to point to a contact-addition POST endpoint. The rest is just good user interface design.

What do you think? Have I forgotten something obvious here? Is there something that won't work? Comment on this here, or on the foaf-protocols mailing list.

Notes

Creative Commons License
print.com sequence diagram by Henry Story is licensed under a Creative Commons Attribution 3.0 United States License.
Based on a work at blogs.sun.com.

Wednesday Sep 24, 2008

Serialising Java Objects to RDF with Jersey

Jersey is the reference implementation of JSR311 (JAX-RS) the Java API for RESTful Web Services. In short JSR311 makes it easy to publish graphs of Java Objects to the web, and implement update and POST semantics - all this using simple java annotations. It makes it easy for Java developers to do the right thing when writing data to the web.

JAX-RS deals with the mapping of java objects to a representation format - any format. Writing a good format, though, is at least as tricky as building RESTful services. So JAX-RS solves only half the problem. What is needed is to make it easy to serialize any object in a manner that scales to the web. For this we have RDF, the Resource Description Framework. By combining JAX-RS and the So(m)mer's @rdf annotation one can remove remove the hurdle of having to create yet another format, and do this in a way that should be really easy to understand.

I have been wanting to demonstrate how this could be done, since the JavaOne 2007 presentation on Jersey. Last week I finally got down to writing up some initial code with the help of Paul Sandoz whilst in Grenoble. It turned out to be really easy to do. Here is a description of where this is heading.

Howto

The code to do this available from the so(m)mer subversion repository, in the misc/Jersey directory. I will refer and link to the online code in my explanations here.

Annotate one's classes

First one needs to annotate one's model classes with @rdf annotations on the classes and fields. This is a way to give them global identifiers - URIs. After all if you are publishing to the web, you are publishing to a global context so global names are the only way to remove ambiguity. So for example our Person class can be written out like this:

@rdf(foaf+"Person")
public class Person extends Agent {
    static final String foaf = "http://xmlns.com/foaf/0.1/";

    @rdf(foaf+"surname") private Collection<String> surnames = new HashSet<String>();
    @rdf(foaf+"family_name") private Collection<String> familynames = new HashSet<String>();
    @rdf(foaf+"plan") private Collection<String> plans = new HashSet<String>();
    @rdf(foaf+"img") private Collection<URL> images = new HashSet<URL>();
    @rdf(foaf+"myersBriggs") private Collection<String> myersBriggss = new HashSet<String>();
    @rdf(foaf+"workplaceHomepage") private Collection<FoafDocument> workplaceHomePages = new HashSet<FoafDocument>();
    ....
}

This just requires one to find existing ontologies for the classes one has, or to publish new ones. (Perhaps this framework could be extended so as to automate the publishing of ontologies as deduced somehow form Java classes? - Probably a framework that could be built on top of this)

Map the web resources to the model

Next one has to find a mapping for web resources to objects. This is done by subclassing the RdfResource<T> template class, as we do three times in the Main class. Here is a sample:

 @Path("/person/{id}")
   public static class PersonResource extends RdfResource<Employee> {
      public PersonResource(@PathParam("id") String id) {
          t = DB.getPersonWithId(id);
      }
   }

This just tells Jersey to publish any Employee object on the server at the local /person/{id} url. When a request for some resource say /person/155492 is made, a PersonResource object will be created whose model object can be found by querying the DB for the person with id 155492. For this of course one has to somehow link the model objects ( Person, Office,... in our example ) to some database. This could be done by loading flat files, querying an ldap server, or an SQL server, or whatever... In our example we just created a simple hard coded java class that acts as a DB.

Map the Model to the resource

An object can contain pointers to other objects. In order for the serialiser to know what the URL of objects are one has to map model objects to web resources. This is done simply with the static code in the same Main class [ looking for improovements here too ]

   static {
      RdfResource.register(Employee.class, PersonResource.class);
      RdfResource.register(Room.class, OfficeResource.class);
      RdfResource.register(Building.class, BuildingResource.class);              
   }

Given an object the serialiser (RdfMessageWriter) can then look up the resource URL pattern, and so determine the object's URL. So to take an example, consider an instance of the Room class. From the above map, the serialiser can find that it is linked to the OfficeResource class from which it can find the /building/{buildingName}/{roomId} URI pattern. Using that it can then call the two getters on that Room object, namely getBuildingName() and getRoomId() to build the URL referring to that object. Knowing the URL of an object means that the serialiser can stop its serialisation at that point if the object is not the primary topic of the representation. So when serialising /person/155492 the serialiser does not need to walk through the properties of /building/SCA22/3181. The client may already have that information and if not, the info is just a further GET request away.

Running it on the command line

If you have downloaded the whole repository you can just run

$ ant run
from the command line. This will build the classes, recompile the @rdf annotated classes, and start the simple web server. You can then just curl for a few of the published resources like this:
hjs@bblfish:0$ curl -i http://localhost:9998/person/155492
HTTP/1.1 200 OK
Date: Wed, 24 Sep 2008 14:37:38 GMT
Content-type: text/rdf+n3
Transfer-encoding: chunked

<> <http://xmlns.com/foaf/0.1/primaryTopic> </person/155492#HS> .
</person/155492#HS> <http://xmlns.com/foaf/0.1/knows> <http://www.w3.org/People/Berners-Lee/card#i> .
</person/155492#HS> <http://xmlns.com/foaf/0.1/knows> </person/528#JG> .
</person/155492#HS> <http://xmlns.com/foaf/0.1/birthday> "29_07" .
</person/155492#HS> <http://xmlns.com/foaf/0.1/name> "Henry Story" .

The representation returned not a very elegant serialisation of the Turtle subset of N3. This makes the triple structure of RDF clear- subject relation object - and it uses relative URLs to refer to local resources. Other serialisers could be added, such as for rdf/xml. See the todo list at the end of this article.

The represenation says simple that this resource <> has as primary topic the entity named by #HS in the document. That entity's name is "Henry Story" and knows a few people, one of which is refered to via a global URL http://www.w3.org/People/Berners-Lee/card#i, and the other via a local URL /person/528#JG.

We can find out more about the /person/528#JG thing by making the following request:

hjs@bblfish:0$ curl -i http://localhost:9998/person/528#JG
HTTP/1.1 200 OK
Date: Wed, 24 Sep 2008 14:38:10 GMT
Content-type: text/rdf+n3
Transfer-encoding: chunked

<> <http://xmlns.com/foaf/0.1/primaryTopic> </person/528#JG> .
</person/528#JG> <http://xmlns.com/foaf/0.1/knows> </person/155492#HS> .
</person/528#JG> <http://xmlns.com/foaf/0.1/knows> </person/29604#BT> .
</person/528#JG> <http://xmlns.com/foaf/0.1/knows> <http://www.w3.org/People/Berners-Lee/card#i> .
</person/528#JG> <http://www.w3.org/2000/10/swap/pim/contact#office> </building/SCA22/3181#it> .
</person/528#JG> <http://xmlns.com/foaf/0.1/birthday> "19-05" .
</person/528#JG> <http://xmlns.com/foaf/0.1/name> "James Gosling" .

... where we find out that the resource named by that URL is James Gosling. We find that James has an office named by a further URL, which we can discover more about with yet another request


hjs@bblfish:0$ curl -i http://localhost:9998/building/SCA22/3181#it
HTTP/1.1 200 OK
Date: Wed, 24 Sep 2008 14:38:38 GMT
Content-type: text/rdf+n3
Transfer-encoding: chunked

<> <http://xmlns.com/foaf/0.1/primaryTopic> </building/SCA22/3181#it> .
</building/SCA22/3181#it> <http://www.w3.org/2000/10/swap/pim/contact#address> _:2828781 .
_:2828781 a <http://www.w3.org/2000/10/swap/pim/contact#Address> .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#officeName> "3181" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#street> "4220 Network Circle" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#stateOrProvince> "CA" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#city> "Santa Clara" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#country> "USA" .
_:2828781 <http://www.w3.org/2000/10/swap/pim/contact#postalCode> "95054" .

Here we have a Location that has an Address. The address does not have a global name, so we give it a document local name, _:2828781 and serialise it in the same representation, as shown above.

Because every resource has a clear hyperlinked representation we don't need to serialise the whole virtual machine in one go. We just publish something close to the Concise Bounded Description of the graph of objects.

Browsing the results

Viewing the data through a command line interface is nice, but it's not as fun as when viewing it through a web interface. For that it is best currently to install the Tabulator Firefox plugin. Once you have that you can simply click on our first URL http://localhost:9998/person/155492. This will show up something like this:

picture of tabualator on loading /person/155492

If you then click on JG you will see something like this:

picture tabulator showing James Gosling

This it turns out is a resource naming James Gosling. James knows a few people including a BT. The button next to BT is in blue, because that resource has not yet been loaded, whereas the resource for "Henry Story" has. Load BT by clicking on it, and you get

picture of tabulator showing Henry Story knowning Bernard Traversat

This reveals the information about Bernard Traversat we placed in our little Database class. Click now on the i and we get

Tabulator with info about Tim Berners Lee

Now we suddenly have a whole bunch of information about Tim Berners Lee, including his picture, some of the people he has listed as knowing, where he works, his home page, etc... This is information we did not put in our Database! It's on the web of data.

One of the people Tim Berner's Lee knows is our very own Tim Bray.

tabulator showing info from dbpedia on Tim Bray

And you can go on exploring this data for an eternity. All you did was put a little bit of data on a web server using Jersey, and you can participate in the global web of data.

Todo

There are of course a lot of things that can be done to improove this Jersey/so(m)mer mapper. Here are just a few I can think of now:

  • Improove the N3 output. The code works with the examples but it does not deal well with all the literal types, nor does it yet deal with relations to collections. The output could also be more human readable by avoiding repetiations.
  • Refine the linking between model and resources. The use of getters sounds right, but it could be a bit fragile if methods are renamed....
  • Build serialisers for rdf/xml and other RDF formats.
  • Deal with publication of non information resources, such as http:// xmlns.com/foaf/0.1/Person which names the class of Persons. When you GET it, it redirects you to an information resources:
    hjs@bblfish:1$ curl -i http://xmlns.com/foaf/0.1/Person
    HTTP/1.1 303 See Other
    Date: Wed, 24 Sep 2008 15:30:14 GMT
    Server: Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.61 OpenSSL/0.9.7e mod_fastcgi/2.4.2 Phusion_Passenger/2.0.2 DAV/2 SVN/1.4.2
    Location: http://xmlns.com/foaf/spec/
    Vary: Accept-Encoding
    Content-Length: 234
    Content-Type: text/html; charset=iso-8859-1
    
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    <title>303 See Other</title>
    </head><body>
    <h1>See Other</h1>
    <p>The answer to your request is located <a href="http://xmlns.com/foaf/spec/">here</a>.</p>
    </body></html>
    
    This should also be made easy and foolproof for Java developers.
  • make it industrial strength...
  • my current implementation is tied much too strongly to so(m)mer. The @rdf annotated classes get rewritten to create getters and setters for every field, that links to the sommer mappers. This is not needed here. All we need is to automatically add the RdfSerialiser interface to make it easy to access the private fields.
  • One may want to add support for serialising @rdf annotated getters
  • Add some basic functionality for POSTing to collections or PUTing resources. This will require some thought.

Bookmarks: digg+, reddit+, del.icio.us+, dzone+, facebook+

Tuesday Jun 24, 2008

Webifying Integrated Development Environments

IDEs should be browsers of code on a Read Write Web. A whole revolution in how to build code editors is I believe hidden in those words. So let's imagine it. Fiction anticipates reality.

Imagine your favorite IDE, a future version of NetBeans perhaps or IntelliJ, which would make downloading a new project as easy as dragging and dropping a project url onto your IDE. The project home page would point to a description of the location of the code, the dependencies of this project on other projects, described themselves via URL references, which themselves would be set up in a similar manner. Let's imagine further: instead of downloading all the code from CVS, think of every source code document as having a URL on the web. ( Subversion is in fact designed like this, so this is not so far fetched at all.) And let's imagine that NetBeans thinks about each software component primarily via this URL.
Since every piece of code and every library has a URL, the IDE would be able to use RESTful architectural principles of the web. A few key advantages of this are

  • Caching: web architecture is the ability to cache information on the network or locally without ambiguity. This is how your web browser works ( though it could work better ). To illustrate: once a day Google changes its banner image. Your browser and every browser on earth only fetches that picture once a day, even if you do 100 searches. Does Google serve one image to each browser? No! numerous caches (company, country, or other) cache that picture and send it to the browser without sending the request all the way to the search engine, reducing the load on their servers very significantly.
  • Universal names: since every resource has a URL, any resource can relate in one way or another to any other resource wherever it is located. This is what enables hypertext and what is enabling hyperdata.

Back to the IDE. So now that all code, all libraries, can be served up RESTfully in a Resource Oriented Architecture what does this mean to the IDE? Well a lot. Each may seem small, but together they pack a huge punch:
  • No need to download libraries twice: if you have been working on open source projects at all frequently you must have noticed how often the same libraries are found in each of the projects you have downloaded. Apache logging is a good example.
  • No need to download source code: it's on the web! You don't therefore need a local cache of code you have never looked at. Download what you need when you need it (and then cache it!): the Just in Time principle.
  • Describe things globally: Since you have universal identifiers you can now describe how source code relates to documentation, to people working on the code, or anything else in a global way, that will be valid for all. Just describe the resources. There's a framework around just for that, that is very easy to use with the right introduction.

The above advantages may seem rather insignificant. After all, real developers are tough. They use vi. (And I do). So why should they change? Well notice that they also use Adobe Air or Microsoft Silverlight. So productivity considerations do in fact play a very important factor in the software ecosystem.
Don't normal developers just work on a few pieces of code? Well speaking for myself here, I have 62 different projects in my /Users/hjs/Programming directory, and in each of these I often have a handful of project branches. As more and more code is open source, and owned and tested by different organizations, the number of projects available on the web will continue to explode, and due to the laziness principle the number of projects using code from other projects will grow further. Already whole operating systems consisting of many tens of thousands of different modules can be downloaded and compiled. The ones I have downloaded are just the ones I have had the patience to get. Usually this means jumping through a lot of hoops:

  1. I have to finding the web site of the code. And I may only have a jar name to go by. So Google helps. But that is a whole procedure in itself that should be unecessary. If you have an image in your browser you know where it is located by right-clicking over it and selecting the URL. Why not so with code?
  2. Then I have to browse a web page, which may not be written in my language, and find the repository of the source code
  3. Then I have to find the command line to download the source code, or the command in the IDE and also somehow guess which version number produced the jar I am using.
  4. Once downloaded, and this can take some time, I may have to find the build procedure. There are a few out there. Luckily ant and maven are catching on. But some of these files can be very complicated to understand.
  5. Then I have to link the source code on my local file system to the jar on my local file system my project is using. In NetBeans this is exceedingly tedious - sometimes I have found it to be close to impossible even. IntelliJ has a few little tricks to automate some of this, but it can be pretty nasty too, requiring jumping around different forms. Especially if a project has created a large number of little jar files.
  6. And then all that work is only valid for me. Because all references are to files on my local file system, they cannot be published. NetBeans is a huge pain here in that it often creates absolute file URLs in its properties files. By replacing them with relative urls one can get publish some of the results, but at the cost of copying every dependency into the local repository. And working out what is local and what is remote can take up a lot of time. It will work on my system, but not on someone else's.
  7. Once that project downloaded one may discover that it depends on yet another project, and so we have to go back to step 1.

So doing the above is currently causing me huge headaches even for very simple projects. As a result I do it a lot less often than I could, missing valuable opportunities as a result. Each time I download a project in order to access the sources to walk through my code and find a bug, or to test out a new component I have to do all that download rigmarole described above. If you have a deadline, this can be a killer.

So why do we have to tie together all the components on our local file system? This is because the IDE's are not referring to the resources with global identifiers. The owner of the junit project should say somewhere, in his doap file perhaps that:

 
   @prefix java: <http://java.net/ont/java#> . #made this up
   @prefix code: <http://todo.eg/#> .

   <http://project.eg/svn/lib/junit-4.0.jar> a java:Jar;
         code:builtFrom <http://junit.sourceforge.net/> .

   #what would be needed here needs to be worked out more carefully. The point is that we don't
   #at any point refer to any local file.

Because this future IDE we are imagining together will then know that it has stored a local copy of the jar somewhere on the local file system, and because it will know where it placed the local copy of the source code, it will know how the cached jar relates to the cached source code, as illustrated in the diagram above. So just as when you click on a link on your web browser you don't have to do any maintenance to find out where the images and html files are cached on your hard drive, and how one resource (you local copy of an image) relates to the web page, so we should not have to do any of this type of work in our Development Environment either.

From here many other things follow. A couple of years ago I showed how this could be used link source code to bugs, to create a distributed bug database. Recently I showed how one could use this to improve build scripts. Why even download a whole project if you are stepping through code? Why not just fetch the code that you need when you need it from the web? One HTTP GET at a time. The list of functional improvements is endless. I welcome you to list some that you come up with in the comments section below.

If you want to make a big impact in the IDE space, that will be the way to go.

Saturday Sep 08, 2007

the limits of a free flickr account

I took some time, but I just hit one of the limits of my free flickr account. Here is the message I am seeing at the top of my account:

You've run into one of the limits of a free account. Your free account will only display the most recent 200 photos you've uploaded. All of your photos beyond 200 will remain hidden from view until you either delete newer photos, or upgrade to a Pro account.

None of your photos have been deleted, and if you upgrade, they'll all come back unharmed.

To get a pro account I need to shell out $24.95 per year. This is not unreasonable, but it is making me pause and think how much I want to continue with this service...

What are the alternatives? Well I have my own server at bblfish.net. I am already paying for that service, so I might as well use it more fully, and upload my pictures there. But pictures take up quite a lot of space so this is not going to end up being cheaper, as it will use up more space and force me at some time to either increase the space on my rented server or even buy my own server. What would I get for that price? I would certainly get more control over my work, but at the cost of more work on my part. Publishing photos could be done simply with an improved BlogEd, which can push those photos to the file system. This would make publishing easy, but would not give me the interactive collaborative features of flickr, where people can directly add notes to the pictures, tag them, etc... Adding that functionality is not a huge amount of work, but it makes maintenance of the service more difficult, which means costs in developer time, which has to be paid somehow too...

What is the price of freedom? Owning your content at URLs you control may long term be worth quite a lot, a bet Tim Bray clearly is making, by hosting everything on his own server...

Wednesday Aug 22, 2007

SPARQLing AltaVista: the meaning of forms

Did you know that AltaVista has a SPARQL endpoint? And that all of its results are served up that way? No? Well take the Red Pill and I will show you how deep the rabbit hole goes...

Take the query for the three words "matrix rabbit hole". Go to AltaVista and enter those words into the search box. Press "Find" and you will end up at the result page http://www.altavista.com/web/results?itag=ody&q=rabbit+hole+matrix&kgs=1&kls=0. This page lists the following information for each result:

  • The title of the page
  • The link to the page
  • An extract of the page containing the relevant words
  • A link to more results from that particular web site
In other words it is just the result of the following SPARQL query [1]:
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PRFIX eg: <http://altavista.eg/ont>
PREFIX pf: <http://jena.hpl.hp.com/ARQ/property#>
CONSTRUCT {
     ?page dc:title ?title;
           eg:summary ?summary;
           eg:moreResults ?more .
} WHERE {
     ?page dc:title ?title;
           eg:content ?content;
           eg:summary ?summary;
           eg:moreResults ?more .
     ?content pf:textMatch "+matrix +rabbit +hole" .
}
LIMIT 10
OFFSET 0

The AltaVista engineers - and I know them well having worked there for 5 years - of course understand User Interface issues very well, and so they don't return the default XML result format. They pass all their results first through a clever and optimised XSLT transform, that gives you the page that you now see in your browser.

In order to do this, the AltaVista engineers developed - and I can now speak openly about this - a clever mapping between html forms and SPARQL queries. Sadly it is such a long time ago that I worked there now, that my memory is a little dim on the exact manner in which they did this. So please forgive my mistakes. But I am sure we can work this out together.
Html Forms consist essentially of a number of key-value pairs where the end user is asked to provide the values, a form processing agent URL, and an action button if the user answers the question asked of him. Given that, the trick is just to create a simple SPARQL template language, so that one can relate a form processing agent to one or more SPARQL query templates. What does a SPARQL query template look like? Well it is really very similar to a SPARQL query, except that it has ??vars which need to be replaced by values from the form. So the SPARQL template associated with the front page form could be expressed like this:

@prefix fm: <http://altavisat.eg/ont#> .

<http://www.altavista.com/web/results>  a fm:FormHandler;
    fm:template """
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PRFIX eg: <http://altavista.eg/ont>
PREFIX pf: <http://jena.hpl.hp.com/ARQ/property#>
CONSTRUCT {
     ?page dc:title ?title;
           eg:summary ?summary;
           eg:moreResults ?more .
} WHERE {
     ?page dc:title ?title;
           eg:content ?content;
           eg:summary ?summary;
           eg:moreResults ?more .
     ?content pf:textMatch ??(\\" ??q \\")
}
LIMIT ???lmt
OFFSET ??( ??stq \* ???lmt )
""" .

So when AltaVista receives a form submission, the AV front end decode it and replaces all the ??key patterns above with the value of key as passed by the form. It then replaces all the ???keyparam patterns with default values set by the user and available from his session state. Finally the ??( ... ) operations are exectued. This can be in the form of a multiplication or a string concatenation as shown above, or other operations such as dealing with defaults. Having done that you end up with the previous SPARQL query, which is ready to be sent to the back end engines. The back engines have a more powerful language to play with, allowing AltaVista to propose new paying services to large customers such as Yahoo [2].

This has a few advantages. It reduces the size of the forms POSTED to AltaVista: a SPARQL query would take a lot more space, which would end up taking a fraction of a second off the processing time and so make it ever so much more painfully obvious that the redirect they are doing to av.rds.yahoo.com is destroying their performance. It would also give end users more freedom than is needed: It is a good policy decision to reduce the uses of a tool, when one aims to shrink one's market.

This mapping could be extreemly useful in a number of other ways.
For one it would help make it clear to machines what the meaning of a form is. Forms are questions asked to an agent. The meaning of the question is usually obvious to a human end user who speaks the language of the web page that is being shown. But for a machine to do the same, it helps to map the form to a semanticall defined query, which can be reasoned with. In this case the answers given by the human is used to construct a question that is sent to the server. In other cases the form is asking the user for his desired, and using this to construct an action. There is some interesting work in mapping different uses of forms to rdf still, but I think this does bring a key element into play.
Having a machine understandable version of a form means that a robot can start putting his rdf glasses and see things right. All that would be needed would be to link the form handler to an XSLT that could transform the resulting html to the SPARQL result format, and each of the thousands of existing web forms suddenly become transparent to the world of machine agents. [3]
It could also help reduce the work of defining new Protocols. The good part of OpenId Attribute Exchange for example is just a complex specification for a limited SPARQL template, if you put your rdf glasses on.[4]

With time you get to see the real structure of the world. As that happens the questions you start asking become a lot more interesting.

Notes

  1. The pf:textMatch relation is defined by the LARQ, the Jena Lucene-ARQ free text indexing for SPARQL extension, and it makes a lot of sense.
    The namesppace is sadly not dereferenceable. IE. Clicking on http://jena.hpl.hp.com/ARQ/property does not give you the definition of the relation. It would be nice if it did.
    Note also how in SPARQL you can have literals as subjects. This is explained in section 12.1.4 of the current SPARQL query language specification.
    Thanks to Andy Seaborne for the link.
  2. I do know of course that AltaVista is part of Yahoo! now. And by the way all of the above is meant to be taken with a pinch of salt red pills.
  3. This is called screen scraping, and is of course more work for the consumer. It is nicer when the information provider has a strong interest in providing a stable format.
  4. A large part of the spec is a duplication of the work that should be done by HTTP verbs such as GET, PUT, POST and DELETE. Using the Atom Protocol to publish a foaf file would deal with a large part of the spec in a few lines. Well that's what it looks like to me after studying the spec for a few hours only, and so I may have missed something.

Friday Aug 17, 2007

Open Data: Information wants to be linked

With over 2 billion relations from the great web community data projects such as Wikipedia, Project Gutenberg, Music Brainz, and many more... the Linking Open Data initiative is tying together a vast pool of quality machine readable information on which one can run any of the over 500 Semantic Web tools. As the value of linked information increases much faster than the networks described by Metcalf's Law, the value of this must be tremendous.

By creating data browsing interfaces such as Tabulator, one has a very simple RESTful, Resource Oriented Architecture API to work with. With various SPARQL endpoints available or to be built, one can treat that information like a hugely powerful database.

Forget Web APIs: long live linked data!

Some of the projects listed are:

Tuesday Jul 03, 2007

Restful semantic web services

Here is my first stab at an outline for what a restful semantic web services would look like.

Let me start with the obvious. Imagine we have an example shopping service, at http://shop.eg/, which sells books. Clearly we would want URLs for every book that we wish to buy, with RDF representations at the given URL. As I find RDF/XML hard to read and write, I'll show the N3 representations. So to take a concrete example, let us imagine our example shopping service selling the book "RESTful Web Services" at the URL http://shop.eg/books/isbn/0596529260 . If we do an HTTP GET on that URL we could receive the following representation:


@prefix : <http://books.eg/ns#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix shop: <http://shopping.eg/ns#> .
@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix currency: <http://bank.eg/currencies#> .


<#theBook> a shop:Book, shop:Product;
   dc:title "Restful Web Services"@en ;
   dc:creator [ a foaf:Person; foaf:name "Sam Ruby"],
             [ a foaf:Person; foaf:name "Leonard Richardson"] ;
   dc:contributor [ a foaf:Person; foaf:name "David Heinemeier Hansson" ];
   dc:publisher <http://www.oreilly.com/>;
   dct:created "08-07-2007T"\^\^xsd:dateTime;
   dc:description """This is the first book that applies the REST design philosophy to real web services. It sets 
 down the best practices you need to  make your design a success, and the techniques you need to turn your 
 design into working code. You can harness the power of the Web for programmable applications: you just 
 have to work with the Web instead of against it. This book shows you how."""@en;
   shop:price "26.39"\^\^currency:dollars;
   dc:subject </category/computing>, </category/computing/web>, </category/computing/web/rest>, 
                   </category/computing/architecture>,</category/computing/architecture/REST>, </category/computing/howto> .


So we can easily imagine a page like this for every product. These pages can be accessible either by browsing the categories pages, querying a SPARQL endpoint, among many other ways. It should be very easy to generate such representations for a web site. All it requires is to build up an ontology of products - which the shop already has available, if only for the purposes of building inventories - and tie these to the database using a tool such as D2RQ, or a combination of JSR311 and @rdf annotations (see so(m)mer).

Now what is missing is a way to let the browser know what it can do with this product. The simplest possible way of doing this would be to create a specialized relation for that web service to POST some information to a Cart resource, describing the item to be added to the cart. Perhaps something like:

<#theBook> shop:addToCart <http://shop.eg/cart/> .

This relation would just mean that one has to POST the url to the cart, to have it added there. The cart itself may then have a shop:buy relation to some resource, which by convention the user agent would need to send a credit card, expiration date, and other information to.
This means that one would have to define a number of RDF relationships, for every type of action one could do in a shop (and later on the web), and explain the types of messages to be sent to the endpoint, and what their consequences are. This is simple but it does not seem very extensible. What if one wants to buy the hard copy version of the book, or 10 items of the book? The hard copy version of course could have its own URL, and so it may be as simple as placing the buy relation on that page. But is this going to work with PCs where one can add and remove a huge number of pieces. I remember Steve Jobs being proud of the huge number of different configurations one could buy his desktop systems with, well over 100 thousand different configurations I remember. This could make it quite difficult to navigate a store, if one is not careful.

On the current web this is dealt with by using html forms, which can allow the user to choose between a large number of variables, by selecting check boxes, combo boxes, drop down menues and more, and then POST a representation to a collection, and thereby create a new action, such as adding the product to the cart, or buying it. The person browsing the site knows what the action does, because it is usually written out in a natural language, in a way that makes it quite obvious to a human being. The person then does that action because he desires to do so, because he wishes his desires to be fulfilled. Now this may seem very simple, but just consider the innumerable types of actions that we can fulfill using the very simple tool of html forms: we can add things to a virtual cart, buy things, comment about things, search for things, organise a meeting, etc, etc.... So forms can be seen both as shortcuts to navigate to a large number of different resources, and to create new resources (usually best done with POST).
If we want software agents to do such tasks for us we need both to have something like a machine understandable form, and some way of specifying what the action of POSTing the form will have on the world. So we need to find a way to do what the web does in a more clearly specified way, so that even machines, or simple user agents can understand it. Let's look at each one:

  • Forms are ways of asking the user to bind results to variables
  • the variables can the be used to build something, such as a URL, or a message.
  • The form then specifies the type of action to do with the constructed message, such as a GET, POST, PUT, etc...
  • The human readable text explains what the result of the action is, and what the meaning of each of the fields are.

Now what semantic technology binds variables to values? Which one asks questions? SPARQL comes immediately to mind. Seeing this and remembering a well known motto of sales people "Satisfy the customer every desire" a very general but conceptually simple solution to this problem occurred to me. It may seem a little weird at first (and perhaps it will continue to seem weird) but I thought it is elegant enough to be used as a starting point. The idea is really simple: the representation returned by the book resource will specify a collection end point to POST RDF too, and it will specify what to POST back by sending a SPARQL query in the representation. It will then be up to the software agent reading the representation to answer the query if he wishes a certain type of action to occur. If he understand the query he will be able to answer, if he does not, there should be no results. He need not do anything with the query at all.

The following is the first thing that occurred to me. The details are less important than the principle of thinking of forms as asking the client a question.

PREFIX shop: <http://shopping.eg/ns#>
PREFIX bdi: <http://intentionality.eg/ns#>
CONSTRUCT {
     ?mycart a shop:Cart ;
             shop:contains [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity ?q ;
                             ]  .
}  WHERE {
       ?mycart a shop:Cart ;
               shop:for ?me ;
               shop:ownedBy <http://shop.eg/>.
       GRAPH ?desire {
               ?mycart shop:contains
                            [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity ?q ;
                            ]  .

       }
       ?desire bdi:of ?me .
       ?desire bdi:fulfillby "2007-07-30T..."\^\^xsd:dateTime .
}

So this is saying quite simply: Find out if you want to have your shopping cart filled up with a number of this book. The user agent (the equivalent of the web browser) asks its data store the given SPARQL query. It asks itself whether it desires to add a number of books to its shopping cart, and if it wishes that desire to be fulfulled by a certain time. If the agent does not understand the relations in the query, then the CONSTRUCT clause will return an empty graph. If it does understand it, and the query returns a result, then it is because it wished the action to take place. The constructed graph may be something like:

@prefix shop: <http://shopping.eg/ns#>
 <http://shop.eg/cart/bblfish/> a shop:Cart ;
             shop:contains [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity 2 ;
                             ]  .

This can then be POSTed to the collection end point http://shop.eg/cart/, with the result of adding two instances of the book to the cart. Presumably the cart would return a graph with the above relations in it plus another SPARQL query explaining how to buy the items in the cart.

So the full RDF for the book page would look something like this:

@prefix : <http://books.eg/ns#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix shop: <http://shopping.eg/ns#> .
@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix currency: <http://bank.eg/currencies#> .


<#theBook> a shop:Book, shop:Product;
   dc:title "Restful Web Services"@en ;
   dc:creator [ a foaf:Person; foaf:name "Sam Ruby"],
             [ a foaf:Person; foaf:name "Leonard Richardson"] ;
   dc:contributor [ a foaf:Person; foaf:name "David Heinemeier Hansson" ];
   dc:publisher <http://www.oreilly.com/>;
   dct:created "08-07-2007T"\^\^xsd:dateTime;
   dc:description """This is the first book that applies the REST design philosophy to real web services. It sets 
 down the best practices you need to make your design a success, and the techniques you need to turn your
  design into working code. You can harness the power of the Web for  programmable applications: you just 
 have to work with the Web instead of against it. This book shows you how."""@en;
   shop:price "26.39"\^\^currency:dollars;
   dc:subject </category/computing>, </category/computing/web>, </category/computing/web/rest>, 
                      </category/computing/architecture>,</category/computing/architecture/REST>,</category/computing/howto>;
   shop:addToCart [ a Post;
                    shop:collection <http://shop.eg/cart/>;
                    shop:query """
PREFIX shop: <http://shopping.eg/ns#>
PREFIX bdi: <http://intentionality.eg/ns#>
CONSTRUCT {
     ?mycart a shop:Cart ;
             shop:contains [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity ?q ;
                             ]  .
}  WHERE {
       ?mycart a shop:Cart ;
               shop:for ?me ;
               shop:ownedBy <http://shop.eg/>.
       GRAPH ?desire {
               ?mycart shop:contains
                            [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity ?q ;
                            ]  .

       }
       ?desire bdi:of ?me .
       ?desire bdi:fulfillby "2007-07-30T..."\^\^xsd:dateTime .
}"""
                  ] .

So there are quite a few things to tie up here, but it seems we have the key elements here:

  • RESTful web services: we use GET and POST the way they are meant to be used,
  • Resource Oriented Architecture: each shoppable item has its resource, that can return a representation
  • the well known motto "hypermedia is the engine of application state": each URL is dereferenceable to further representations. Each page of a containing a buyable item describes how one can proceed to the next step to buy the product. In this case the SPARQL query returns a graph to be POSTed to a given url.
  • with the clarity of the Semantic framework thrown in too. Ie. We can proove certain things about the statements made, which is very helpful in bringing clarity to a vocabulary. Understanding the consequences of what is said is part and parcel of understanding itself.
Have I reached buzzword compliance yet? :-)

Notes

From discussions around the net (on #swig for example) I was made aware of certain problems.

  • SPARQL is a little powerful, and it may seem to give too much leverage to the service, who could ask all kinds of questions of the user agent, such as the SPARQL equivalent of "What is your bank account number". Possible answers may be:
    • Of course a user agent that does shopping automatically on the web, is going to have to be ready for all kinds of misuses, so whatever is done, this type of problem is going to crop up. Servers also need to protect themselves from probing questions by user agents. So this is something that both sides will need to look at.
    • Forms are pretty powerful too. Are forms really so different from queries? They can ask you for a credit card number, your date of birth, the name of your friends, your sexual inclinations, ... What can web formst not ask you?
  • SPARQL is a language that does a couple of things going for it: it has a way of binding variables to a message, and it builds on the solid semantic web structure. But there may be other ways of doing the same thing. OWL-S also uses rdf to describe actions, create a way to bing answers to messages, and descibe the preconditions and postconditions of actions. It even uses the proposed standard of Semantic Web Rules Language SWRL. As there seems to be a strong relation between SPARQL and a rule language (one can think of a SPARQL query as a rule), it may be that part of the interest in this solution is simply the same reason SWRL emerged in OWL-S. OWL-S has a binding to SOAP and none to a RESTful web service. As I have a poor grasp of SOAP I find that difficult to understand. Perhaps a binding to a more restful web service such as the one proposed here would make it more amenable to a wider public.

Wednesday Jun 06, 2007

RESTful Web Services: the book

RESTful Web Services is a newly published book that should be a great help in giving people an overview of how to build web services that work with the architecture of the Web. The authors of the book are I believe serious RESTafarians. They hang out (virtually) on the yahoo REST discuss newsgroup. So I know ahead of time that they will most likely never fail on the REST side of things. Such a book should therefore be a great help for people desiring to develop web services.

As an aside, I am currently reading it online via Safari Books, which is a really useful service, especially for people like me who are always traveling and don't have space to carry wads of paper around the world. As I have been intimately involved in this area for a while - I read Roy Fielding's thesis in 2004, and it immediately made sense of my intuitions - I am skipping through the book from chapter to chapter as my interests guide me, using the search tool when needed. As this is an important book, I will write up my comments here in a number of posts as I work my way through it.

What of course is missing in Roy's thesis, which is a high level abstract description of an architectural style, are practical examples, which is what this book sets out to provide. The advantage of Roy's level of abstraction is that it permitted him to make some very important points without loosing himself in arbitrary implementation debates. Many implementations can fit his architectural style. That is the power of speaking at the right level of abstraction: it permits one to say something well, in such a way that it can withstand the test of time. Developers of course want to see how an abstract theory applies to their everyday work, and so a cook book such as "RESTful Web Services" is going to appeal to them. The danger is that by stepping closer to implementation details, certain choices are made that turn out to be in fact arbitrary, ill conceived, non optimal or incomplete. The risk is well worth taking if it can help people find their way around more easily in a sea of standards. This is where the rubber hits the road.

Right from the beginning the authors, Sam Ruby and Leonard Richardson coin the phrase "Resource Oriented Architecture".

Why come up with a new term, Resource-Oriented Architecture? Why not just say REST? Well, I do say REST, on the cover of this book, and I hold that everything in the Resource-Oriented Architecture is also RESTful. But REST is not an architecture: it's a set of design criteria. You can say that one architecture meets those criteria better than another, but there is no one "REST architecture."

The emphasis on Resources is I agree with them fundamental. Their chapter 4 does a very good job of showing why. URIs name Resources. URLs in particular name Resources that can return representations in well defined ways. REST stands for "Representation of State Transfer", and the representations transferred are the representations of resources identified by URLs. The whole thing fits like a glove.

Except that where there is a glove, there are two, one for each hand. And they are missing the other glove, so to speak. And the lack is glaringly obvious. Just as important as Roy Fielding's work, just as abstract, and developed by some of the best minds on the web, even in the world, is RDF, which stands for Resource Description Framework. I emphasize the "Resource" in RDF because for someone writing a book on Resource Oriented Architecture, to have only three short mentions of the framework for describing resources standardized by non less that the World Wide Web Consortium is just ... flabbergasting. Ignoring this work is like trying to walk around on one leg. It is possible. But it is difficult. And certainly a big waste of energy, time and money. Of course since what they are proposing is so much better than what may have gone on previously, which seems akin to trying to walk around on a gloveless hand, it may not immediately be obvious what is missing. I shall try to make this clear in the series of notes.

Just as REST is very simple, so is RDF. It is easiest to describe something on the web if you have a URL for it. If you want to say something about it, that it relates to something else for example, or that it has a certain property, you need to specify which property it has. Since a property is a thing, it too is easiest to speak about if it has a URL. So once you have identified the property in the global namespace you want to say what its value is, you need to specify what the value of that property is, which can be a string or another object. That's RDF for you. It's so simple I am able to explain it to people in bars within a minute. Here is an example, which says that my name is Henry:

<http://bblfish.net/people/henry/card#me> <http://xmlns.com/foaf/0.1/name> "Henry Story" .

Click on the URLs and you will GET their meaning. Since resources can return any number of representations, different user agents can get the representation they prefer. For the name relation you will get an html representation back if you are requesting it from a browser. With this system you can describe the world. We know this since it is simply a generalization of the system found in relational databases, where instead of identifying things with table dependent primary keys, we identify them with URIs.

So RDF, just as REST, is at its base very easy to understand and furthermore the two are complementary. Even though REST is simple, it nevertheless needs a book such as "RESTful web services" to help make it practical. There are many dispersed standards out there which this books helps bring together. It would have been a great book if it had not missed out the other half of the equation. Luckily this should be easy to fix. And I will do so in the following notes, showing how RDF can help you become even more efficient in establishing your web services. Can it really be even easier? Yes. And furthermore without contradicting what this book says.

About

bblfish

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today