Wednesday Aug 22, 2007

SPARQLing AltaVista: the meaning of forms

Did you know that AltaVista has a SPARQL endpoint? And that all of its results are served up that way? No? Well take the Red Pill and I will show you how deep the rabbit hole goes...

Take the query for the three words "matrix rabbit hole". Go to AltaVista and enter those words into the search box. Press "Find" and you will end up at the result page http://www.altavista.com/web/results?itag=ody&q=rabbit+hole+matrix&kgs=1&kls=0. This page lists the following information for each result:

  • The title of the page
  • The link to the page
  • An extract of the page containing the relevant words
  • A link to more results from that particular web site
In other words it is just the result of the following SPARQL query [1]:
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PRFIX eg: <http://altavista.eg/ont>
PREFIX pf: <http://jena.hpl.hp.com/ARQ/property#>
CONSTRUCT {
     ?page dc:title ?title;
           eg:summary ?summary;
           eg:moreResults ?more .
} WHERE {
     ?page dc:title ?title;
           eg:content ?content;
           eg:summary ?summary;
           eg:moreResults ?more .
     ?content pf:textMatch "+matrix +rabbit +hole" .
}
LIMIT 10
OFFSET 0

The AltaVista engineers - and I know them well having worked there for 5 years - of course understand User Interface issues very well, and so they don't return the default XML result format. They pass all their results first through a clever and optimised XSLT transform, that gives you the page that you now see in your browser.

In order to do this, the AltaVista engineers developed - and I can now speak openly about this - a clever mapping between html forms and SPARQL queries. Sadly it is such a long time ago that I worked there now, that my memory is a little dim on the exact manner in which they did this. So please forgive my mistakes. But I am sure we can work this out together.
Html Forms consist essentially of a number of key-value pairs where the end user is asked to provide the values, a form processing agent URL, and an action button if the user answers the question asked of him. Given that, the trick is just to create a simple SPARQL template language, so that one can relate a form processing agent to one or more SPARQL query templates. What does a SPARQL query template look like? Well it is really very similar to a SPARQL query, except that it has ??vars which need to be replaced by values from the form. So the SPARQL template associated with the front page form could be expressed like this:

@prefix fm: <http://altavisat.eg/ont#> .

<http://www.altavista.com/web/results>  a fm:FormHandler;
    fm:template """
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PRFIX eg: <http://altavista.eg/ont>
PREFIX pf: <http://jena.hpl.hp.com/ARQ/property#>
CONSTRUCT {
     ?page dc:title ?title;
           eg:summary ?summary;
           eg:moreResults ?more .
} WHERE {
     ?page dc:title ?title;
           eg:content ?content;
           eg:summary ?summary;
           eg:moreResults ?more .
     ?content pf:textMatch ??(\\" ??q \\")
}
LIMIT ???lmt
OFFSET ??( ??stq \* ???lmt )
""" .

So when AltaVista receives a form submission, the AV front end decode it and replaces all the ??key patterns above with the value of key as passed by the form. It then replaces all the ???keyparam patterns with default values set by the user and available from his session state. Finally the ??( ... ) operations are exectued. This can be in the form of a multiplication or a string concatenation as shown above, or other operations such as dealing with defaults. Having done that you end up with the previous SPARQL query, which is ready to be sent to the back end engines. The back engines have a more powerful language to play with, allowing AltaVista to propose new paying services to large customers such as Yahoo [2].

This has a few advantages. It reduces the size of the forms POSTED to AltaVista: a SPARQL query would take a lot more space, which would end up taking a fraction of a second off the processing time and so make it ever so much more painfully obvious that the redirect they are doing to av.rds.yahoo.com is destroying their performance. It would also give end users more freedom than is needed: It is a good policy decision to reduce the uses of a tool, when one aims to shrink one's market.

This mapping could be extreemly useful in a number of other ways.
For one it would help make it clear to machines what the meaning of a form is. Forms are questions asked to an agent. The meaning of the question is usually obvious to a human end user who speaks the language of the web page that is being shown. But for a machine to do the same, it helps to map the form to a semanticall defined query, which can be reasoned with. In this case the answers given by the human is used to construct a question that is sent to the server. In other cases the form is asking the user for his desired, and using this to construct an action. There is some interesting work in mapping different uses of forms to rdf still, but I think this does bring a key element into play.
Having a machine understandable version of a form means that a robot can start putting his rdf glasses and see things right. All that would be needed would be to link the form handler to an XSLT that could transform the resulting html to the SPARQL result format, and each of the thousands of existing web forms suddenly become transparent to the world of machine agents. [3]
It could also help reduce the work of defining new Protocols. The good part of OpenId Attribute Exchange for example is just a complex specification for a limited SPARQL template, if you put your rdf glasses on.[4]

With time you get to see the real structure of the world. As that happens the questions you start asking become a lot more interesting.

Notes

  1. The pf:textMatch relation is defined by the LARQ, the Jena Lucene-ARQ free text indexing for SPARQL extension, and it makes a lot of sense.
    The namesppace is sadly not dereferenceable. IE. Clicking on http://jena.hpl.hp.com/ARQ/property does not give you the definition of the relation. It would be nice if it did.
    Note also how in SPARQL you can have literals as subjects. This is explained in section 12.1.4 of the current SPARQL query language specification.
    Thanks to Andy Seaborne for the link.
  2. I do know of course that AltaVista is part of Yahoo! now. And by the way all of the above is meant to be taken with a pinch of salt red pills.
  3. This is called screen scraping, and is of course more work for the consumer. It is nicer when the information provider has a strong interest in providing a stable format.
  4. A large part of the spec is a duplication of the work that should be done by HTTP verbs such as GET, PUT, POST and DELETE. Using the Atom Protocol to publish a foaf file would deal with a large part of the spec in a few lines. Well that's what it looks like to me after studying the spec for a few hours only, and so I may have missed something.

Tuesday Jul 03, 2007

Restful semantic web services

Here is my first stab at an outline for what a restful semantic web services would look like.

Let me start with the obvious. Imagine we have an example shopping service, at http://shop.eg/, which sells books. Clearly we would want URLs for every book that we wish to buy, with RDF representations at the given URL. As I find RDF/XML hard to read and write, I'll show the N3 representations. So to take a concrete example, let us imagine our example shopping service selling the book "RESTful Web Services" at the URL http://shop.eg/books/isbn/0596529260 . If we do an HTTP GET on that URL we could receive the following representation:


@prefix : <http://books.eg/ns#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix shop: <http://shopping.eg/ns#> .
@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix currency: <http://bank.eg/currencies#> .


<#theBook> a shop:Book, shop:Product;
   dc:title "Restful Web Services"@en ;
   dc:creator [ a foaf:Person; foaf:name "Sam Ruby"],
             [ a foaf:Person; foaf:name "Leonard Richardson"] ;
   dc:contributor [ a foaf:Person; foaf:name "David Heinemeier Hansson" ];
   dc:publisher <http://www.oreilly.com/>;
   dct:created "08-07-2007T"\^\^xsd:dateTime;
   dc:description """This is the first book that applies the REST design philosophy to real web services. It sets 
 down the best practices you need to  make your design a success, and the techniques you need to turn your 
 design into working code. You can harness the power of the Web for programmable applications: you just 
 have to work with the Web instead of against it. This book shows you how."""@en;
   shop:price "26.39"\^\^currency:dollars;
   dc:subject </category/computing>, </category/computing/web>, </category/computing/web/rest>, 
                   </category/computing/architecture>,</category/computing/architecture/REST>, </category/computing/howto> .


So we can easily imagine a page like this for every product. These pages can be accessible either by browsing the categories pages, querying a SPARQL endpoint, among many other ways. It should be very easy to generate such representations for a web site. All it requires is to build up an ontology of products - which the shop already has available, if only for the purposes of building inventories - and tie these to the database using a tool such as D2RQ, or a combination of JSR311 and @rdf annotations (see so(m)mer).

Now what is missing is a way to let the browser know what it can do with this product. The simplest possible way of doing this would be to create a specialized relation for that web service to POST some information to a Cart resource, describing the item to be added to the cart. Perhaps something like:

<#theBook> shop:addToCart <http://shop.eg/cart/> .

This relation would just mean that one has to POST the url to the cart, to have it added there. The cart itself may then have a shop:buy relation to some resource, which by convention the user agent would need to send a credit card, expiration date, and other information to.
This means that one would have to define a number of RDF relationships, for every type of action one could do in a shop (and later on the web), and explain the types of messages to be sent to the endpoint, and what their consequences are. This is simple but it does not seem very extensible. What if one wants to buy the hard copy version of the book, or 10 items of the book? The hard copy version of course could have its own URL, and so it may be as simple as placing the buy relation on that page. But is this going to work with PCs where one can add and remove a huge number of pieces. I remember Steve Jobs being proud of the huge number of different configurations one could buy his desktop systems with, well over 100 thousand different configurations I remember. This could make it quite difficult to navigate a store, if one is not careful.

On the current web this is dealt with by using html forms, which can allow the user to choose between a large number of variables, by selecting check boxes, combo boxes, drop down menues and more, and then POST a representation to a collection, and thereby create a new action, such as adding the product to the cart, or buying it. The person browsing the site knows what the action does, because it is usually written out in a natural language, in a way that makes it quite obvious to a human being. The person then does that action because he desires to do so, because he wishes his desires to be fulfilled. Now this may seem very simple, but just consider the innumerable types of actions that we can fulfill using the very simple tool of html forms: we can add things to a virtual cart, buy things, comment about things, search for things, organise a meeting, etc, etc.... So forms can be seen both as shortcuts to navigate to a large number of different resources, and to create new resources (usually best done with POST).
If we want software agents to do such tasks for us we need both to have something like a machine understandable form, and some way of specifying what the action of POSTing the form will have on the world. So we need to find a way to do what the web does in a more clearly specified way, so that even machines, or simple user agents can understand it. Let's look at each one:

  • Forms are ways of asking the user to bind results to variables
  • the variables can the be used to build something, such as a URL, or a message.
  • The form then specifies the type of action to do with the constructed message, such as a GET, POST, PUT, etc...
  • The human readable text explains what the result of the action is, and what the meaning of each of the fields are.

Now what semantic technology binds variables to values? Which one asks questions? SPARQL comes immediately to mind. Seeing this and remembering a well known motto of sales people "Satisfy the customer every desire" a very general but conceptually simple solution to this problem occurred to me. It may seem a little weird at first (and perhaps it will continue to seem weird) but I thought it is elegant enough to be used as a starting point. The idea is really simple: the representation returned by the book resource will specify a collection end point to POST RDF too, and it will specify what to POST back by sending a SPARQL query in the representation. It will then be up to the software agent reading the representation to answer the query if he wishes a certain type of action to occur. If he understand the query he will be able to answer, if he does not, there should be no results. He need not do anything with the query at all.

The following is the first thing that occurred to me. The details are less important than the principle of thinking of forms as asking the client a question.

PREFIX shop: <http://shopping.eg/ns#>
PREFIX bdi: <http://intentionality.eg/ns#>
CONSTRUCT {
     ?mycart a shop:Cart ;
             shop:contains [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity ?q ;
                             ]  .
}  WHERE {
       ?mycart a shop:Cart ;
               shop:for ?me ;
               shop:ownedBy <http://shop.eg/>.
       GRAPH ?desire {
               ?mycart shop:contains
                            [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity ?q ;
                            ]  .

       }
       ?desire bdi:of ?me .
       ?desire bdi:fulfillby "2007-07-30T..."\^\^xsd:dateTime .
}

So this is saying quite simply: Find out if you want to have your shopping cart filled up with a number of this book. The user agent (the equivalent of the web browser) asks its data store the given SPARQL query. It asks itself whether it desires to add a number of books to its shopping cart, and if it wishes that desire to be fulfulled by a certain time. If the agent does not understand the relations in the query, then the CONSTRUCT clause will return an empty graph. If it does understand it, and the query returns a result, then it is because it wished the action to take place. The constructed graph may be something like:

@prefix shop: <http://shopping.eg/ns#>
 <http://shop.eg/cart/bblfish/> a shop:Cart ;
             shop:contains [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity 2 ;
                             ]  .

This can then be POSTed to the collection end point http://shop.eg/cart/, with the result of adding two instances of the book to the cart. Presumably the cart would return a graph with the above relations in it plus another SPARQL query explaining how to buy the items in the cart.

So the full RDF for the book page would look something like this:

@prefix : <http://books.eg/ns#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix shop: <http://shopping.eg/ns#> .
@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix currency: <http://bank.eg/currencies#> .


<#theBook> a shop:Book, shop:Product;
   dc:title "Restful Web Services"@en ;
   dc:creator [ a foaf:Person; foaf:name "Sam Ruby"],
             [ a foaf:Person; foaf:name "Leonard Richardson"] ;
   dc:contributor [ a foaf:Person; foaf:name "David Heinemeier Hansson" ];
   dc:publisher <http://www.oreilly.com/>;
   dct:created "08-07-2007T"\^\^xsd:dateTime;
   dc:description """This is the first book that applies the REST design philosophy to real web services. It sets 
 down the best practices you need to make your design a success, and the techniques you need to turn your
  design into working code. You can harness the power of the Web for  programmable applications: you just 
 have to work with the Web instead of against it. This book shows you how."""@en;
   shop:price "26.39"\^\^currency:dollars;
   dc:subject </category/computing>, </category/computing/web>, </category/computing/web/rest>, 
                      </category/computing/architecture>,</category/computing/architecture/REST>,</category/computing/howto>;
   shop:addToCart [ a Post;
                    shop:collection <http://shop.eg/cart/>;
                    shop:query """
PREFIX shop: <http://shopping.eg/ns#>
PREFIX bdi: <http://intentionality.eg/ns#>
CONSTRUCT {
     ?mycart a shop:Cart ;
             shop:contains [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity ?q ;
                             ]  .
}  WHERE {
       ?mycart a shop:Cart ;
               shop:for ?me ;
               shop:ownedBy <http://shop.eg/>.
       GRAPH ?desire {
               ?mycart shop:contains
                            [ a shop:LineItem;
                               shop:SKU <http://shop.eg/books/isbn/0596529260#theBook> ;
                               shop:quantity ?q ;
                            ]  .

       }
       ?desire bdi:of ?me .
       ?desire bdi:fulfillby "2007-07-30T..."\^\^xsd:dateTime .
}"""
                  ] .

So there are quite a few things to tie up here, but it seems we have the key elements here:

  • RESTful web services: we use GET and POST the way they are meant to be used,
  • Resource Oriented Architecture: each shoppable item has its resource, that can return a representation
  • the well known motto "hypermedia is the engine of application state": each URL is dereferenceable to further representations. Each page of a containing a buyable item describes how one can proceed to the next step to buy the product. In this case the SPARQL query returns a graph to be POSTed to a given url.
  • with the clarity of the Semantic framework thrown in too. Ie. We can proove certain things about the statements made, which is very helpful in bringing clarity to a vocabulary. Understanding the consequences of what is said is part and parcel of understanding itself.
Have I reached buzzword compliance yet? :-)

Notes

From discussions around the net (on #swig for example) I was made aware of certain problems.

  • SPARQL is a little powerful, and it may seem to give too much leverage to the service, who could ask all kinds of questions of the user agent, such as the SPARQL equivalent of "What is your bank account number". Possible answers may be:
    • Of course a user agent that does shopping automatically on the web, is going to have to be ready for all kinds of misuses, so whatever is done, this type of problem is going to crop up. Servers also need to protect themselves from probing questions by user agents. So this is something that both sides will need to look at.
    • Forms are pretty powerful too. Are forms really so different from queries? They can ask you for a credit card number, your date of birth, the name of your friends, your sexual inclinations, ... What can web formst not ask you?
  • SPARQL is a language that does a couple of things going for it: it has a way of binding variables to a message, and it builds on the solid semantic web structure. But there may be other ways of doing the same thing. OWL-S also uses rdf to describe actions, create a way to bing answers to messages, and descibe the preconditions and postconditions of actions. It even uses the proposed standard of Semantic Web Rules Language SWRL. As there seems to be a strong relation between SPARQL and a rule language (one can think of a SPARQL query as a rule), it may be that part of the interest in this solution is simply the same reason SWRL emerged in OWL-S. OWL-S has a binding to SOAP and none to a RESTful web service. As I have a poor grasp of SOAP I find that difficult to understand. Perhaps a binding to a more restful web service such as the one proposed here would make it more amenable to a wider public.

Wednesday Jun 06, 2007

RESTful Web Services: the book

RESTful Web Services is a newly published book that should be a great help in giving people an overview of how to build web services that work with the architecture of the Web. The authors of the book are I believe serious RESTafarians. They hang out (virtually) on the yahoo REST discuss newsgroup. So I know ahead of time that they will most likely never fail on the REST side of things. Such a book should therefore be a great help for people desiring to develop web services.

As an aside, I am currently reading it online via Safari Books, which is a really useful service, especially for people like me who are always traveling and don't have space to carry wads of paper around the world. As I have been intimately involved in this area for a while - I read Roy Fielding's thesis in 2004, and it immediately made sense of my intuitions - I am skipping through the book from chapter to chapter as my interests guide me, using the search tool when needed. As this is an important book, I will write up my comments here in a number of posts as I work my way through it.

What of course is missing in Roy's thesis, which is a high level abstract description of an architectural style, are practical examples, which is what this book sets out to provide. The advantage of Roy's level of abstraction is that it permitted him to make some very important points without loosing himself in arbitrary implementation debates. Many implementations can fit his architectural style. That is the power of speaking at the right level of abstraction: it permits one to say something well, in such a way that it can withstand the test of time. Developers of course want to see how an abstract theory applies to their everyday work, and so a cook book such as "RESTful Web Services" is going to appeal to them. The danger is that by stepping closer to implementation details, certain choices are made that turn out to be in fact arbitrary, ill conceived, non optimal or incomplete. The risk is well worth taking if it can help people find their way around more easily in a sea of standards. This is where the rubber hits the road.

Right from the beginning the authors, Sam Ruby and Leonard Richardson coin the phrase "Resource Oriented Architecture".

Why come up with a new term, Resource-Oriented Architecture? Why not just say REST? Well, I do say REST, on the cover of this book, and I hold that everything in the Resource-Oriented Architecture is also RESTful. But REST is not an architecture: it's a set of design criteria. You can say that one architecture meets those criteria better than another, but there is no one "REST architecture."

The emphasis on Resources is I agree with them fundamental. Their chapter 4 does a very good job of showing why. URIs name Resources. URLs in particular name Resources that can return representations in well defined ways. REST stands for "Representation of State Transfer", and the representations transferred are the representations of resources identified by URLs. The whole thing fits like a glove.

Except that where there is a glove, there are two, one for each hand. And they are missing the other glove, so to speak. And the lack is glaringly obvious. Just as important as Roy Fielding's work, just as abstract, and developed by some of the best minds on the web, even in the world, is RDF, which stands for Resource Description Framework. I emphasize the "Resource" in RDF because for someone writing a book on Resource Oriented Architecture, to have only three short mentions of the framework for describing resources standardized by non less that the World Wide Web Consortium is just ... flabbergasting. Ignoring this work is like trying to walk around on one leg. It is possible. But it is difficult. And certainly a big waste of energy, time and money. Of course since what they are proposing is so much better than what may have gone on previously, which seems akin to trying to walk around on a gloveless hand, it may not immediately be obvious what is missing. I shall try to make this clear in the series of notes.

Just as REST is very simple, so is RDF. It is easiest to describe something on the web if you have a URL for it. If you want to say something about it, that it relates to something else for example, or that it has a certain property, you need to specify which property it has. Since a property is a thing, it too is easiest to speak about if it has a URL. So once you have identified the property in the global namespace you want to say what its value is, you need to specify what the value of that property is, which can be a string or another object. That's RDF for you. It's so simple I am able to explain it to people in bars within a minute. Here is an example, which says that my name is Henry:

<http://bblfish.net/people/henry/card#me> <http://xmlns.com/foaf/0.1/name> "Henry Story" .

Click on the URLs and you will GET their meaning. Since resources can return any number of representations, different user agents can get the representation they prefer. For the name relation you will get an html representation back if you are requesting it from a browser. With this system you can describe the world. We know this since it is simply a generalization of the system found in relational databases, where instead of identifying things with table dependent primary keys, we identify them with URIs.

So RDF, just as REST, is at its base very easy to understand and furthermore the two are complementary. Even though REST is simple, it nevertheless needs a book such as "RESTful web services" to help make it practical. There are many dispersed standards out there which this books helps bring together. It would have been a great book if it had not missed out the other half of the equation. Luckily this should be easy to fix. And I will do so in the following notes, showing how RDF can help you become even more efficient in establishing your web services. Can it really be even easier? Yes. And furthermore without contradicting what this book says.

About

bblfish

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today