Spam Silver Lining

When I'm waiting for that really important email, and nothing appears in my inbox for hours, and I suspect Norton AV has again croaked and is blocking, I check my spam folder for recent messages, to verify that the mail is still getting through. It's a constant, incessant ping!

SPARQL Last Call

W3C have issued the last call for SPARQL comments. They feel compelled to tell us that you pronounce SPARQL "sparkle". It is not too late to request a change in pronunciation to "sparquel" before it goes final.
I just don't think I can convert to "sparkle" now. Nevertheless, I used to think I could never convert to "lih-nix" from "lye-nix", but I've been overwhelmed by society (and by Linus).

Reliable POST

A couple of proposals for lightweight reliable POST are circulating: Mark Nottingham's Post Once Exactly (POE) and Paul Prescod's Reliable Delivery in HTTP. Similar in spirit, both techniques propose that servers generate one-off URLs for clients to POST to. Generally the pattern is

-> GET url
<- entity and/or header containing one-off URL
-> POST one-off-url

The one-off URL behaves specially: it only changes application state the first time you POST to it.


The two proposals differ in how the server responds to multiple POSTs. Paul proposes the server simply return the same response it returned when it processed the first request

The response of subsequent POSTs should be the same as if there had been only one POST so that the client can get the correct response even if there is a network outage in the middle of the first response.

Mark proposes that under POE the server return 405 Method Not Allowed on the second and subsequent POSTs to a POE resource:

If the server had received and accepted the first request, it will respond with
S: 405 Method Not Allowed HTTP/1.1
Allow: GET
...
If the response status is "405 Method Not Allowed" the client can infer that the earlier POST succeeded. A 2xx response indicates that earlier POST did not succeed, but that this one has. When the client receives either of these responses, it knows that the request has been accepted, and it can stop retrying.

Under Paul's proposal if the first POST failed (e.g. 401 Unauthorized), then even if the user corrected the problem, POSTing the corrected form to the resource would still return the same error status. POE's approach really permits exactly one POST to succeed. POE does seem to impose some new semantic over HTTP: If a client receives a 405 it can stop retrying. In practice clients, naive ones not underststanding POE, would never retry after receiving a 405 anyway. It is a new feature on the HTTP landscape that a resource could return success or various failure status codes, until at some point it changes state and complains that POST is not allowed. Nothing wrong with that, and it won't break any clients. But it has not been common to see that behavior.

What about those special POE headers? The proposal acknowledges they're unneccessary -- why use them? If your web site has a form page that POSTs to a POE resource, you can now put text next to the submit button saying "Press repeatedly!" The problem there is that if your browser displays a 405 error, you would not understand that your POST had succeeded -- unless the server also returned a comforting HTML entity telling you that.

It's a minor problem that POE overloads the semantics of 405, becuase it's not a failure really. If you are a POE-aware agent, then the special POE headers tell you to interpret the 405 slightly differently: The operation really did succeed! But it succeeded before this latest POST. I'd prefer that the superfluous operation return a success code.

Maybe we need a synthesis of the two approaches. Paul's instincts were right: A second POST to a resource that had earlier successfully processed a POST, should return the same 2XX or 3XX code, and the same entity, as the first one. After all, does a client really need to understand that this second POST was superfluous? If so, POE proposes that a GET to the POE resource return the created entity, if any, and we should retain that behavior. This way an HTML page could have a simple hyperlink to the POE URL, that would return that entity indicating "transaction succeeded". POE does not say what the server should return in response to a GET if the POST has not yet been processed. A 404 Not found would not be helpful. I suppose it has to be a 200 OK with an explanatory entity: an HTML page saying "Still waiting..." or some such.

If the initial POST to the resource fails with 4XX or 5XX, the server ought to continue to accept POST attempts until finally one of them reults in a 2XX or 3XX success. The semantic we want is that POST succeed exactly once.


[Security concern about one-off URLs: servers must prevent malefactors from predicting one-off URLs and hijacking them. It's probably good enough to generate very long random numbers as part of the URL.]

Tagging is easy. Semweb is hard.

Clay Shirky's Ontology is Overrated post is a great survey of why tagging is hot, semweb is not.

But there is a bit of strawman in the argument. Clay sets up ontologies to be hierarchcial organizations of concepts having firm inside/outside boundaries, and easily shows how inadequate that scheme is to describe the web.

But the semweb people actually designed in many of the features Clay likes about tagging. Anyone can make a new OWL ontology describing resources in an idiosyncratic way. OWL concepts overlap: a resource can belong to thousands of concepts. And most definitely, the link topologies are not restricted to hierarchies: an RDF graph looks just like the web.

Tagging is not taking off becuase it describes the web better than semweb. Tagging is taking off because it's easy, and semweb is hard. del.icio.us wouldn't have gotten very far if you had to define your own classes and properties.

But everything you do in del.icio.us, could be done using semweb. A tag is an OWL class with a name and that's about it: No other properties, but the members of the class (your tagged URIs) imply something about the class. You could take each user's tags and create his own personal ontology. No need to adopt anyone else's ontology. But to expand on Clay's "mind reading" analogy: If I determine that another user says "movies" to mean the same thing I do when I say "cinema", I could make that mapping through an OWL equivalence. Then all that user's "movies" tags become trusted indicators for movies in my searches.

Clay is on the money with the observation that the real meanings of the terms are emerging from the statistical fog. You can make observations about the relations between tags based on the URLs that share them. "This URL probably represents a movie, as you think of 'movie'" is the function I need.

Service specific operations and machines

Service independent operations are valuable when the agent invoking them is the kind of agent that talks to lots of different services: a web browser + a human to make sense of what he browses and make choices accordingly.

If the agent is service specific, then heck, just design in all the specific operations you want.

Machine to machine conversations are almost always service specific. You have to program the client to understand how to proceed through the legal application states. The guy programming the client needs to know... First you do this to get this result, then you use that result to make a second query, and so on. It's no help to have a service independent operation set if you're using it that way.

Service independent operations would be valuable in the machine to machine case if you could invent a surrogate for the human: an intelligent machine agent able to make choices, given some output from the last operation. I've said that before, I know. Just thought it would be useful to state it a little differently.

(Caveat: written after a 16 hour day constructing SOAP services for machine to machine cases).

WSDL 2.0

Dave Orchard's remarks on WSDL 2 encourage me to replace my own cooked up SDL with WSDL 2. Sounds a little daunting though: "WSD WG decided that the specs were for toolkit authors not wsdl document authors."

I will report experimental results here.

RESTful Web Service Descriptions

I've fleshed out some of the details I left dangling in my previous post about WITSML. The WITSML stuff is so specialized, I've created a separate blog for it.


But if you are interested in the web description language (web-http-desc) discussion there is some meat here for you as well. This draft documents RESTful access to WITSML services, and writing it was a great exercise for the description discussion. (I didn't write it as an exercise; our product implements some of that protocol right now; but I had never written it down in one place).

WITSML is "Well Information Transfer Standard" markup language. It defines not only document formats, but a SOAP API that has the following operations:

AddToStore
GetFromStore
UpdateInStore
DeleteFromStore

Look familiar? Well GetFromStore is a query, and has a query string parameter, but generally, you can do 90% of WITSML just by GET, PUT, and DELETE on objects, and the other 10% are vanilla things you could do with POST.

If you are interested in the web description language discussion, I invite you to have a look at the WITSML+REST draft and comment here or on the list.

Messages not Models

Visiting the WITSML SIG meeting and expo last week, sponsored by POSC, recalled the evolution oil and gas E&P IT systems are undergoing. Unlike lots of other industries, the E&P guys have tried in the past to normalize on an industry logical data model. One standards effort was promulgated by POSC, the Epicentre data model -- it is enormous, beautiful, and unused. PPDM is a separate standards effort. Landmark and Schlumberger each offer proprietary models OpenWorks and GeoFrame. Yet each company using one of these models populates the database assigning slightly different semantics to properties, and many have tweaked the models to suit their own practices. So interoperability has suffered.

Enter WITSML. Oh, it doesn't (yet) attempt to address the full range of E&P information -- just a small segment concerned with drilling operations. But it is the very first message exchange system for this industry -- not just an interchange file format, but a message format together with a protocol.

Now that we agree on the messages, who cares about the models? You keep your model, I'll keep mine, and we agree we'll exchange resource representations, not models. It's a great use case for REST.

An E&P application can sit on top of whatever model it prefers and expose its services to the world using the REST prescription:

1. Identification of resources: assign a URI to resources like well, wellbore, etc.

2. Self-describing messages: Accept and deliver XML descriptions of well, wellbore, etc. The representations do have to map to components of the underlying logical data model.

3. Hypertext as the engine of application state: This is the key. In the past, the trouble has been caused by applications maintaining state in the database. A client has to understand the model and has to keep up with the evolution of that model. Using hypertext as the engine of application state, the server in effect programs the client to proceed through the states of the application correctly. As I've pointed out elsewhere, whether the client is a human operating a browser, a human operating a rich client, or an automaton exercising a web service, the server delivering instructions (a "form") to the client effectively is programming that client: "Here are the next steps you can take from here: a) do a GET on this URL or b) serialize these parameters and do a POST to this URL".

So WITSML could be the beginnings of a pattern for extracting the industry from slavery to models.

Oh, except for that silly SOAP API they've crufted together. What I've outlined here is only the possibility enabled by having defined the messages. But because of the RPC style of the WITSML SOAP exchanges, resources aren't identified by URIs, and hypertext most definitely is not the engine of application state. These problems are solvable. More on that later.

Namespace URIs, or not

Dave Orchard defends using the HTTP scheme for XML namespsaces. While Michael Day claims XML Namespaces Don't Need URIs.

Origins of the WS stack

Mark is right on the money:


Web services were created because it was felt that Web architecture wasn't sufficient to integrate disparate applications together over the Internet. Actually, that's not quite right. The explanation that seems to better reflect reality is that the Web was never considered as a platform suitable for meeting the objectives of Web services, as can be demonstrated by the numerous articles talking about how Web services evolved from the likes of CORBA, DCOM, RMI, etc.., without mentioning the Web!! The Web just didn't resemble what folks knew a distributed computing solution to look like, so it just never registered in the heads to consider it.

The guys who built SOAP, meaning Don Box principally, were DCOM guys solving problems in distributed objects. Mark's post prompted me to revisit the old DCOM list. Here's a great post from Don in 1998 reasoning that XML's "self-describing" nature would address the type problems DCOM had. Ah, yes. ITypeInfo was an interface you could expose on an object describing its methods and properties. I think it was hard to use it sensibly for multiple interfaces on an object. Since it was parameterized by a type library description that was installed on the client computer, I guess it was also fragile if you changed the interfaces of the server object. That all seems... so... far... away now.

A parallel trend was to overload port 80 and tunnel DCOM through it, since port 80 was perceived to be "open" everywhere. That was a nifty trick that wasn't very useful, since you had to persuade your firewall to permit garbage to come over port 80 (by tricking it into believing the garbage was SSL).

All we really wanted was DCOM over the Internet.

Those two trends, self describing RPC payloads and leveraging the web, came together as SOAP.