Monday, August 29, 2005
World's most useful blog
It's sobering to realize how black is the information hole around New Orleans right now. Hurricane Katrina. Cell towers down and powerless. Land lines inoperative. We have isolated little spotlights from CNN and Fox News, but they sent their teams to some downtown hotels, same as any corporate traveler, so these reports have focused on downtown. It's a lot like reporting from Baghdad hotels about the invasion of Iraq. N.O. is huge and dense. So check out the Times-Picayune's blog for the real dope. They deserve a Pulitzer for getting the information out to those of us really needing it. They're assembling reports there from all over the city. I'm a New Orleans ex-patriate, and right now I'm awaiting here in Austin the arrival of some family refugees who made it out, and are headed to live with us for... who knows how long. The power's going to be off for a month, and you can't get a drink of water. On that blog, and nowhere else really, I've been able to get some glimmer of info about my relatives' and friends' neighborhoods.
Friday, August 19, 2005
I Need a New Language: Rel?
I'm working through a Ruby experiment, sans Rails, and it's going well. I've also lately done a little Python CGI, and I've been burning bucks at Amazon buying the Lisp and Ruby books.
But, ho hum. Oh yeah, dynamic languages are great, you can modify classes at runtime, yeah, yeah. Blah blah closures blah blah.
People, we need to get past the object oriented paradigm, or model, or whatever it is. I'm groping for the one true programming model. Short of that, I'm hoping for one where I never have to do the stupid object-relational mapping again. Hibernate's popularity is emblematic of our decline. If you've got a ton of O/R mapping in your program, you've got a ton of dead code that does nothing for you. Every web app on the planet does this:
It's a failure of the languages we use, that we need application code to map the tables in which we store data, to some in-memory constructs called objects. And it's a failure of the architectures we use, that we end up flattening those objects into HTML representations.
I want a language for table programming. I think you can write programs in this language that do everything we expect of an application programming language -- building GUIs, reacting to mouse events, listening to sockets -- everything. Don't model your domain as objects. Model it as relations.
We need to explore how we can construct soup to nuts programs using the language of data. I haven't yet looked at Rel. But I wanted it to exist -- I wanted some implementation of Tutorial D to exist. Using that as a starting point, I can imagine programs that have a relvar (Date's term for a relational variable: essentially a table or a view) for MouseState. Whenever the mouse moves, the system adds a new row to MouseState. Yes, a lot of rows! We don't really need to persist them all. When a row gets added to a table or a view the system should invoke some code -- a trigger. The trigger code modifies other relvars in the system. And from that trigger code other events flow.
A programming language built around locating the right bit of code to invoke in response to some state transition -- that would be a useful evolution of OO polymorphism, which locates the right code to invoke based only on the shape of an object. Let's get off of this rutted OO cartpath. We're not inventing anything new here.
But, ho hum. Oh yeah, dynamic languages are great, you can modify classes at runtime, yeah, yeah. Blah blah closures blah blah.
People, we need to get past the object oriented paradigm, or model, or whatever it is. I'm groping for the one true programming model. Short of that, I'm hoping for one where I never have to do the stupid object-relational mapping again. Hibernate's popularity is emblematic of our decline. If you've got a ton of O/R mapping in your program, you've got a ton of dead code that does nothing for you. Every web app on the planet does this:
- Make some queries to the db;
- Build some objects possibly having object references;
- Iterate over the objects and flatten to tabular HTML.
It's a failure of the languages we use, that we need application code to map the tables in which we store data, to some in-memory constructs called objects. And it's a failure of the architectures we use, that we end up flattening those objects into HTML representations.
I want a language for table programming. I think you can write programs in this language that do everything we expect of an application programming language -- building GUIs, reacting to mouse events, listening to sockets -- everything. Don't model your domain as objects. Model it as relations.
We need to explore how we can construct soup to nuts programs using the language of data. I haven't yet looked at Rel. But I wanted it to exist -- I wanted some implementation of Tutorial D to exist. Using that as a starting point, I can imagine programs that have a relvar (Date's term for a relational variable: essentially a table or a view) for MouseState. Whenever the mouse moves, the system adds a new row to MouseState. Yes, a lot of rows! We don't really need to persist them all. When a row gets added to a table or a view the system should invoke some code -- a trigger. The trigger code modifies other relvars in the system. And from that trigger code other events flow.
A programming language built around locating the right bit of code to invoke in response to some state transition -- that would be a useful evolution of OO polymorphism, which locates the right code to invoke based only on the shape of an object. Let's get off of this rutted OO cartpath. We're not inventing anything new here.
Thursday, August 04, 2005
Spam Silver Lining
When I'm waiting for that really important email, and nothing appears in my inbox for hours, and I suspect Norton AV has again croaked and is blocking, I check my spam folder for recent messages, to verify that the mail is still getting through. It's a constant, incessant ping!
Friday, July 29, 2005
SPARQL Last Call
W3C have issued the last call for SPARQL comments. They feel compelled to tell us that you pronounce SPARQL "sparkle". It is not too late to request a change in pronunciation to "sparquel" before it goes final.
I just don't think I can convert to "sparkle" now. Nevertheless, I used to think I could never convert to "lih-nix" from "lye-nix", but I've been overwhelmed by society (and by Linus).
I just don't think I can convert to "sparkle" now. Nevertheless, I used to think I could never convert to "lih-nix" from "lye-nix", but I've been overwhelmed by society (and by Linus).
Saturday, July 09, 2005
Reliable POST
A couple of proposals for lightweight reliable POST are circulating: Mark Nottingham's Post Once Exactly (POE) and Paul Prescod's Reliable Delivery in HTTP. Similar in spirit, both techniques propose that servers generate one-off URLs for clients to POST to. Generally the pattern is
-> GET url
<- entity and/or header containing one-off URL
-> POST one-off-url
The one-off URL behaves specially: it only changes application state the first time you POST to it.
The two proposals differ in how the server responds to multiple POSTs. Paul proposes the server simply return the same response it returned when it processed the first request
Mark proposes that under POE the server return 405 Method Not Allowed on the second and subsequent POSTs to a POE resource:
Under Paul's proposal if the first POST failed (e.g. 401 Unauthorized), then even if the user corrected the problem, POSTing the corrected form to the resource would still return the same error status. POE's approach really permits exactly one POST to succeed. POE does seem to impose some new semantic over HTTP: If a client receives a 405 it can stop retrying. In practice clients, naive ones not underststanding POE, would never retry after receiving a 405 anyway. It is a new feature on the HTTP landscape that a resource could return success or various failure status codes, until at some point it changes state and complains that POST is not allowed. Nothing wrong with that, and it won't break any clients. But it has not been common to see that behavior.
What about those special POE headers? The proposal acknowledges they're unneccessary -- why use them? If your web site has a form page that POSTs to a POE resource, you can now put text next to the submit button saying "Press repeatedly!" The problem there is that if your browser displays a 405 error, you would not understand that your POST had succeeded -- unless the server also returned a comforting HTML entity telling you that.
It's a minor problem that POE overloads the semantics of 405, becuase it's not a failure really. If you are a POE-aware agent, then the special POE headers tell you to interpret the 405 slightly differently: The operation really did succeed! But it succeeded before this latest POST. I'd prefer that the superfluous operation return a success code.
Maybe we need a synthesis of the two approaches. Paul's instincts were right: A second POST to a resource that had earlier successfully processed a POST, should return the same 2XX or 3XX code, and the same entity, as the first one. After all, does a client really need to understand that this second POST was superfluous? If so, POE proposes that a GET to the POE resource return the created entity, if any, and we should retain that behavior. This way an HTML page could have a simple hyperlink to the POE URL, that would return that entity indicating "transaction succeeded". POE does not say what the server should return in response to a GET if the POST has not yet been processed. A 404 Not found would not be helpful. I suppose it has to be a 200 OK with an explanatory entity: an HTML page saying "Still waiting..." or some such.
If the initial POST to the resource fails with 4XX or 5XX, the server ought to continue to accept POST attempts until finally one of them reults in a 2XX or 3XX success. The semantic we want is that POST succeed exactly once.
[Security concern about one-off URLs: servers must prevent malefactors from predicting one-off URLs and hijacking them. It's probably good enough to generate very long random numbers as part of the URL.]
-> GET url
<- entity and/or header containing one-off URL
-> POST one-off-url
The one-off URL behaves specially: it only changes application state the first time you POST to it.
The two proposals differ in how the server responds to multiple POSTs. Paul proposes the server simply return the same response it returned when it processed the first request
The response of subsequent POSTs should be the same as if there had been only one POST so that the client can get the correct response even if there is a network outage in the middle of the first response.
Mark proposes that under POE the server return 405 Method Not Allowed on the second and subsequent POSTs to a POE resource:
If the server had received and accepted the first request, it will respond with
S: 405 Method Not Allowed HTTP/1.1
Allow: GET
...
If the response status is "405 Method Not Allowed" the client can infer that the earlier POST succeeded. A 2xx response indicates that earlier POST did not succeed, but that this one has. When the client receives either of these responses, it knows that the request has been accepted, and it can stop retrying.
Under Paul's proposal if the first POST failed (e.g. 401 Unauthorized), then even if the user corrected the problem, POSTing the corrected form to the resource would still return the same error status. POE's approach really permits exactly one POST to succeed. POE does seem to impose some new semantic over HTTP: If a client receives a 405 it can stop retrying. In practice clients, naive ones not underststanding POE, would never retry after receiving a 405 anyway. It is a new feature on the HTTP landscape that a resource could return success or various failure status codes, until at some point it changes state and complains that POST is not allowed. Nothing wrong with that, and it won't break any clients. But it has not been common to see that behavior.
What about those special POE headers? The proposal acknowledges they're unneccessary -- why use them? If your web site has a form page that POSTs to a POE resource, you can now put text next to the submit button saying "Press repeatedly!" The problem there is that if your browser displays a 405 error, you would not understand that your POST had succeeded -- unless the server also returned a comforting HTML entity telling you that.
It's a minor problem that POE overloads the semantics of 405, becuase it's not a failure really. If you are a POE-aware agent, then the special POE headers tell you to interpret the 405 slightly differently: The operation really did succeed! But it succeeded before this latest POST. I'd prefer that the superfluous operation return a success code.
Maybe we need a synthesis of the two approaches. Paul's instincts were right: A second POST to a resource that had earlier successfully processed a POST, should return the same 2XX or 3XX code, and the same entity, as the first one. After all, does a client really need to understand that this second POST was superfluous? If so, POE proposes that a GET to the POE resource return the created entity, if any, and we should retain that behavior. This way an HTML page could have a simple hyperlink to the POE URL, that would return that entity indicating "transaction succeeded". POE does not say what the server should return in response to a GET if the POST has not yet been processed. A 404 Not found would not be helpful. I suppose it has to be a 200 OK with an explanatory entity: an HTML page saying "Still waiting..." or some such.
If the initial POST to the resource fails with 4XX or 5XX, the server ought to continue to accept POST attempts until finally one of them reults in a 2XX or 3XX success. The semantic we want is that POST succeed exactly once.
[Security concern about one-off URLs: servers must prevent malefactors from predicting one-off URLs and hijacking them. It's probably good enough to generate very long random numbers as part of the URL.]
Tuesday, June 28, 2005
Tagging is easy. Semweb is hard.
Clay Shirky's Ontology is Overrated post is a great survey of why tagging is hot, semweb is not.
But there is a bit of strawman in the argument. Clay sets up ontologies to be hierarchcial organizations of concepts having firm inside/outside boundaries, and easily shows how inadequate that scheme is to describe the web.
But the semweb people actually designed in many of the features Clay likes about tagging. Anyone can make a new OWL ontology describing resources in an idiosyncratic way. OWL concepts overlap: a resource can belong to thousands of concepts. And most definitely, the link topologies are not restricted to hierarchies: an RDF graph looks just like the web.
Tagging is not taking off becuase it describes the web better than semweb. Tagging is taking off because it's easy, and semweb is hard. del.icio.us wouldn't have gotten very far if you had to define your own classes and properties.
But everything you do in del.icio.us, could be done using semweb. A tag is an OWL class with a name and that's about it: No other properties, but the members of the class (your tagged URIs) imply something about the class. You could take each user's tags and create his own personal ontology. No need to adopt anyone else's ontology. But to expand on Clay's "mind reading" analogy: If I determine that another user says "movies" to mean the same thing I do when I say "cinema", I could make that mapping through an OWL equivalence. Then all that user's "movies" tags become trusted indicators for movies in my searches.
Clay is on the money with the observation that the real meanings of the terms are emerging from the statistical fog. You can make observations about the relations between tags based on the URLs that share them. "This URL probably represents a movie, as you think of 'movie'" is the function I need.
But there is a bit of strawman in the argument. Clay sets up ontologies to be hierarchcial organizations of concepts having firm inside/outside boundaries, and easily shows how inadequate that scheme is to describe the web.
But the semweb people actually designed in many of the features Clay likes about tagging. Anyone can make a new OWL ontology describing resources in an idiosyncratic way. OWL concepts overlap: a resource can belong to thousands of concepts. And most definitely, the link topologies are not restricted to hierarchies: an RDF graph looks just like the web.
Tagging is not taking off becuase it describes the web better than semweb. Tagging is taking off because it's easy, and semweb is hard. del.icio.us wouldn't have gotten very far if you had to define your own classes and properties.
But everything you do in del.icio.us, could be done using semweb. A tag is an OWL class with a name and that's about it: No other properties, but the members of the class (your tagged URIs) imply something about the class. You could take each user's tags and create his own personal ontology. No need to adopt anyone else's ontology. But to expand on Clay's "mind reading" analogy: If I determine that another user says "movies" to mean the same thing I do when I say "cinema", I could make that mapping through an OWL equivalence. Then all that user's "movies" tags become trusted indicators for movies in my searches.
Clay is on the money with the observation that the real meanings of the terms are emerging from the statistical fog. You can make observations about the relations between tags based on the URLs that share them. "This URL probably represents a movie, as you think of 'movie'" is the function I need.
Monday, June 27, 2005
Service specific operations and machines
Service independent operations are valuable when the agent invoking them is the kind of agent that talks to lots of different services: a web browser + a human to make sense of what he browses and make choices accordingly.
If the agent is service specific, then heck, just design in all the specific operations you want.
Machine to machine conversations are almost always service specific. You have to program the client to understand how to proceed through the legal application states. The guy programming the client needs to know... First you do this to get this result, then you use that result to make a second query, and so on. It's no help to have a service independent operation set if you're using it that way.
Service independent operations would be valuable in the machine to machine case if you could invent a surrogate for the human: an intelligent machine agent able to make choices, given some output from the last operation. I've said that before, I know. Just thought it would be useful to state it a little differently.
(Caveat: written after a 16 hour day constructing SOAP services for machine to machine cases).
If the agent is service specific, then heck, just design in all the specific operations you want.
Machine to machine conversations are almost always service specific. You have to program the client to understand how to proceed through the legal application states. The guy programming the client needs to know... First you do this to get this result, then you use that result to make a second query, and so on. It's no help to have a service independent operation set if you're using it that way.
Service independent operations would be valuable in the machine to machine case if you could invent a surrogate for the human: an intelligent machine agent able to make choices, given some output from the last operation. I've said that before, I know. Just thought it would be useful to state it a little differently.
(Caveat: written after a 16 hour day constructing SOAP services for machine to machine cases).
Tuesday, June 21, 2005
WSDL 2.0
Dave Orchard's remarks on WSDL 2 encourage me to replace my own cooked up SDL with WSDL 2. Sounds a little daunting though: "WSD WG decided that the specs were for toolkit authors not wsdl document authors."
I will report experimental results here.
I will report experimental results here.
RESTful Web Service Descriptions
I've fleshed out some of the details I left dangling in my previous post about WITSML. The WITSML stuff is so specialized, I've created a separate blog for it.
But if you are interested in the web description language (web-http-desc) discussion there is some meat here for you as well. This draft documents RESTful access to WITSML services, and writing it was a great exercise for the description discussion. (I didn't write it as an exercise; our product implements some of that protocol right now; but I had never written it down in one place).
WITSML is "Well Information Transfer Standard" markup language. It defines not only document formats, but a SOAP API that has the following operations:
AddToStore
GetFromStore
UpdateInStore
DeleteFromStore
Look familiar? Well GetFromStore is a query, and has a query string parameter, but generally, you can do 90% of WITSML just by GET, PUT, and DELETE on objects, and the other 10% are vanilla things you could do with POST.
If you are interested in the web description language discussion, I invite you to have a look at the WITSML+REST draft and comment here or on the list.
But if you are interested in the web description language (web-http-desc) discussion there is some meat here for you as well. This draft documents RESTful access to WITSML services, and writing it was a great exercise for the description discussion. (I didn't write it as an exercise; our product implements some of that protocol right now; but I had never written it down in one place).
WITSML is "Well Information Transfer Standard" markup language. It defines not only document formats, but a SOAP API that has the following operations:
AddToStore
GetFromStore
UpdateInStore
DeleteFromStore
Look familiar? Well GetFromStore is a query, and has a query string parameter, but generally, you can do 90% of WITSML just by GET, PUT, and DELETE on objects, and the other 10% are vanilla things you could do with POST.
If you are interested in the web description language discussion, I invite you to have a look at the WITSML+REST draft and comment here or on the list.
Wednesday, May 18, 2005
Messages not Models
Visiting the WITSML SIG meeting and expo last week, sponsored by POSC, recalled the evolution oil and gas E&P IT systems are undergoing. Unlike lots of other industries, the E&P guys have tried in the past to normalize on an industry logical data model. One standards effort was promulgated by POSC, the Epicentre data model -- it is enormous, beautiful, and unused. PPDM is a separate standards effort. Landmark and Schlumberger each offer proprietary models OpenWorks and GeoFrame. Yet each company using one of these models populates the database assigning slightly different semantics to properties, and many have tweaked the models to suit their own practices. So interoperability has suffered.
Enter WITSML. Oh, it doesn't (yet) attempt to address the full range of E&P information -- just a small segment concerned with drilling operations. But it is the very first message exchange system for this industry -- not just an interchange file format, but a message format together with a protocol.
Now that we agree on the messages, who cares about the models? You keep your model, I'll keep mine, and we agree we'll exchange resource representations, not models. It's a great use case for REST.
An E&P application can sit on top of whatever model it prefers and expose its services to the world using the REST prescription:
1. Identification of resources: assign a URI to resources like well, wellbore, etc.
2. Self-describing messages: Accept and deliver XML descriptions of well, wellbore, etc. The representations do have to map to components of the underlying logical data model.
3. Hypertext as the engine of application state: This is the key. In the past, the trouble has been caused by applications maintaining state in the database. A client has to understand the model and has to keep up with the evolution of that model. Using hypertext as the engine of application state, the server in effect programs the client to proceed through the states of the application correctly. As I've pointed out elsewhere, whether the client is a human operating a browser, a human operating a rich client, or an automaton exercising a web service, the server delivering instructions (a "form") to the client effectively is programming that client: "Here are the next steps you can take from here: a) do a GET on this URL or b) serialize these parameters and do a POST to this URL".
So WITSML could be the beginnings of a pattern for extracting the industry from slavery to models.
Oh, except for that silly SOAP API they've crufted together. What I've outlined here is only the possibility enabled by having defined the messages. But because of the RPC style of the WITSML SOAP exchanges, resources aren't identified by URIs, and hypertext most definitely is not the engine of application state. These problems are solvable. More on that later.
Enter WITSML. Oh, it doesn't (yet) attempt to address the full range of E&P information -- just a small segment concerned with drilling operations. But it is the very first message exchange system for this industry -- not just an interchange file format, but a message format together with a protocol.
Now that we agree on the messages, who cares about the models? You keep your model, I'll keep mine, and we agree we'll exchange resource representations, not models. It's a great use case for REST.
An E&P application can sit on top of whatever model it prefers and expose its services to the world using the REST prescription:
1. Identification of resources: assign a URI to resources like well, wellbore, etc.
2. Self-describing messages: Accept and deliver XML descriptions of well, wellbore, etc. The representations do have to map to components of the underlying logical data model.
3. Hypertext as the engine of application state: This is the key. In the past, the trouble has been caused by applications maintaining state in the database. A client has to understand the model and has to keep up with the evolution of that model. Using hypertext as the engine of application state, the server in effect programs the client to proceed through the states of the application correctly. As I've pointed out elsewhere, whether the client is a human operating a browser, a human operating a rich client, or an automaton exercising a web service, the server delivering instructions (a "form") to the client effectively is programming that client: "Here are the next steps you can take from here: a) do a GET on this URL or b) serialize these parameters and do a POST to this URL".
So WITSML could be the beginnings of a pattern for extracting the industry from slavery to models.
Oh, except for that silly SOAP API they've crufted together. What I've outlined here is only the possibility enabled by having defined the messages. But because of the RPC style of the WITSML SOAP exchanges, resources aren't identified by URIs, and hypertext most definitely is not the engine of application state. These problems are solvable. More on that later.
Subscribe to:
Posts (Atom)