Clojure

Clojure is a Lisp that runs on the JVM. I agree with everything on this page describing the design rationale. Objects are overrated; functional and immutable are good. I can't wait to get a breather and figure out whether and where Clojure fits into our technology arsenal.

The real measure will be how well it can leverage the Java platform. There are plenty of great languages, but they have inconsistent support for, say, sockets, or window systems. Can I override a protected method of a Swing base class? Can I connect to an HTTPS server? JRuby has knocked Java integration out of the park, so you get all that capability for free, plus a powerful language.

Update: I think the bullet point "Extend Java in Java, consume Java from Clojure" on the rationale page suggests you can't easily write a Swing app in Clojure. Shoot.

RIA using JRuby and Web Start

I went dark over the last eight weeks, building a rich web application to integrate with our product. It's been years since I built a desktop GUI application, but I was pretty good at it once. It's not always a lot of fun. But some things a man just has to do himself.

The application does tons of vector drawing.... it draws hundreds of thousands, even millions, of line segments. Yet it needed to be responsive... scrolling, zooming, panning. It's data intensive, too. It downloads dozens of megabytes of data for each view. I needed to develop this program fast, so I needed a programming system and platform allowing me to iterate and adapt to new requirements recursively revealed. I preferred a declarative graphics language where possible, and for any procedural code, a powerful dynamic language. Ideally the program should run on all OS platforms; but the business could probably tolerate a Windows only solution, if that offered superior results. I made tradeoffs among these requirements.

I was not constrained by download footprint, a concern which eliminates Java from consideration in lots of other scenarios. Anyway, that problem is going away soon, they tell us.

So here's the spoiler: I chose Java Web Start + JRuby. And now a short dissertation on how I arrived at that choice:

I looked to browser and browser++ technologies like SVG, Flex, AIR, Silverlight, Java applets/Web Start/JavaFX. (Open Laszlo deserves a look too, but I didn't eval it). It would be great for this app to run within the browser -- that would be the most seamless experience. But using an out-of-browser technology like AIR or Web Start isn't a deal breaker if you do it right.

SVG was very promising. It would be a pure browser solution. I really valued the declarative model and the standards alignment. But it does not work in IE. I also did some performance experiments with Opera, Safari, and Firefox. Drawing polylines of a hundred thousand points or so, Opera and Safari performed well, but Firefox performed poorly. It's conceivable, if unlikely, that I could distribute the app as a FF-only solution -- even if I bottle it up as a a XULRunner app. But the FF performance was inadequate. And distributing only for Safari or Opera is not realistic.

I made a significant evaluation effort with Flex/AIR. Surprisingly I learned that the Flex 2-D graphics model is procedural, not declarative. You can, however, bottle up your procedures and use them declaratively from mxml, the Flex XML language. I felt I was going to spend a significant amount of time learning Flex -- time I needed to spend on my app -- and that I needed the assistance of tools like Flexbuilder. I also became concerned that I was developing a program away from the design center of Flex. Would I encounter design/ performance constraints?

Silverlight performed very well, and it's got an XML declarative model just like SVG, with the power of a bunch of WPF components to boot. I was confident that all the capabilities of the Windows platform would be available to my program. Microsoft distributes Silverlight for Windows and Mac, but not Linux, which they are leaving to Novell. I read that as: No Linux. But Windows and Mac are enough coverage, even if I would be sorry to leave aside my Ubuntu dev machine. My fate on the Mac would be in Redmond's hands, of course. Does Microsoft have my best interests at heart ?

Java applets or Web Start, using Java 2D, I also knew to be capable of these rendering tasks. I'm satisfied that I can make Java applications now that look pretty sweet. The down sides were the procedural approach and the static Java language. It's the 1999 state of the art. Layout code so impenetrable you need GUI builders.

JavaFX deserves a separate mention. The capabilities of the entire Java platform are there for you. Yet you program in a declarative/dynamic language optimized for building Java 2D apps. Perfect! Unfortunately, it's pre-alpha. Sun hasn't yet released it under a license allowing you to put your work into a customer's hands.

I wanted the wide reach of Java. I think that the 5% of customers on non-Windows platforms may be more important than their numbers.

I also wanted the enormous hardened Java platform. It's as capable as the Windows platform. You won't encounter many problems you can't address or workaround.

Couldn't I have all that, plus a dynamic language to stitch it all together?

Guess what. JRuby is stable and 1.x. It's dynamic, yet it compiles on the fly to JVM byte codes. It's got great Java integration, meaning you can invoke any Java API. Profligacy is a terse little Ruby lib for stitching up Swing components. JRuby's just a jar I download with my Web Start app, and launch with the main Ruby script from my Java entry point. The rest of the program is in Ruby, occasionally calling into Java libraries. No, there's no declarative GUI language here. It's one of the tradeoffs.

My work environment: Fire up NetBeans for its good Ruby editor. In a command window kick off jirb and execute a script to load and run my main code. Edit code and reload it into the running program in jirb. Lather, rinse, repeat.

I would argue that JRuby accelerated my application's performance, because I iterated and applied two major architectural optimizations that would have introduced impenetrable spaghetti in Java. The bottleneck for speed and responsiveness has never been JRuby. It turned out that even Java2D + acceleration could not keep up acceptably with the level of interactivity I wanted. The SVG or any other pure declarative approach would have failed (although I think JavaFX's scene graph might be able to optimize the drawing). I had to go in and surgically rearrange the innards of the program. The changes were effective but had low impact on stability, thanks to the flexibility and power of Ruby.

If Sun can focus more energy on these dynamic languages built on the JVM, that would be a pretty powerful story: The deep platform capabilities, the broad reach, the rich development environment.

And Speaking of Paul Graham...

Arc to be open sourced this winter. It's going to be hard to top the expectations.

The Truth about Lisp

Majorly funny. Leon Bambrick wrote (back in 2006):

If you're good enough to use lisp, you'll soon be frustrated with lisp. Lisp is not an adequate lisp. By the time my bus had made it two blocks I'd written some simple lisp macros that were so powerful they made lisp completely obsolete and replaced it with a new language. Fortunately, that new language was also called lisp. And i was able to prove, mathematically, that the new lisp i'd created was both far superior to lisp in every conceivable way, but also exactly equivalent to lisp in every possible way. I was very excited by this. But also found it very boring....Paul Graham himself was completely written in lisp, by an earlier version of himself, also written in lisp, by an earlier version of lisp. It's lisp, paul graham, lisp, paul graham, all the way down.

Why objects suck

class Curve {
String name;
float[] data;
Color color;
}

class Plot {
Curve[] curves;
void addCurve(Curve c);
void removeCurve(String curveName)
}

"No, no, no... you're mixing in style information. Pull the style out of Curve and separate the concerns." Oh. OK...

class Curve {
String name;
float[] data;
}

class CurveStyle {
String curveName;
Color color;
}

class Plot {
Curve[] curves;
void addCurve(Curve c);
void removeCurve(String curveName);
void setCurveStyle(String curveName, CurveStyle style);
}

Well, all right then! I'm sure this in a patterns book somewhere. I feel warm satisfaction with the pure, objectified minimum entropy of my factorization. Until I implement Plot.removeCurve... oh shit... I have to search for all the dangling CurveStyles referring to the curve, and delete them. As OO programmers we do loads of that every day. You come to think of it as natural. The best OO languages have list comprehensions that make it easier to do. That's nice, but it's not enough. The problem is that "objects" aren't a logical model.

Wouldn't it be more "natural" for the dangling CurveStyles to delete themselves when no longer relevant? There's a logical model for that, the relational model:

Style

CurveName	Color
MSFT	Red
GOOG	Blue

Curve

CurveName	Data
MSFT	1,4,-9
GOOG	13,2,22

Logic tells you that the style row cannot exist when the Curve row has been deleted. A foreign key relationship forces the system to delete the style when you delete the curve.

I want my programming language to do that for me, and that's part of what I was wishing for two years ago.

FIQL: Query language for feeds

My first reaction to Mark Nottingham's FIQL draft for querying Atom and RSS feeds was: Do we really need to bake in a prescription for constructing query URLs? Section 4 seems to suggest that. It seems to take the control over the URL namespace away from the server.

Here's Mark's example HTTP FIQL (I'm already saying "fickle") query:

http://example.org/feed.rss?title==*great*;ex:rating=gt=4

Element names become the query keys. So if your feed language has element <ex:rating>, you can use that as a query key.

FIQL describes rules for constructing URLs based on your content type. Contrast that to HTML forms which prescribe how to construct URLs based on content.

Is that so bad? It's a little too API-like for me. Any Atom feed now has that query namespace imposed on them: If you want to honor queries, these terms become part of your URI space. Amazon, Blogger, you, I all have this URI subspace imposed on us.

If, instead, we define a FIQL content type, application/fiql, and you POST a FIQL document, the server can respond with the query result, or can construct a URI of its own design and redirect to it.

The server retains complete control of its URI namespace.

I'm interested in how we work through this issue because I just suggested a similar strategy for WITSML queries a few days ago.

What is wrong with this picture?

Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
content-disposition: attachment; filename=Hill%20Country%20water%20issues[1].ppt
Content-Type: application/unknown

IIS servers don't come provisioned with the .ppt extension mapped to application/vnd.ms-powerpoint? Microsoft server? Microsoft application?

What hope can there be for authoritative metadata?

My Firefox browser figures it out just fine, and launches Open Office. Presumably FF went through all this first, just to determine it to be application/octet-stream; then ran a further sniffer to identify it as a Power Point.

It's the hyperlinks, Stupid!

Henry Story thinks Echo2 is "Web 2.0 in Java", and they do have a killer demo. But it's yet another example of incredibly brilliant developers going to great lengths to bring the desktop app to the browser, and ignoring that the value of the web lies in hyperlinks.

So navigate that demo: Click the "Next" arrow. Your address bar doesn't change even though you have navigated to what most people would call a clearly identifiable different resource.

I can't give you the link to "page 2" because it isn't addressable on the web. Sorry. Go to page 1, then click to go to page 2.

Under the hood, it's the usual collection of RPCs masquerading as URLs -- some GET, some POST -- using port 80 since we know it's open, after all. Every single GET uses all the usual tricks to make sure that nothing -- not even JPEGs -- gets cached.

(Why don't we just give Javascript in browsers an API to open up a socket and execute any protocol you like? Seriously: Wouldn't that be better than abusing HTTP? If you don't want to use the HTTP protocol, you shouldn't have to.)

I love the look of that demo, and I think their technology is clever. With a little effort I am sure they can make the framework webby.

I was just thinking that!

I've been working on the technology requirements for a "rich web application". All the pieces are in place for a cross platform solution. With Javascript, HTML, SVG, and CSS, you can build a pretty rich application without resorting to JavaFX, AIR, Flex, Silverlight, Click Once, or Web Start.

All the browsers support JS, HTML, SVG and CSS. Except one. Per Rob Sayre:

If Microsoft were really interested in making life easier for web developers, they could do so, without a standards committee. They would need to fix the (nasty) bugs in IE’s JScript engine, implement SVG, implement canvas, implement more of CSS, support a standard event model, and on and on. Then, the behavior of IE would be a lot closer to Firefox, Opera, and Safari.

It's no secret Microsoft doesn't see an advantage in a web built on cross platform technologies. I'm not very sure about Adobe, either.

Why should we, as a a software company, invest in technology from companies that are actively working to subvert what we want to do?

Rich Web Applications

Here's the five step test to determine whether your "rich internet application" is a rich web application:

1. You click on a hyperlink in a web browser.
2. Your RWA opens up, and updates its code if necessary
   
3. Your RWA renders the document, and you edit it.
   
4. You save the file -- to its URL
   
5. You e-mail the hyperlink to someone; they open up the document and see your edits.

You should be able to click a link to a Word document, edit the document, and save it -- to the URL you got it from. The RWA is a rich, articulate way to render and edit particular kinds of documents.

If saving isn't a feature of your app, fine: But honor (1) ,(2), and (3).

You can do this with .Net Click Once, Java Web Start, and for all I know, Adobe AIR. But the design of the first two, at least originally, amounted to a way to install and launch your application by clicking a hyperlink, and update the code automatically. Worthy objectives. But no actual Web Start or Click Once apps I have used can do this: Click a document, edit it, save on server. You couldn't possibly do it with Web Start until JDK 1.5. And to do it with Click Once you have to sign the application and party in the registry.

The first thing is to register your RWA as the helper application for your document's mime type and file extension. Web Start enabled doing so when they added <association> to the JNLP 1.5 format. Web Start registers itself as the handler for your file extension; the browser launches javaws to open your file, and javaws looks at the file extension and invokes your application. Or: sign your Click Once application and you can set your application as the mime type handler in the registry.

That makes your application launchable when you click a link.

You will have to design the URL of the document into the document itself. Browsers download documents and save them to temp files; they don't tell you the URL of the document. When the browser launches your helper app, your app needs to know where to HTTP PUT updates, or to DELETE the resource. You can also design in other URLS your client understands: a base URL under which you can POST to implement "File/New", for example. Notice Atom has <link rel="self"> and Atompub service documents tell you where to POST things; you could build a good RWA around that scaffolding. Make your own documents describe how to navigate the states of your application by traversing hyperlinks.

Your RWA should have an address bar. Show the users where they are. Let them type in any URL they want to. If they type in one for a mime-type you don't understand, hand off the download to the registered helper application, or back to the browser.

All those buttons desktop apps use to navigate? Menu commands? Keyboard shortcuts? Back them with URLs you GET from or POST to. Display each URL in the address bar as the user traverses your application. After clicking any button that does GET, whatever you see in your RWA should be what your friend will see when you email him the URL in the address bar.

Automatic code update (step 2) is a necessary feature for a RWA. It's not enough just to have a link launchable application. You can evolve your schema, and change the meanings of elements in your document format, as long as you know that only the most modern code, that understands the new meanings, will be interpreting the document.

Make the extra effort to leverage the web architecture, and I won't blog about your beautiful but shitty application.