Finder Not Responding: Lion and Mountain Lion

After upgrading to Mountain Lion, Finder frequently became not responsive. The culprit was the "All My Files" feature in the sidebar. Finder gets hung sorting and displaying the 15 gazillion files on my disk.

Here's how to fix it:

1. Open Finder Preferences.
2. Under Sidebar, uncheck "All My Files"
3. Under General, set "New Finder windows show:" to anything but "All My Files".

I liked All My Files. I hope Apple will put some effort into optimizing this view.

MVC Frameworks. Keep 'em.

I said it here in 2006. MVC on the server is a poor fit for web apps. Here's a more up-to-date critique.

Subsurfr, which I introduced here a few days ago, is an example of what that author calls a "serverless" web app. All the app logic is in the js + html. It uses back end resource servers for maps and drilling data, but there's no PHP/Rails server side logic. That's going to change a tiny bit in the near future to support personalization, but substantially the app state will remain in the hypertext.

Jakob Nielsen: Web will beat mobile apps

Although he thinks that apps offer a better user experience right now, and offers evidence to back that up, Jakob Nielsen officially predicts that web apps will surpass mobile apps.

More ammunition came a short time ago from the Financial Times, who released a rich iPad web app that is every bit as articulate as a native app. The FT's designers discuss that decision here.

The excellent FT iPad web app suggests that native app UX only surpasses web app UX because typical web apps don't put in the effort.

There's very little I find in native mobile apps that I could not do as well in a web app (HTML5 plus concessions to using platform specific CSS and JS calls). In some ways, the UX already is better in a web app. I almost always find that native apps lack some critical functionality that the web app has. And the web app has seamless hyperlinkability, in and out.

Exploring the Bakken using WebGL

In the spirit of Done is Better than Perfect, we recently unleashed Subsurfr, a 3-D subsurface visualization web app for oil and gas.

We primed it using 17,000 wells from North Dakota. Since the Bakken shale is a very active play, some geologists and engineers are going to be very happy to have this new arrow in their quiver. Here's a fun look at the very busy Bakken from a well planner's pov.

Building this app pushed a lot of boundaries (for us, anyway). First among them was WebGL. And to broaden the reach to IE, we use Chrome Frame. But also, Subsurfr is a mashup, and exercises Cross Origin Resource Sharing (CORS) capabilities to permit it to use data served directly to the browser from a variety of sources. We also experimented with BrowserID, and OpenID + OpenAuth before chucking identity altogether this iteration -- the business driver isn't in place.

WebGL is a lot more mature and stable than I would have expected. After all, the spec has only been finalized in the last few months. But, after you get past the unfamiliarity of the GL computing model (GLSL, vertex shaders, fragment shaders), and the
trial and error nature of debugging (you can't even print to a console from GLSL) , I found our code just works on every WebGL platform, including Firefox on my Android phone.

Although three.js seems to be the weapon of choice for most new WebGL projects, I chose the lower level threedlibrary (tdl). I didn't do a lot of comparison shopping before I made that decision. It turned out to be a good choice. Because tdl is close to the metal, I never ran into any WebGL capability that I needed but could not use. Yet, as a WebGL noob, I needed the structure of tdl -- tdl taught me how to use WebGL properly while I was actively developing working code. 'Cause that's how I roll. I'm not going to spend $50 on a WebGL book and research the 5 top libraries and do a three week test project in each one before coding.

The web pages making up Subsurfr itself are totally pregenerated -- presently about 35,000 of them. 'Course, lots of the data still comes down via ajax to the browser from the cross-origin servers. By serving only static pages, Subsurfr can run on a small nginx server. How the ajax data servers are going to hold up -- ask me in a few weeks.

CORS itself is a minor challenge. First, it does not run in IE < 10. That's not a problem for Subsurfr because Subsurfr already has to use Chrome Frame to run in IE. However, we use MapQuest as our map server, and they never heard of CORS. Consequently, we chose to proxy our XHR calls to MapQuest via our own server which adds the CORS headers. Additionally, we use the dojo toolkit to execute those XHR calls, and dojo has its own issues there. Finally, our own drilling data server had to be retrofitted to send the CORS headers. So we have several workarounds and patches to get all the CORS stuff going. Still beats IFRAMEs.

Anyway -- check out subsurfr.com to see some real world WebGL in action. If you are interested in Subsurfr from an oil and gas perspective, I've written more about it here.





.

Inflexible, unmaintainable, fragile code in Clojure

I constructed a subsystem in Clojure. We had a nice, compact problem to solve. The rest of our app talks to this Clojure bit through a small interface. The attraction of Clojure was that I could enjoy all the nimbleness of hacking Lisp interactively. Using such a powerful language, I could adapt to emerging requirements, experiment, twiddle, and workaround faster than using Java. And early on, that largely proved true.

But I am here to tell you: It is possible to write inflexible, unmaintainable, fragile code in Clojure.

As they say in bicycle racing, the riders make the race. I never said I was a good programmer.

It's my own fault. I dived into Clojure and learned key concepts on the run. And these key concepts are fundamental to how you should structure your program. FP has a learning curve. Managing shared state using Refs, Agents and STM -- better to understand these concepts up front. Even now, I'm unsure how I'd restructure my code to use them properly. I'm very glad not to have to worry about deadlocks. But there are parts of this code I still am unsure how to to test for consistency under multithread access.

Consequently, I'm slogging through this bowl of spaghetti of my own making, cursing the author.


[Edited to add "fragile" ... as I made one more change and my tests came tumbling down... ]

Wolfram Alpha isn't sure what to do with your input

I have yet to get any answers to any interesting, real question I have had, from Wolfram Alpha.

If RMS has his way, the GPL is dead

The GPL has never made me feel particularly free. Only in a kind of newspeak can you say that constraining the kinds of agreements I can make with people who buy my software enhances my freedom .

I'm not saying I don't like and use GPL software, or that the GPL is unfair. Just don't tell me that handcuffs are extra freedom.

Only copyright law enables GPL to constrain my actions from beyond the grave. In a world without copyright, GPL would be toothless. I would be able to use any freely available software I could get my hands on, incorporate it into my own, and distribute only binaries.

Richard Stallman proposes reducing copyright protection to three years for software. And he admits "It would be necessary to prohibit the use of contracts to apply restrictions on copying that go beyond those of copyright." That prohibition would apply to FOSS too, of course. No copyright after three years; no contract allowing the author to constrain your actions after that.

That would mean that you could link to three year old GPL libraries and ship binaries!

The REST Hypothesis

I'm glad Joe Gregorio frankly assessed Atompub's meager adoption.

Atompub served as an experiment to confirm the REST hypothesis: If you construct distributed systems that look like the world wide web, the world will adopt them broadly and quickly -- the way we adopted the web itself. *

Lesson learned: the world's most successful distributed applications run on the web's architecture. That doesn't mean that by designing around web architecture, you will build the world's most successful distributed application.

Atompub came along at the time a critical mass of thought was building in favor of adopting the web architecture for designing new systems, rather than the ill-named "web services", SOAP and RPC model. Early drafts of Atompub allowed for a SOAP envelope.

The RESTful style prevailed, on its many merits. We got caching, and a resource oriented model, and a small, uniform interface.

But what the real, browser plus HTML web has, that RESTful systems don't, is the user agent. The human in front of her browser. An intelligence that reads and understands the meaning of "Author name" and "Title", and fills in an HTML form using queries against her personal database, stored in her brain.

As Joe put it, the problem is that Atompub clients aren't web browsers.

RESTful systems that aren't web browsers try to substitute understanding of media types for that intelligence. Compare Atom clients to HTML browsers.

An "Atom application" is some sort of content management system: a system that understands the semantics of feed documents. An Atom agent populates <author> and <title> elements. Only machines understand the meaning of <link rel="edit">. It's a world constrained to a narrow range of meanings.

But an HTML application can do anything. A human reads some text next to a form field, labelled "Author name", or "Preferred airline", and enters meaningful answers. If you are writing the kind of content management system where "Preferred airline" is a meaningful concept, HTML might be the way to go. There's no such concept in an Atom feed document. You get to present your own user interface, too.

RESTfully designed systems might profit from superior evolvability, cacheability, and interoperability over RPC systems. They are of the web. But they are not the web. The web is in your browser. The web is HTML.


*The hypothesis was not part of Roy Fielding's thesis. It's a hypothesis that many REST proponents, including me, have deployed.

Cheney Wheelchair Horror

Best comment so far:

If only he had a small cat to pet as they rolled him to the car. You know something that he could stroke until he snapped its neck to suck the blood from its spine.

(by "Brian")

Clojure vs JavaFX Script: only 5x slower!

Chris Oliver compares his JavaFX script to JRuby and Groovy and finds JavaFX 25 times faster.

I failed to get JavaFX compiler running on my Ubuntu, but I easily ran Chris's JRuby 1.1.6 test; so for calibration, my laptop comes in at 3:52, vs 4:22 on Chris's machine. Presumably we can scale results on my machine by 1.13.

Here's my Clojure version of the test:


(defn tak [#^Integer x #^Integer y #^Integer z]
(if (>= y x)
z
(recur (tak (- x 1) y z)
(tak (- y 1) z x)
(tak (- z 1) x y))))

(dotimes [n 1000] (tak 24 16 8))



It runs in about 50 seconds, or 56 after normalizing to Chris' machine speed. This puts Clojure north of JRuby and Groovy, but still 5 times slower than JavaFX.

If you remove the type hints, it runs in 4 minutes, or about the same time as JRuby -- and about 4 or 5 times slower than with type hints.

Clojure uptick

Needing to build a standalone piece of server software as an add on to our Java product, I seized the opportunity to see how far Clojure has come. Getting my dev system set up, and learning the language, I noticed the results Google was returning for my queries were mostly very recent... and lots from just last month.



Could Clojure be approaching critical mass?

I've only written a hundred lines or so of code, but here are some random things I love:

1. Shorthand for maps: {:a 1 :b 3}.
2. sorted-map and hash-map -- thanks for letting me choose. Same shorthand syntax for either.
3. Shorthand to select a value from a map: the map is like a function name (my-map :a).
4. Shorthand for lambdas: #(...code...)
5. Shorthand for generating temporary symbols in macros: my-var#. No need to call (gensym).
4. No looping required -- use recur.
5. Java libraries are easy to use.

I haven't yet fiddled with modifying state.

It's free, it's functional, it's got a hardened standard library, it's got a REPL. You can write a program and deliver it on any platform. What could hold it back?

Flying the F-22

My gut reaction to the internet chatter about distributed version control systems, like Git, was, "What advantage could it possibly offer us, that would justify me and developers spending so much time re-learning how to manage source code?" CVS kinda sucked, but was good enough, and Subversion was better... what more could you want? It's just version control.

But we had a branching requirement, and a customer requiring a branch we'd maintain for a few years.

From the Subversion manual:

Tracking Merges Manually

Merging changes sounds simple enough, but in practice it can become a headache. The problem is that if you repeatedly merge changes from one branch to another, you might accidentally merge the same change twice. When this happens, sometimes things will work fine. When patching a file, Subversion typically notices if the file already has the change, and does nothing. But if the already-existing change has been modified in any way, you'll get a conflict.

Ideally, your version control system should prevent the double-application of changes to a branch. It should automatically remember which changes a branch has already received, and be able to list them for you. It should use this information to help automate merges as much as possible.

Unfortunately, Subversion is not such a system....It means that until the day Subversion grows this feature, you'll have to track merge information yourself. The best place to do this is in the commit log-message.

OK! So, relearning version control might not be a bad idea!

Git is all about changesets, and it can merge changes from branch to branch painlessly. A few months ago, we imported a year or so of changes from SVN to a Git repo (our tree structure was way too f***ed up for git-svn), and never looked back.

I'm a Piper Cub pilot in the cockpit of a F-22 Raptor. There are a lot of switches. I kind of understand some of them. I'm afraid to touch most of them. Every now and then we'll try a new trick -- hey, I think I can just push this changeset to you, from the branch I'm working on to yours... SHIT, THAT WORKED! We haven't crashed yet, and we have the rumbling sense of untapped power.

For reasons unrelated to version control, it turned out we could manage the customer requirements without a long running branch. But we're not going back on Git.

REST Blessed

Gartner has blessed REST architecture by giving it a three letter acronym and a $495 report. But you can get the gist of the paper for free in Nick Gall's blog post: WOA: Putting the Web Back in Web Services.

Commercially, it's a big help that I can now point a customer's solution architect to a Gartner report, rather than to a guy's thesis and 20,000 posts to the REST-discuss mailing list. I hope that doesn't sound cynical -- it's not. It's just that architecture arguments are hard to win just on the merits. You need a little mindshare too. Thanks, Nick!

Electronic Vote Fraud

I think I know how we can use electronic voting to increase our confidence in vote counts -- as opposed to what we have.

What we have is a mad world where you can construct the most extreme hypotheses about what is happening to your vote, and they could be true. You can't prove they're not.

When I complete my vote, I want to receive a printed receipt with a) my vote in clear text, b) a hash of my identity, and c) my digital signature of the above, for non-repudiation.

I should be able to log onto the county's web site, and verify that my vote was recorded for my candidate:

Voter	Candidate
38907389343666	McCain
84567867435432	Obama
45878963063630	McCain
92573532235672	Obama
...	...

Anybody can look at this table and count the votes for McCain and Obama ,and confirm that they match the announced totals. They can confirm, too, that their own vote was counted for the proper candidate.

My voting receipt matches one row of the table. The voting machine prints it out, and I review it. I should be able to confirm the hash of my identity by some public algorithm. It could be a hash of my voter registration number plus some secret password I make up.

My receipt also has a digital signature that I apply only after I review its accuracy. Yep, I voted for that guy. I insert the paper into the machine and it prints a string that's the digital signature of my identity and my vote. That way, I can't claim the machine got my vote wrong -- I reviewed the clear text vote, and signed it.

Voting this way would be lots more verifiable than paper ballots and hanging chads. You'd be able to verify your vote was counted.

Since early voting began Tuesday in Travis County, Texas, the email chain below has grown virally (local news story). And how can you prove these fears are wrong?

From: redacted
Date: October 22, 2008 11:48:50 PM CDT
To: redacted
Subject: DOUBLE CHECK YOUR BALLOT before pushing VOTE

no matter who you are voting for:
A friend of a friend of mine has this horror story about voting this morning
in Austin, TX.

He voted at Fiesta Mart on 38th and IH-35.

He voted a straight Democratic ticket.

When he was reading the 'voted for' listing at the end of his
ballot, all of those listed were Democratic candidates EXCEPT FOR
PRESIDENT. The list showed that he had voted for John McCain!!!

And he voted the straight Democratic ticket.k

He reported it to the election official and that person was as
shocked by it as my friend. They corrected the vote BEFORE he hit
the CAST BALLOT button.

My friend said the experience made him sick to his stomach. He said
he was the youngest person in the voting area and all he could
think is that the older people around him may not proofread their
ballot before pressing the CAST VOTE button. They may believe that
they voted for Obama but the voting machine may have registered a
vote for McCain.

He called the Travis County voting office and they said they would
look into this. When he called me I gave him the telephone number
for the Democratic Party and he then called them to report what had
happened and they said they would look into it, also.

Please PROOFREAD your ballot choices BEFORE hitting the CAST BALLOT
button. This is vitally important. (For EVERYONE who votes,
whichever way you vote.)

Leaps

Dear Lazyweb,

1. In 1929, how many Wall Streeters actually did leap to their deaths from their office windows? I'd be surprised if it were as many as 10. Growing up during the sixties, I had the impression it was raining stockbrokers on Black Tuesday; but now I find it hard to believe.

2. They had names. Who were they?

3. Why did they kill themselves? In 1929, was there a greater sense of personal responsibility for the margin calls they wouldn't be able to make? A greater fear of the shame that would attach to them personally?

4. Has any Wall Streeter killed himself or herself during the current financial crisis?

5. Have any of the current super wealthy attempted to guarantee the system using their personal wealth, as Rockefeller did in 1907?

I pray the answer to 4 is no.... yet I wonder about what level of personal responsibility we should expect, and why it's different than 1929 and earlier.

Dell Precision M2400 + Ubuntu Hardy

Welcome, Googlers. You are here because you have put Ubuntu Hardy (or perhaps Intrepid) on your shiny new M2400 (or other Precision laptop). And you've run into issues with your wireless; to wit, it don't work.

I've got the nVidia Quadro 370M, and Intel wireless 5100 cards. A fresh install of Hardy on a Dell Precision M2400 supports the nVidia (howto). But the 5100 wireless card support is non-existent.

To get both of these devices working, you're going to have to build a new kernel, and also compile a beta nVidia driver interface. It's actually an automated and fairly bulletproof process, so don't lose your nerve.

I think there are some other approaches that could work. I tried the compat-wireless strategy -- it's a technique using the recompiled Intel 4965 driver plus new firmware (oh yeah... you're going to need that 5000 series firmware from the Intel site anyway, so go get it). That strategy actually worked... as long as I ran "make load" on every boot... and occasionally my keyboard would freeze, requiring a reboot. I never could get that approach to feel stable, and my googling left with the impression that nobody had taken it much further than I had.

But to focus on the strategy I got working:

The problem is that the new Intel iwlagn drivers are part of the very latest kernel. I do not believe they are in even the Ububtu Intrepid kernel, so I did not attempt to promote to that alpha version of the distro. You can google to find lots of people dist-upgrading to Intrepid only to find they still don't have support for the 5100.

But there is support in the 2.6.27-rc6 kernel, the latest stable development pre-patch.

Build the 2.6.27-rc6-ultimate kernel

If you have X + nVidia set up the way you like, now would be a good time to save a copy of /etc/X11/xorg.conf.

You want to build this kernel the Ubuntu way. I suggest you follow the command line instructions on that page, and avoid the KernelCheck GUI program. I tried KernelCheck first and it is a really handy way to build a kernel. But it seems not to have left artifacts you will need later, when you build the nVidia driver interface. I admit there could have been pilot error in my case -- but I know the command line method works correctly.

When you get to the "make xconfig" step, choose Find from the Edit menu and search for "iwl". Turn on all those options.

Don't forget to select 64 bit options.

I emulated KernelCheck's naming pattern and identified my kernel with the name "ultimate", so my make-kpkg command was

make-kpkg --initrd --append-to-version=-ultimate kernel_image kernel_headers modules_image

So my uname -r is "2.6.27-rc6-ultimate".

Follow the instructions all the way and install your kernel. Don't worry, your old kernel is still there and in case of trouble, you can always choose it in the boot menu.

Remember that firmware you downloaded above? Put it at /lib/firmware/2.6.27-rc6-ultimate/iwlwifi-5000-1.ucode. Then reboot, and you'll have wireless... but no nVidia.

Build the nVidia driver

When you reboot into your new kernel, the system will drop you into a default 800 x 600 X session. Don't attempt to configure X yet if it prompts you.

There's no pre-built nVidia driver for this kernel. I also found that only the latest beta nVidia driver interface compiled against this kernel.

In order to build and install the driver, you need to kill X and make it stay killed. Ctrl-Alt-F1 to get a console, then "killall gdm". Then, following the instructions on the nVidia site, run the install script you downloaded. It should even edit the xorg.conf file to install itself, but in my case I still had to go back and set the resolutions -- I just replaced xorg.conf with the one I had saved at the beginning.

Our only departure from the Ubuntu way was using the nVidia install script -- it puts nvidia kernel module in a place of its choosing, and it won't be managed by apt for us. But we're not going to be getting new kernels from apt anyway -- we built our own, and we're now going to take responsibility for upgrading kernels (as needed) and nVidia at the same time.

Tidying up

You want to be able to suspend and hibernate, and it just takes a couple of edits. To hibernate, you'll need as much swap space as you have RAM. With this computer, unlike my last one, I've done that, and hibernate now works after the edits.

The NetworkManager annoyingly asks you to enter your keyring password whenever you log in. Googling around, I found some solutions. One of them worked for me. I'm not sure I used the best solution, so I'm punting and letting you find your own answers.

Hey -- this backlit keyboard is pretty nice!

Update Sept 17: Using the Hardy config file as your initial config (config-2.6.24-19-generic), simply enabling iwl* as above leaves you missing two modules that Ubuntu installs only as part of the linux-ubuntu-modules package. Your sound card and webcam require extra modules. In make xconfig, be sure also to select "USB Video class" (if you have the webcam) and "Intel HD Audio".

Check your DNS

Dan Kaminsky has published a widget you can use to test your DNS server for the cache poisoning design flaw. You know, the one that allows a malefactor to send your PayPal requests to their own fake servers? The one that doesn't require any vulnerability on your computer, just the standard, unpatched DNS server that you use?

Both my home and business ISPs failed the test. I followed Dan's advice and pointed our routers to OpenDNS. I guess it's a good day to be an investor in OpenDNS. Except I can't figure how they're going to execute on their plan to serve me ads via DNS. Anyway, this is their moment of glory, and they seem to be holding up under the load. If Dan has any stock in OpenDNS, that would be clever of him. I am not suggesting he does. Unless it turns out that he does. Then in that case, I am suggesting it retroactively, and don't say I didn't tell you so.

Get with it, Pops

Dare gives evidence that older Googlers are coming away dissatisfied from the experience.

Here are a few disapproving quotes:

• It was obvious that they do not care that I had 12 years software engineering experience
• orientation towards cool, but not necessarily useful or essential software
• don’t have a career path for their employees
• There is no legacy code
• As all organizations mature they tend to add PROCESS

Isn't this the static vs dynamic argument? Old models of software development vs new? On premise software vs software as a service? Will legacy code even be important at Google? Shouldn't our goal be disposable code?

Aren't perspectives like this the reason Microsoft is perceived as a falling star while Google is a rising one?

Asynchronous HTTP POST

I've got to process a huge POST asynchronously. If it were small enough I'd just return 201 Created, with the URL of the new resource in the Location header. But this is a massive file upload that requires a bunch of processing, and checking, before I can create the resource. It takes so long to process, your TCP connection will time out.

202, right?

This is a job for 202 Accepted, right? I had always understood 202 was how to respond ansynchronously to a POST. I return some hypertext with a link you can follow to see the status. You follow that link, load the status page, and hit refresh until it shows "100% done" and displays yet another link to the resource you created. That's what the RFC says to do.

In a machine to machine case, e.g. Atompub, I have to define meaningful content types so that clients can follow the hyperlinks to learn how the POST came out. If you're thinking of using the Location header: It isn't blessed for 202; even if it were, we couldn't guarantee that the resource at the returned URL will ever exist.

Why not 303?

I prefer to respond with 303 See Other. Clients follow that Location header to a status page, eliminating the extra manual step required by 202. Retrieving that page would itself return 202 Accepted, until I've created the resource, or failed to. Then that URL would return 201 Created, or whatever the result would have been in the synchronous case.

This way, user agents just follow the semantics of HTTP, and never need to understand any application entities.

Summarizing:

• Any user agent will knows to follow the 303 to the status page automatically. This is the URI giving "the response to the request", which is a status page. Any time you want the response to this POST request, go to that URI.


• GET on that status URI, for some time, returns 202 and an entity giving "an indication of the request's current status".


• Finally, at some time when you check that URI, you get the final "response to the request".

The surprising part might be encountering 202 or 201 in response to GET. But nothing says you can't, and in fact that is the "response to the request".

(Ben Ramsey discusses returning 202 from the POST. It gets a little messy.)

Ruby: DSL for Writing Programs

Ruby:

PEOPLE = [{:first=>'scott', :last=>'guthrie', :age=>32},
              {:first=>'susanne', :last=>'guthrie', :age=>32},
              {:first=>'bill', :last=>'gates', :age=>50}]

def avg_age
  guthries = PEOPLE.select {|p| p[:last] == 'guthrie'}
  guthries.inject(0) {|s, p| s + p[:age]} / guthries.length 
end

Common Lisp:

(defvar  +people+ 
  (list 
   (list :first "scott" :last "guthrie" :age 32)
   (list :first "susanne" :last "guthrie" :age 32)
   (list :first "bill" :last "gates" :age 50)))

(defun avg-age ()
  (let ((guthries  (remove-if (lambda (p) (not (equal "guthrie" (getf p :last)))) +people+)))
    (/ (reduce #'+ (map 'list  (lambda (p) (getf p :age)) guthries)) (length guthries))))

These snippets calculate the average age of the people having last name "guthrie". The functions are both two lines long, but the Ruby one is more readable.

S-expressions give Lisp powerful ways to build code in code. But writing programs using S-expressions seems cumbersome compared to Ruby's syntax. The syntax of Ruby shortens the code you have to write to do everyday tasks. It's like a domain specific language for, er, writing computer programs.

You give up some programming power -- a fair trade if you rarely need that power.