Technomancy

in which an interview is posted

2012-01-27T03:58:25Z

The folks over at The Setup just posted an interview with me wherein I rant about hardware, interactivity, and Emacs.

Note: this was written up over a month ago; if I were interviewed today I couldn't help but mention the Nix package manager. I use it on my Debian Squeeze system to complement apt-get; I get all the system-level stuff that has to be stable from Debian and anything that needs to be fresh from Nix.

in which local hacking locations are surveyed

2011-12-31T02:25:58Z

I'm lucky to have the chance to do my job remotely from wherever I like. Half the time I work out of my code lab, which is a converted shed in my back yard. But I venture out fairly frequently to local coffee shops. Here are a few of my favourites. I should mention that I'm a big fan of single-origin pour-over brews, so the list is biased in favour of places which serve that well rather than the more traditional espresso drinks. All these have at least one location in North Seattle.

Neptune

I'm here a lot because it's the only one on this list I don't need to get on the freeway for. The decor is a bit sparse, but they're quite friendly here and serve a mean pour-over. They've been known to even serve syphon-brewed coffee, which is always a treat to watch when they fire up the bunsen burner. The proprietor also runs tehcoffee.com, a (Heroku-hosted) site for competition-level baristas can rank and review others. The Greenwood neighborhood is great too; lots of lunch options and bookstores to browse. Bonus fact: Allie, the inventor of the mythical Alot beast has a blend here named after her.

Milstead & Co.

This one has only opened relatively recently, but it's made a splash for a number of things, not the least of which is the spectacular waterfront view from under the Aurora bridge. The selection here is excellent; this is the only place on this list that serves beans from multiple roasters, and they offer plenty of variety in brewing methods. The interior is spacious with high ceilings; it's very inviting. Mr. Milstead himself is usually around, and his excitement about his craft is infectious and obvious once you get him talking. I'd be here all the time if it weren't the furthest (south) from where I live.

Zoka

This one is my go-to place. I often run into people I know here, (and sometimes people I don't), and we hold the Seattle Clojure group meetings here as well. The University District location in particular works well for meetings just because it's so spacious; I don't think I've ever seen a larger coffee shop. It's a popular place for students to study, so it's pretty quiet in the back. It can be crowded at times though, probably based on the school schedule. They've got a great selection sourced from all over the world that rotates every so often. It's always fun to read their blog posts about traveling to visit the growers. The "Zoka bars" they serve here are a ridiculously rich treat. The decor here is very warm and comfortable with wood paneling everywhere.

Trabant

I don't come here often because the parking situation is awful, but if you are in the area (or taking the bus), definitely check out Trabant. They've got good chai, but the thing that keeps bringing me back here is the Clover machine. This contraption is capable of extracting rich flavours out of a bean like nothing else. Unfortunately Starbucks purchased the company that sold them, so there aren't very many places with Clovers that offer good beans. (As the owner of Trabant said about the acquisition, "wearing Air Jordans doesn't make you play like Michael Jordan.") Anyway, the downtown Trabant is very spare, minimalist, and quiet, while the one in the University district (pictured) is bustling and quirky.

Honorable Mentions

Vivace: I'm of the opinion that Vivace has the best espresso in Seattle. It's too far for me to visit regularly, but members of the famed Seattle Ruby Brigade can often be found working here in the afternoon.
Seattle Coffee Works: I've only been here a couple times, but I was floored by their Yirgacheffe both times. They serve their pour-overs at a separate "slow bar" where the baristas are happy to geek out over their creations if you show interest. Again, this one's too far away (Pike St.) to make me a regular.
Green Bean: This one is actually a non-profit which supports a number of other non-profits on a rotating basis. The staff here are really friendly. I like to come here early and get breakfast with my family since they've got a kids' corner.
Chocolati: I don't even come here for their coffee, but their hot chocolate is superb. They've also got great truffles. The Greenwood location is a converted house that offers plenty of space to work, while the Greenlake location is a lot smaller but has nice views of the water.

in which version two seems a treacherous stage

2011-12-01T07:15:11Z

I recently got back from the Clojure Conj conference in Raleigh. The sessions were great, but the main highlight for me was meeting with the developers of Cake (an alternate Clojure build tool) and discussing how we could collaborate on the future of build tools in Clojure. As was announced elsewhere, we will be taking some features from Cake and merging them into Leiningen 2.0 as well as just having more hackers involved in development efforts.

The development of Leiningen 1.x pretty much just fell out of the usage patterns we saw during my time at Sonian as an early adopter of Clojure. We used Maven for eight months, tried to make it work, and then took our experience from the pain we saw there to Leiningen. Some features (especially checkout dependencies) arose directly from finding certain operations with multi-module Maven builds extremely cumbersome. But in general nearly everything about Leiningen so far has been obvious. I wouldn't say it practically wrote itself, but once the central model of a project map and resolving/applying tasks from defns was established, the only really tricky thing was being strict about establishing a narrow scope and knowing when to avoid adding features.

"Version 2" is the most dangerous phase in a software project's life.

— Chas Emerick

So now we're digging into the question of what bigger-picture changes could be made to improve Leiningen if we leave backwards-compatibility out of it. The biggest improvement we've seen implemented so far is switching over dependency resolution to the Aether library via Chas Emerick's Pomegranate library. Aether is a library that contains all the dependency management features from Maven extracted into an independent module—basically designed with exactly Leiningen's use case in mind.

But there's more coming. Cake has had the ability to run project code in the same process as the build tool, significantly speeding up many operations, which is in the process of being ported over to Leiningen. I've been working on explicitly delimiting the parts of Leiningen that comprise its public API as part of a separate "leiningen-core" library which other tools can depend upon. We're looking at adding something like "profiles" do bundle up configuration sets which can be activated in given contexts.

With all these ideas flying around I hope we can buckle down and get some solid design discussions resulting in well-factored features. If you've been looking to contribute, now is a great time. Join the #leiningen channel on Freenode and hop on the mailing list.

in which the lesser-known are brought to the forefront

2011-10-28T17:21:54Z

In my time hacking in Clojure I've found a bunch of under-appreciated Clojure libraries of which I'm rather fond. I thought it'd be helpful to share, so here they are:

lein-multi

As support for Clojure 1.3 becomes widespread, it becomes more important for projects to think about backwards compatibility. You can specify a range of versions for Clojure in project.clj with [org.clojure/clojure "[1.2.0,1.3.0]"], but when you're developing, it will just find the latest and pull that in, so it's easy for dependence on 1.3-specific features to sneak in.

Enter lein-multi, a plugin for running arbitrary tasks against multiple dependency sets. It's most commonly used with the test task, but it's a general higher-order task that can be applied to any other just as well. Seeing lein-multi in someone's project.clj is a good indicator that they take backwards-compatibility seriously.

[lein-multi "1.0.0"]

clj-stacktrace

The most common complaint I hear about developing in Clojure is the fact that its error messages often obscure the true cause of the problem. While practice helps here, it's still true that there's a lot of irrelevant detail here that can overwhelm the trained eye. Mark McGranaghan's clj-stacktrace library does a great job at summarizing stack traces by aligning each frame, making the namespace distinct from the function name, and even coloring frames differently based on whether they come from Clojure, Java, or user code. The next version should let you filter out frames that are deemed irrelevant on a per-project basis.

The web framework Noir uses clj-stacktrace for its HTML error pages in development mode. To use it in your project's repl as well as in clojure.test error reporting, place this code in ~/.lein/init.clj after running $ lein plugin install clj-stacktrace 0.2.4:

(require 'leiningen.hooks.clj-stacktrace-test)

(def settings {:repl-options [:init (require 'clj-stacktrace.repl)
                              :caught 'clj-stacktrace.repl/pst+]})

slingshot

Another common question you hear is "How do I generate custom exception classes?". It's awkward and somewhat un-idiomatic to do this, so people generally try to get by with the Exception classes that ship with the JDK. But why shouldn't exceptions get the same level of dynamicity and flexibility that Clojure affords other data types?

This is basically the question Slingshot addresses. It provides enhanced try+ and throw+ counterparts to the built-in error mechanisms that let you throw arbitrary data types like maps. Then rather than dispatching in your catch blocks based on class, you can use arbitrary predicates. You can even perform destructuring on maps that are thrown. It's a big step up in expressiveness:

(defn asplode! []
  (throw+ {:bad? true :tachyon-level 21}))

(defn ignorable? [e]
  (and (:silent e) (not (:fatal? e))))

(try+ (asplode!)
  (catch ignorable? _)
  (catch :bad? e
    (log/warn "Bummer dude!" e))
  (catch :fatal? {:keys [exit-code]}
    (System/exit exit-code)))

[slingshot "0.8.0"]

lein-difftest

If you've ever written a test where you expect two lengthy data structures to be equal, you'll remember how annoying it is to try to compare the failure message where "expected" and "actual" are each spat out on a single line and you're supposed to try to hunt down the difference. Using lein difftest makes it easy:

It also uses clj-stacktrace to report errors. This one is better off installed as a user-level plugin since you're likely to want to be able to use the difftest task across all your projects:

$ lein plugin install lein-difftest 1.3.7

robert-bruce

From the do-one-thing-and-do-it-well department, we have Robert Bruce, which concerns itself only with determinedly retrying a given function. It provides all the options you could imagine for how to perform the retries, including pausing between them (with exponential decay on backoffs), retry limits, callbacks on failure, and so on.

[robert/bruce "0.7.1"]

Happy hacking!

Do you have a favourite that I've missed here? Leave a comment about it. Remember that the version numbers provided here are current as of the time of this writing but may be outdated when you read this.

in which version two turns the inside out

2011-09-28T04:12:54Z

I just got back last week from the Strange Loop conference in St. Louis. The conference showcased some really quality keynotes and session talks, but I had been invited to give a workshop teaching Emacs, which was a blast. The workshop was three hours long, but I still felt like I was zooming through. My outline is available, though it may not be all that useful unless you were there.

The talk focused on the task of crafting your own set of dotfiles. Naturally I did talk some about the Emacs Starter Kit, though my approach there has changed fairly radically. I've been developing version 2 of the Starter Kit since the beginning of the summer, but now I think it's time to make the release official and give it wider exposure.

The Emacs Starter Kit, version 2

Version 2 of the Starter Kit turns things inside out. The original version was very prescriptive; you used it by cloning a git repo to use as your entire ~/.emacs.d directory, and you fit your own customizations inside the structure it provided. This was mostly due to the fact that when it was released the packaging story for elisp was very immature. Use of package.el was not very widespread, so the Starter Kit had to include a copy as well as bundle a bunch of libraries that had not been packaged yet. As it turned out, I think the Starter Kit had a role in helping popularize package.el.

A lot has changed since 2008—primarily that package.el has been included in Emacs and Marmalade has picked up lots of steam, becoming the de-facto community package source with over 250 packages. It's now feasible for the Starter Kit to be distributed as just another package, leaving you free to structure your dotfiles however you like. All it takes is setting yourself up to get packages from Marmalade:

(require 'package)
(add-to-list 'package-archives
             '("marmalade" . "http://marmalade-repo.org/packages/"))
(package-initialize)

Place that in ~/.emacs.d/init.el and evaluate it with M-x eval-buffer. From there you can run M-x package-refresh-contents to pull in the latest package lists (analogous to apt-get update) followed by M-x package-install starter-kit. And then you're set.

However, it's unlikely that the starter-kit is the only package you're interested in. Rather than picking packages by hand, I'm a fan of keeping a list of all the packages you use in your dotfiles. That way when you move to a new machine, you won't have to remember what packages you've come to depend upon; checking out your dotfiles on a new machine will pull in everything you've specified. Here's my list along with some code to handle it:

(when (not package-archive-contents)
  (package-refresh-contents))

(defvar my-packages '(starter-kit starter-kit-lisp starter-kit-eshell
                                  starter-kit-bindings scpaste
                                  clojure-mode clojure-test-mode
                                  markdown-mode yaml-mode tuareg
                                  marmalade oddmuse scpaste))

(dolist (p my-packages)
  (when (not (package-installed-p p))
    (package-install p)))

What's Changed

Being a 2.0 release, you can't expect 100% backwards compatibility with the old Starter Kit. In particular, it is much more modular. The base of the Starter Kit simply gives you better default settings and a few new commands. There are separate modules supporting various languages like Ruby, Lisp (Clojure, Emacs Lisp, Scheme, and Common Lisp), and Javascript, though right now the Lisp one is the most polished. If anyone is interested in helping maintain alternate language modules, please contact me on Github. Key bindings have also been spun out into their own module since they don't follow Emacs binding conventions strictly.

There is also no more elpa-to-submit directory; if you rely on packages that aren't on Marmalade yet, it's up to you to either check them into your own dotfiles or better yet upload them yourself. There's also no more explicit recentf support since a new feature called ido-use-virtual-buffers allows recent files to show up in the regular C-x b buffer switcher, simplifying things. The addition of the ido-ubiquitous package makes it so regular completion gets enhanced with ido magic, so functionality like ido-imenu is no longer needed.

The other breakage is that the Starter Kit is now intended only for use with Emacs 24 onward. Yes, technically Emacs 24 has not yet been released, but it's in active pretest and is readily available for most platforms.[1]. I've been using it since 23 was released and have had no trouble with it.

[1] It's not hard to compile from source, but precompiled versions are readily available for Debian-based systems, Mac OS X, and Windows.

in which static types are friends, not foes

2011-08-15T01:37:00Z

I'm a big fan of composability in user interfaces. Unix has traditionally been strong in this area with its culture of pipes and standard in/out, making it easy to chain together small tools with orthogonal purposes. Unfortunately this usually does not extend to GUI tools which often tend towards monolithic globs of functionality. There are a few exceptions, my favourites including xbindkeys and mpd.

The most versatile of these I've found has been dmenu, a graphical option chooser in the style of Emacs' ido-mode: it presents a list of options, and as you type it narrows to just the options which match the input that's been entered so far. I've built a number of tools on top of this including a music frontend to mpd and a script that allows me to perform a number of Skype actions from the keyboard.

The problem is that dmenu is pretty minimalistic compared to ido. The main annoyance to me is that it doesn't support flex matching—in other words, only exact matches are shown. In Emacs, ido lets me match a few characters against the front of the string and a few against the tail. Any input is accepted as long as it uniquely identifies one of the choices. Since dmenu is written in C I didn't exactly relish the idea of picking up skills in that language to add this functionality, so I wrote a replacement in OCaml instead:

let rec read_lines lines =
  try read_lines (read_line () :: lines)
  with End_of_file -> lines

let lines = read_lines []

let lines_matching pattern matched line =
  try let _ = Str.search_forward pattern line 0 in 
      line :: matched
    with Not_found -> matched

let escape = function
  | ' ' -> ".*"
  | c -> Char.escaped c

let pattern input =
  Str.regexp (String.concat "" (List.map escape input))

let matched input lines =
  List.fold_left (lines_matching (pattern input)) [] lines

let rec draw_matches matches =
  Graphics.open_graph " 1440x15";
  Graphics.set_window_title "erythrina";
  Graphics.draw_string (String.concat " | " matches)

let finish input lines =
  Graphics.close_graph ();
  match matched input lines with
    | f :: _ -> print_string f
    | [] -> ()

let butlast input =
  match List.rev input with
    | [] -> []
    | _ :: rest -> List.rev rest

let rec main input =
  draw_matches (matched input lines);
  match Graphics.read_key ()  with
    | (* enter *) '\r' -> finish input lines
    | (* escape *) '\027' -> Graphics.close_graph ()
    | (* backspace *) '\b' -> main (butlast input)
    | (* any other *) c -> main (List.append input [c])

let _ = main []

In 39 lines of OCaml I've achieved near feature-parity to the 700 lines of C that make up dmenu. The only missing features are tab-completion and font/color customization, though I have added the flex-matching feature mentioned above that dmenu lacks, which makes tab completion less important.

This is my second foray back into the land of static typing since university. (Supposedly I learned C++ in school, but I've long since mercifully forgotten it all.) I picked up Mirah last year and spent a little more time hacking in it earlier this year for an Android app. While Mirah is a huge improvement over Java, its type inference unfortunately only extends to locals, (a common shortcoming of most JVM-hosted type systems from what I gather) meaning if you tend to write small methods you end up specifying all your types anyway. OCaml on the other hand infers all types using Hindley Milner inference, allowing the types to be effectively invisible.

I've only spent a few days writing OCaml, but I've got to say I'm impressed. The type system stayed out of the way, only making a fuss when I'd clearly made an error. Pattern matching is a dream: you'll notice that all the conditionals in the code above come from match rather than if. It's fast, it's pleasantly interactive, (especially via tuareg and Emacs) and the executables produced by the compiler are tiny and quick to start, which is valuable for anyone who spends a lot of time on the JVM and is looking for something to fill those pesky gaps for which the JVM is admittedly lousy.

My only real complaints surround the fact that the standard library is slim and quirky. It throws exceptions in places you wouldn't expect, like hitting the end of the file while reading or searching for a regex when the text contains no match. The regex support is a bit odd, but that's not too surprising considering how old the language is.

Of course the standard library is augmented by a number of third-party libraries. It seems like up until recently it's been assumed that you'll use either apt-get as the main way to pull these in or run make install from source. As I've blogged about recently, apt-get is a bit of a drag for libraries that are obscure or change frequently. It looks like Oasis is making it easier to build projects while ODB is the beginning of a rich library dependency system, though it's still got a long way to go. My needs have been simple enough that I've been able to stick with the standard library for my current project, but it's great to see progress in this direction.

Anyway, it's always a bit disorienting to toss yourself into a new environment like this. But OCaml has proven to be quite a treat. I recommend starting with this OCaml Tutorial which also links off to many other helpful resources. Have fun!

in which an operating system makes trade-offs

2011-07-29T19:27:51Z

I've been a user of Debian's apt-get package manager for nearly as long as I've been serious about computers, and I love it. Going back to a system that lacked apt's depth and stability would be like having to write with my non-dominant hand or worse—going back to typing on a QWERTY layout. I have seamless access to everything I need[1], and it never breaks unless I do something crazy like add unsupported third-party repositories or run the unstable distribution. Who could ask for more? So I'm quite proud to see Leiningen getting packaged for Debian with lots of help from the Java Maintainers team.

It's interesting to get a look inside the sausage factory of packaging. The biggest hurdle for me as I was learning the ropes was that all the packaging introductions assume you are building a C program and have familiarity with makefiles. You can tell that the packaging culture has its roots in C, and packages from other runtimes feel a bit foreign.[2] Even more so with a language like Clojure... you're off into unexplored territory, blazing your own trail.

The biggest change from what I'm used to is the notion that only a single version of a package can be installed at a time. I think I understand the motivation here: it's crucial that when a security vulnerability is reported the affected packages and all those that depend on them have access to the fix. I'm used to developer-focused package managers that allow many versions to be installed side-by-side, but that places the burden of security updates on the developers using it since it's common for packages to depend on exact versions rather than allowing bugfix versions to sneak in. It's a lot more work to track down security issues, but this is not a big deal in the context of developer-centric systems since these particular users are less likely to mind having to pay attention to that sort of detail; it's what they're paid to do.

So there's a real tension here; end users want packages to just work and be safe without thinking about them while developers demand repeatability and control. Developers need the flexibility to choose exactly when they want to pull in new versions of libraries, while end users want things to hum along out of sight.

The problem is that the one-version-of-a-package-at-a-time policy doesn't always work in practice. Over time backwards compatibility is the exception rather than the rule; in many cases there simply is no substitute for having multiple versions of a package installed simultaneously. And apt's answer here is just to create a new package with the incompatible version number as part of the package name. There's no clue that these packages are related, and upgrades won't pick the new versions up. This is quite a shame, because I love everything else about apt. I look at complicated production deployment schemes and think to myself how much simpler things would be if deploying were just a matter of adding our internal apt repository and running apt-get install, but... versions.

Of course, these gripes are not really relevant to having Leiningen in Debian.[3] Users of Debian-based systems will be well-served by getting Leiningen through their OS, and once it's on their system, they're free to use it to solve their developer-centric problems. This plays to the strengths of each system, and everybody wins.

Update: Leiningen has been accepted into Debian.

[1] Except Emacs; that's the only thing I still build myself from source. Though http://emacs.naquadah.org looks like it delivers perfectly passable builds of 24 if you're not interested in any of the experimental branches.

[2] It's especially noticeable with Ruby and the JVM. Part of this is due to the fact that the JVM has only been free software for a relatively short period of time, and a lot of the existing culture around Java distribution goes against the grain of repeatable builds by ignoring source and shuttling binary bytecode around everywhere.

[3] And it's looking like it'll make it into Ubuntu 11.10 as well.

in which systems are captured

2011-06-29T15:42:38Z

When working on a project, I try to take care that a dev environment can be brought up with as little fuss as possible. This requires some discipline even with simple programs, but once you get into the realm of systems, more up-front effort is required for repeatability.

We've all been in the position of being the new hire on the team. On some teams you're lucky if you can get to actually running code on your machine in your first week. You think to yourself that it could surely be simplified, but once you're all set up there's not a lot of incentive to come back and make it easy for the next time.

It's easy enough to track language-level dependencies using Leiningen, Rubygems, or whatever is standard for your language, but very few large systems end there. You almost always have further external dependencies that are best handled by the system-level package manager, be they databases, message queues, or even your language runtime itself.

I've found the best way to make sure this stuff is kept up is to make it the status quo. At work nearly everyone develops in a VM, and when a new dependency is needed, instead of everyone having to hunt down and install it, it just gets added to the setup script. This reduces redundancy as well as helping to eliminate Works on My Machine issues.

Recently I've found Vagrant to be a really helpful tool for streamlining this process even further. Before I was using raw VirtualBox, which was all right for launching fresh VMs, though the interface was always a little awkward. Vagrant streamlines this greatly, making it trivial to configure, suspend, and rebuild VMs. It also helps keep the configuration of the VM alongside the project for which it's meant.

Vagrant allows you to provision new VMs with Chef. While this is pretty handy if you've already got recipes written, it's a bit intimidating as the mental model needed to operate Chef is quite baroque. But it's easy enough to bypass Chef and provide an arbitrary shell script as a provisioner. You need to take a bit of care to ensure the script is idempotent as it's much more useful if you can re-run it whenever the dependencies change, but it's a great way to get started.

There are a few gotchas. Vagrant is rather particular about what version of Virtualbox (and the VBox extension pack) you use, and reinstalling Virtualbox only makes it confused. The OSS edition isn't supported, but thankfully an apt repository is available:

$ sudo apt-add-repository "deb http://download.virtualbox.org/virtualbox/debian $(lsb_release -cs) contrib"
$ wget -q -O - http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc | sudo apt-key add -
$ sudo apt-get update && sudo apt-get install virtualbox-4.0
$ sudo gem install vagrant

On a related note, I've had had a couple people ask what a good project to get started with Clojure web programming would be. The most widely-used Clojure web app that I know of is Clojars, the community repository. But it's been pretty dormant recently; there are plenty of great features just waiting to be implemented, but getting it set up is hard work given all the moving parts.

Vagrant is perfect for smoothing this over. I've got a vagrant branch of clojars that makes it as simple as vagrant up (modulo a small issue with the nailgun scp server I'm still working out). So this should allow folks to contribute to the codebase more easily.

I've also helped out with Justin Lilly's Clojure/Emacs environment which provides a fully-configured Emacs 24 install intended to help facilitate our Seajure hackfests. Give it a try if you've had trouble configuring Emacs for Clojure hacking. And feel free to crib from these setups for automating your own projects.

in which radical simplification ensues

2011-05-17T04:02:23Z

While Emacs support has come a long way since I first got started with Clojure, I'm reminded every time I help newbies get set up that there are still way too many moving parts. This is in part due to the fact that things center around SLIME, a tool which was originally developed to work with Common Lisp. SLIME doesn't have stable releases; the developers expect everyone to run from CVS trunk (I wish I were joking), so I had to do some ad-hoc packaging just in order to prevent constant breakage. But synchronizing compatible versions between two separate package managers on two separate runtimes is a headache.

I've cooked up something much simpler for the next swank-clojure release:

[download video]

This makes it possible for a single piece of elisp to bootstrap an entire SLIME environment. It's also set up so it can be extensible—if a project like Slamhound wants to distribute an elisp library, it will be able to be loaded automatically upon connecting SLIME. The number of steps is reduced to three:

Install clojure-mode via Marmalade or from git.
lein plugin install swank-clojure 1.3.1
Invoke M-x clojure-jack-in from a project

The video uses swank-clojure 1.4.0-SNAPSHOT, which includes some new debugging features, but the most stable release (1.3.1 as noted above) will also work. Hopefully this eases a few headaches!

Update: Right now there are known issues with having multiple versions of swank-clojure present, both in lib/dev and ~/.lein/plugins, so check to make sure they aren't conflicting. This will be fixed in the next version of Leiningen. It's probably best to remove swank-clojure from :dev-dependencies in project.clj entirely and have each user who wants to use swank install it as a local plugin instead.

If you have issues, please post to the swank-clojure mailing list or open a github issue. Blog comments are a lousy place for bug reports.

in which a violent metaphor features prominently

2011-05-01T06:05:12Z

This past week I've had a lot of fun with Slamhound, my most recent project. Often times when writing Clojure, your namespace declarations are a pain to keep in order—you've got to specify all the namespaces you depend upon, and once you've got a lot of them in there, it's hard to keep them straight; often duplicates sneak in and things just get really messy. So Slamhound takes a source file, discards the namespace declaration, and tries to determine how to reconstruct it from the code alone. I had a lot of fun putting it together, so here's a little walk-through of how it works.

A big part of why it was so much fun is that the code is very functional. It takes a file as input, throws it into a pipeline, and spits out an answer. The only impure part is the reading of the file.[1] The pipeline is built in the slam.hound/reconstruct function:

(defn reconstruct [filename]
  (-> (io/reader filename)
      asplode
      regrow
      stitch-up))

But first a little back-story: Slamhound is named after the explosive device from William Gibson's novel Count Zero. At the start of the story, the main character Turner has an encounter with one, after which he spends three months in surgery while his skin and organs are regrown before he's stitched back together. Files in Slamhound go through the same process: asplode, re-grow, stitch-up. (I've quoted code selections inline, but for the complete listing a link is given for each phase.)

Phase 1: asplode

(defn asplode [rdr]
  (let [rdr (PushbackReader. rdr)
        ns-map (ns-to-map (read rdr))
        stripped-ns (apply dissoc ns-map [:use :require :import])
        body (take-while #(not= ::done %)
                         (repeatedly #(read rdr false ::done)))]
    [stripped-ns body]))

Like its counterpart in the book, the first phase is quick, simple, and messy. First we need to construct a PushbackReader in order for clojure.core/read to operate. This gives us the ns form as a list. Then in ns-to-map (not shown) we perform the translation into a map by splitting each clause (:use, :require, and :import) out, since the map form is a lot easier to operate on in during the regrow stage. But often times you have left-over clauses from earlier revisions, so we dissoc them out in order to start from scratch. Then we repeatedly read until EOF to get the body. The stripped ns and body are passed to the next phase.

Phase 2: regrow

(defn regrow
  ([[ns-map body]]
     (force pre-load)
     (regrow [ns-map body] nil))
  ([[ns-map body] last-missing]
     (if-let [{:keys [missing type]} (check-for-failure ns-map body)]
       (if (= last-missing missing)
         (throw (Exception. (str "Couldn't resolve " missing)))
         (recur [(grow-step missing type ns-map) body] missing))
       ns-map)))

The re-grow phase is where the pain-staking progress is made. The first arity forces the pre-load delay which ensures all the source is loaded for compilation purposes, then hands off to the second arity. There's a recursive loop here: we attempt to compile the body with the current ns-map in check-for-failure (see below.) If it fails, we analyze the problematic missing piece in grow-step, which searches for all possible candidates, sorts in disambiguate, and picks one. Then the compilation is retried by recurring.

Right now if the retry results in the same failure, it gives up, though there's no reason in the future it couldn't continue down the list of candidates. Of course, if compilation succeeds, the re-grow process is finished, and it returns the ns-map with the new clauses for the next phase, stitch-up.

But check-for-failure deserves special attention as it's where the meat of the process happens:

(defn check-for-failure [ns-map body]
  (let [sandbox-ns `slamhound.sandbox#
        ns-form (stitch/ns-from-map (assoc ns-map :name sandbox-ns))]
    (binding [*ns* (create-ns sandbox-ns)]
      (try
        (eval `(do ~ns-form ~@body nil))
        (catch Exception e
          (or (failure-details (.getMessage e))
              (throw e)))
        (finally
         (remove-ns (.name *ns*)))))))

Here we see that a unique sandbox namespace is created using auto-gensym. Next we reach ahead a bit into the next stitch-up phase for the ns-from-map function, which takes our map representation and turns it back into an executable ns declaration. Stepping into this sandbox namespace with binding, we attempt to eval the body. If it fails, we analyze the exception with failure-details. This tells us whether it was a :use, :require, or :import clause that caused the failure as well as the name of the missing class or var. These details are used by candidates to determine the ns-map for the next attempt.

(defmethod candidates :require [type missing]
  (for [n (all-ns)
        :when (= missing (last (.split (name (ns-name n)) "\\.")))]
    [(ns-name n) :as (symbol missing)]))

The candidates multimethod operates on each of the three clause types (:use/:require/:import), but they're all slight variations on the same theme. The :require version is given here: it simply loops through all the available namespaces and returns the ones which match the missing name from the compilation failure.

Phase 3: stitch up

(defn stitch-up [ns-map]
  (-> ns-map
      collapse-clauses
      sort-subclauses
      ns-from-map
      prettify))

Once we've got a successful compilation, we have what we need. All that's left is to turn it back from a map into code form, which is handled by the stitch phase. Technically only the the ns-from-map function we saw used above is necessary, but the ns declaration looks a lot nicer if you collapse all the clauses down first:

(:use [clojure.test :only [deftest]]
      [clojure.test :only [testing]]
      [clojure.test :only [is]]
      [clojure.test :only [are]])

;; after collapsing becomes:

(:use [clojure.test :only [deftest testing is are]])

Sorting subclauses doesn't hurt either, though it's strictly cosmetic. Finally we turn it back into code form and pretty-print it using clojure.pprint/pprint. And that's it!

1: Technically there's one exception to this; see if you can spot it.

in which secrets are kept

2011-04-01T03:03:55Z

Back when Leiningen was first launched it coincided with the launch of Clojars, the public community repository for Clojure projects. This worked out really well for Clojure as far as the timing was concerned—it allowed the ecosystem to grow quickly.

But Clojars doesn't work for everything; some situations call for libraries to be shared on a team without making them public. For cases like this it's necessary to publish to private repositories. This has been a really common question with Leiningen that hasn't had a good answer until recently; some folks bite the bullet and run lein install on every development machine while others set up shared static-file repositories maintained mostly by hand. At work we've nginx'd up a directory that gets deployed to from our Hudson (soon to be Jenkins) jobs.

But the 1.5.0 release of Leiningen has added the deploy task along with a deploy guide. Now you can deploy to private repositories like instances of Archiva and Nexus. Just configure your :repositories in project.clj with the URL and credentials for uploading:

  :repositories {"snapshots" {:url "http://blueant.com/archiva/snapshots"
                              :username "milgrim" :password "locative.1"}
                 "releases" "http://blueant.com/archiva/internal"}

If you're shy about checking passwords into project.clj, you can put them in ~/.lein/init.clj as mentioned in the deploy guide. Note that the naming is significant; if you are deploying a SNAPSHOT version, it will go to the "snapshots" repository, while stable versions will go to the "releases" repository. Hopefully this is useful for teams collaborating on multiple projects in private.

in which the local meeting celebrates a year

2011-03-01T05:08:38Z

This month marks the first anniversary of Seajure, the Seattle Clojure group. I've had a lot of fun conducting these meetings. It sounds like our group is a little different from the way most people run things, so I thought I'd write a bit about what has worked for us.

One thing I knew I wanted from the start was for meetings to be very code-centric. The Seattle Ruby group used to have talks regularly, and they found that in that kind of situation after a few months things usually fall into patterns of just having the same person present over and over.

So we focus on code instead. We try to come up with a small project that's somewhat useful on its own but not too ambitious to build in a couple hours. Everyone logs into a shared tmux session running Emacs and can all collaborate on the code as it's being written. Of course still only one person can type at a time, but it's a lot less hassle than having to set up a projector, and it's much more interactive.

This only really works because we have a small group; with larger attendance levels the group may need to be divided somehow. We have swarm-coding sessions for only about half of our meetings when we have a good idea for a project. Other times we'll just discuss new features of Clojure or specific tools; one meeting we had an Emacs workshop to get people started. We also spend about half an hour starting out just chatting and making sure everyone has been introduced.

Normally I stress the importance of keeping the same location consistently since people have an easier time planning for it if it doesn't shift around, but the coffee shop in which we hold our meetings is closing early for winter hours, so we're in the University District Trabant for our March meeting. If you're in the Seattle area, join the join the mailing list and come on by!

in which a peek is given, sneakily

2011-01-31T07:17:52Z

Here's a little glimpse of something I've been cooking up:

I've hinted before about getting interested in mobile development for Android, but this past year Leiningen has really snatched up the bulk of my hacking time. Now that it's matured and stabilized a bit I've had a chance to take another look at Mirah, a fringe JVM language that's perfect for Android by virtue of bringing along zero additional runtime baggage. Ferrante is my first nontrivial app, and it's really been fun to piece together, blazing my own trail.

It's still got some spots where rough around the edges, but there's enough to poke at for the intrepid developer or curious beta tester. Source is on Github. I'll be sure to post more as I get things figured out; with a little more work I think using Mirah for mobile development could really be a lot of fun.

in which the author sighs and realizes he publishes very few posts which do not contain the word package

2010-12-30T16:32:42Z

As someone who maintains a fair amount of Emacs libraries, I've long been in search of improvements to the release process. I've lobbied for including package.el in Emacs itself, which finally happened over the summer. But political concerns interfered a bit, and it is only configured to point to the FSF package repository out of the box. This repository only accepts code for which copyright has been granted to the Free Software Foundation. At the time of this writing it has only six packages in it compared to ELPA's 130, making it somewhat less useful than it could be. For a number of reasons[1] the copyright assignment policy doesn't work for the packages I maintain, so this was a bit disappointing to me.

On the other hand, I was able to add support for multiple package sources to package.el before it was included in Emacs. This allows users to add alternate third-party sources like the original ELPA repository. I should note here that there's been some confusion regarding the difference between package.el and ELPA which I must admit to having helped spread at one point. To be precise, package.el is the package manager, while ELPA is the original package source from which most packages so far have been hosted. The term "ELPA" is often mistakenly used to refer to package.el because for a long time package.el was hard-coded to only download from ELPA, but now that it supports multiple sources it's important to make a distinction between them.

Unfortunately ELPA is still manually curated with package submission over email, so it can take weeks or even months for new versions of my libraries to become available. I've started my own package source at http://repo.technomancy.us/emacs, but it turns out maintaining a package source consisting of a bunch of static files is not a lot of fun and can be error prone. Luckily Nathan Wizenbaum has cooked up Marmalade, a community package source that allows users to upload their own packages much like Clojars or Rubygems.org. While it hasn't seen much use yet, it's quite promising as a way for elisp authors to get their code out to users. The following snippet will add Marmalade as a repository:

(require 'package)
(add-to-list 'package-archives
             '("marmalade" . 
               "http://marmalade-repo.org/packages/") t)
(package-initialize)

You'll need to run M-x package-refresh-contents manually to download the latest package list. If you are using Emacs 23 you can download a compatible package.el. The version from tromey.com doesn't support multiple archive sources.

Moving forward I'm planning on uploading the packages I maintain over to Marmalade. It offers a much faster turnaround time for updates than the old system of submitting by email to ELPA. I've already uploaded my Clojure libraries and am working on a version of the Emacs Starter Kit that is structured as a set of packages in order to be more modular. (The new Starter Kit only targets Emacs 24 and up, so at this point it's intended for the adventurous.) Happy hacking!

[1] The main reason being that I could never bring myself to answer the question "can you apply this patch?" with "well that depends, do you have a fax machine?".

in which the perils of the gilardi scenario are overcome

2010-11-10T05:43:43Z

There's an interesting gotcha that's bitten me a few times while exploring the fringes of Clojure. The eval function is not often used, but there are some cases when it's justifiable. It comes up while writing Leiningen plugins since code needs to be constructed in the context of Leiningen and run in the context of a project.

The Gilardi Scenario refers to a case where you want to evaluate some code that both loads in a new var and refers to that var in the same piece of code:

(eval `(when ~(:some-set project)
         (require '~'clojure.set)
         (clojure.set/difference
          ~(:some-set project) #{:bad-key})))

java.lang.ClassNotFoundException: clojure.set

The problem is that Clojure has to compile this whole block of code as a single unit. At compile time, the clojure.set namespace has not been loaded yet, so the attempt to look up the difference var is doomed to failure.

In Clojure 1.1, the compiler introduced special handling to work around this. If the top-level form being evaled is a do, then it will be broken up and each subform will be evaled separately. Unfortunately this only goes so far; it only applies to top-level dos, so even though our when above macro-expands to a form that contains do, the compiler can't apply its special-case.

Another way around it is to look up the var at runtime. The ns-resolve function works well for that.

(eval `(when ~(:some-set project)
         (require '~'clojure.set)
         ((ns-resolve '~'clojure.set '~'difference)
          ~(:some-set project) #{:bad-key})))

This delays the lookup of clojure.set/difference to runtime, incurring a small penalty but tidily avoiding the Gilardi scenario. It also looks a bit ugly due to the quote-unquote-quote notation. This is necessary inside a backtick since symbols are fully-qualified to avoid accidental aliasing existing names. I'm not sure if this is an intentional design decision of Clojure, but it's certainly a sign that you're off the beaten path and into possibly-inadvisable territory.

Leiningen used to encourage the ns-resolve solution above since it needed to support eval of arbitrary forms from plugins. Even if your form had a do at the top-level to invoke the compiler's special-case, you could easily have situations where add-hook has been used to wrap the form in enclosing forms, making your do nested deeply where the compiler wouldn't split it out.

(defn get-readable-form [java project form init]
  (let [cp (str (.getClasspath (.getCommandLine java)))
        form `(do ~init
                  (def ~'classpath ~cp)
                  (set! ~'*warn-on-reflection*
                        ~(:warn-on-reflection project))
                  ~form)]
    (prn-str form)))

But the latest version of Leiningen makes it work without ns-resolve. Even though eval-in-project accepts any form to eval, it splices it into an existing do form that places all requires (the init argument in the function above) into the top-level of the do where the compiler can work its magic. So if you've got a namespace that your form is going to depend upon, just pass in '(require 'my.ns) as the init arg to eval-in-project to deftly maneuver your way through the Gilardi Scenario.

in which the conjference is recapitulated

2010-10-31T05:15:28Z

I got back last night from a week in BAMA for the Clojure Conj conference. The consistent quality of the talks surpassed any other conference I've attended so far, and there was a palpable high-energy air of potential with so many focused hackers congregating for the first time. The only downside was that the breakneck pace of fitting all the talks into two days left little time for ad-hoc chatter and hackfests, something I've greatly enjoyed at Ruby conferences.

I got a chance to speak on Leiningen. [slides] I opened with basic usage followed by a few little-known but useful features. After that I took some time to explain how Leiningen tasks are written and extended. I spent a little more time on this since it's something that folks have asked about recently. The gist I tried to get across is that it's best to structure core concepts like tasks around functions rather than one-off mechanisms like macros or custom deftypes if at all possible as it encourages composability and is instantly recognizable to users at a glance.

It turned out the emphasis on functions in order to support composability was a common theme that everyone seemed to come back around to, most notably Mark McGranahan on Ring and Stuart Halloway on simplicity in general.

Near the end of the conference Rich gave a self-professed non-technical rant on the simple subject of thinking hard about hard problems. He said that one of the things he's most thankful of in his life is the opportunity to spend years thinking about a few problems. The idea that you could think about a hard problem for months on end, in what he termed "hammock time", is something most people (myself included) just hadn't ever considered.

When I met him after the Emerging Languages conference earlier this year, he mentioned how the notion of linear version numbers no longer meshes with the reality of distributed development. On Clojars there's a notion of non-canonical versions being distributed under a separate group, which makes Maven unable to weed out duplicates. But if they were kept under the same group, it would have to use version numbers to distinguish between the official version and the patched one-offs. Even in a non-forked project you have branches from which you may wish to publish artifacts. Being linear, version numbers are really not well-suited for this. They are lexographically sorted, which means there's no way to denote that 1.3.0-alpha3 may be newer than 1.3.0-par2. Dependency management needs to accept the fact that versions branch into trees. But representing it as such is really only the first step; the other tricky part is coming up with a strategy to choose a branch and how to decide between different dependencies that both may request different branches of a third dependency. It's a juicy problem that was also mentioned a few times at the Conj, and it may be the kind of "hammock time" problem to keep in the back of my mind for a few months or years.