Technomancy

in which we watch while a veritable tower of babel is constructed

2010-07-24T22:45:24Z

This week I was lucky enough to attend the Emerging Languages conference, a special event nestled snugly in a corner of O'Reilly's OSCON open source conference. Emerging Languages brought together language designers and implementors together to share the things that made their languages unique and to cross-pollinate ideas.

The presentations covered a wide variety of languages. The talk on Go was interesting in that it wasn't so much about Go as about the historical heritage of Go and the languages that led to it. A lot of the interesting ideas there came out of Tony Hoare's Communicating Sequential Processes paper, which looks fascinating. The talk on Frink was a delightful romp through the esoteric and very human world of unit calculation, though the licensing issues surrounding that language rule it out for most uses. It's been a while since I've done web work, but the talk on CoffeeScript made me hope that I never have to write another line of Javascript. I also got to see Charlie present on Mirah, formerly Duby, which I've posted about before: a language that gives you low-level bytecode-equivalent output to Java but reduces the pain/verbosity by offering a more reasonable syntax and type inference.

The second day started off strong with an engaging demo of the visual Kodu language. It's unique in that it's designed to run on XBox consoles and can be programmed entirely through the controller by manipulating icons. Once again there are licensing issues and it won't run on anything but Windows or an XBox, but in this case the developers are actively working towards remedying the problem. They have clearly put a lot of thought and research into keeping it engaging especially for kids.

After this Rich Hickey presented on a Clojure feature tentatively called Pods, which are the new name for an experiment he had discussed much earlier called cells. The gist is that while transients can be a boon to performance, they introduce mutability (albeit very constrained mutability) outside the reference model. Pods separate out a very clear reference policy where you're always dealing with persistent values coming in and out, and the change happens isolated inside the pod. (I posted a link to the above photo in the Clojure IRC channel, which caused it to erupt in cries of "what are pods?" and "where's the documentation for this?"—a reminder that there are many Clojurians who feel the need to constantly stay abreast of every latest change no matter how recent.)

Another highlight of the second day was the talk on Factor, a modern cousin to Forth with an excellent compiler and nice tooling. Factor's been on my radar for a while since stack-based languages really sound like an interesting twist to language design, and everything I read about their compiler seems to indicate it's very cutting-edge and well-designed. Factor was also the only language presented I want to learn that isn't hosted; that is, it compiles straight to the metal. The demo focused on showing some the ways that Factor retains an astonishing amount of flexibility and dynamicity even though it compiles to very fast machine code. The Emacs integration via FUEL also impressed me.

There were many, many more languages presented; they came at such a rate that if you blinked you'd look up to see the presentation half-through already. On the whole this was helpful since it forced presenters to focus on a "hook" or two to get you interested enough to dig deeper rather than give an overview of features which could easily be read from a web site, but such a wild ride left everyone with a minor case of mental whiplash.

It's been a while since I've attended an event that showcased this level of energy. I hope to look forward to attending Emerging Languages 2011.

in which readers are pointed elsewhere

2010-06-28T00:35:28Z

Michael Fogus, co-author of the excellent forthcoming book The Joy of Clojure, has posted a lengthy micro-interview with your humble author on his blog. Have a look!

in which is divulged more detail than is generally interesting

2010-05-11T21:20:17Z

So this post is ridiculously self-indulgent; you've been warned. I've mostly written it out here for my future self to look back on in a mix of amusement and embarassment. It's just a write-up of the tools I use daily in the style of the interviews at usesthis.com.

What hardware do you use?

I use a Thinkpad X200s primarily. It's hard to find a machine that's acceptably light (2.4 lbs/1.1 kg) but still has a decent resolution; most "small" laptops these days have a vertical resolution no greater than my phone, which would be embarrassing for me to use. The Thinkpad has the best keyboard of any laptop; I appreciate the crisp response. It also has an integrated trackpoint, which is nice in that you can move the pointer without taking your hands off the home row, but I try to avoid it as much as possible. I'm much more productive when using 100% keyboard commands. Sometimes I'll just pluck the trackpoint out entirely just to make sure I'm not using it without thinking.

I work from coffee shops frequently since I have a remote job, but when I'm in my code lab I use a standing desk with an external 23-inch monitor. I rotate the monitor to portrait orientation when I'm not remote pairing with tmux. I have a recliner next to the desk onto which I fall back to for a couple hours out of the day. While I'm standing I wear what has been affectionately termed "keyboard pants". It's a Kinesis Freestyle with kneepads attached. The kneepads are worn significantly above the knee—the idea is to allow myself to type in as neutral of a position as possible. My arms are relaxed in a downward position, and my wrists are totally straight. It takes some getting used to, but from an RSI perspective it's quite beneficial. It also just feels great not to be sitting all day.

I keep a Nexus One phone in my pocket. As far as I know it's the only major phone currently available that's designed to be rootable by the end user, and hence the only phone I am comfortable purchasing. As a nice bonus it also happens to be one of the best phones available, though it seems absurd to call it a phone. I hardly make any actual voice calls on it, though when I do it's VOIP calling through Sipdroid. I have CyanogenMod installed, which has a really nice auto-tether over USB.

And what about software?

I start with a boring old Ubuntu GNOME install. To my embarrassment I actually even use metacity, the default window manager. My secret sauce is devilspie, which is a rules engine for window placement and behaviour. I fullscreen and undecorate all my commonly-used programs and have certain programs only show up on a given virtual desktop. It's flexible enough to trick people into thinking I use a real WM. My other main desktop-level customization is xbindkeys, a little app that embeds a Scheme interpreter to configure bindings. This is nice because it decouples them from the window manager. The only part of GNOME I use on a regular basis is the panel. I like having CPU/network/memory graphs available so I can tell when my machine is hard at work. The panel wifi tool is pretty handy too.

Apart from that it's Emacs, Emacs, and Emacs. I am mostly able to maintain the illusion that Emacs is the only program I actually interact with regularly. I generally have four or five instances running at once which is uncommon; most Emacs users keep single instances with uptimes in the weeks range. I use this as a namespacing technique to keep real work separated from play/chat/mail. My staple modes are magit, clojure-mode, slime, paredit, org-mode, and erc. I used to waver between jabber.el or elim and leaving the embrace of Emacs for something like Pidgin, which was not pleasant at all. But now I'm pretty settled on bitlbee, which allows you to connect to your IM accounts via an IRC client.

I have all my Emacs config (accumulated over years of obsessive tweaking) bundled up as the Emacs Starter Kit. Lots of other developers use this as a base off which to build or just to steal ideas from, which I love. (It's a little funny to get bug reports for your own dotfiles.) I try to save all my other config in git as well in order to make it easy for me to get up and running quickly on a new machine.

The other key to maintaining the Emacs-all-the-time illusion is Conkeror. Emacs doesn't have a modern web browser in it yet, so this is an attempt to trick Mozilla Xulrunner into thinking it's Emacs. Apart from being implemented in JS instead of lisp, it's a very good approximation. The buffer switcher features completion that's far better than anything I've used in a mainstream browser, and its mouseless browsing support is excellent. It's also very stable as long as you don't have any garbage plugins like Adobe Flash installed. Its only flaw is being based on Gecko rather than Webkit. I've experimented a bit with mouseless extensions for Chromium, but the extension mechanism in that browser is crippled in comparison to what's possible in Mozilla, so while the speed boost was nice, this ended up being very frustrating in practice.

Update: I recant my complaints regarding the speed of Gecko in light of the latest Mozilla nightlies; they are absolutely zippy. I am no longer tempted to go back to Chromium. You can use Firefox rather than Xulrunner to launch Conkeror by doing bin/firefox -app ~/src/conkeror/application.ini.

Unfortunately I need to use skype for work. While I've been pretty impressed with its ability to use a very limited amount of bandwidth, it's quite unstable, and its UI is very awkward. Since it's not free software there's very little hope of this improving soon. The one thing that eases this pain is the skype plugin for gnome-do. It allows calls to be initiated from the keyboard. While I am a fan of the ideas behind gnome-do, this is the only thing I use it for since in general Emacs does a better job at the kinds of things it does.

What would your dream setup be?

I really wish I could buy a modern laptop with a 4:3 aspect ratio. I find the loss of vertical space in widescreens very annoying for anything other than watching movies. Apart from that the Thinkpad X200s is nearly everything I want in a laptop. In a perfect world I would have a battery with the weight of a 4-cell and the capacity of a 9-cell. Half the weight of my laptop is the battery.

I recently switched to using gmail in the browser, and while it's decent for everyday mail it is markedly inferior to gnus for mailing lists. I'm only using it because it syncs with my phone, and setting up offlineimap to work with gnus and gmail sounds like a lot of work. This is definitely the least-satisfying part of an all-Emacs setup, at least if you want to read mail on a mobile device.

I imagine if they ever get the senseboard working it will be a pretty sweet innovation. I could also use a HUD for my phone for when I'm in the car. Or possibly on my sunglasses for when I'm just walking around; you know... I'm not picky.

in which a year is reflected upon

2010-04-24T22:44:44Z

It's been a year since the PeepCode screencast on Clojure was released. While it's aged surprisingly well given the relative youth of the Clojure language (1.0 hadn't even been released at the time), there are a few things that could use some updates. I thought it would be helpful for me to step through Mire, the sample project that's built up in the screencast, and update it to reflect the changes that have since occurred in the Clojure ecosystem.

(defproject mire "0.13"
  :description "A multiuser text adventure game/learning project."
  :main mire.server
  :dependencies [[org.clojure/clojure "1.2.0-master-SNAPSHOT"]
                 [org.clojure/clojure-contrib "1.2.0-SNAPSHOT"]]
  :dev-dependencies [[leiningen/lein-swank "1.2.0-SNAPSHOT"]
                     [leiningen-run "0.3"]])

Commit 8dd4a5d: Moving to Leiningen

Probably the most obvious thing about the Mire project as it's seen in the screencast that shows its age is its ad-hoc build. (Step 12 of the screencast) At the time there weren't any good ways to build and distribute Clojure projects, so Mire simply contained a copy of Clojure and Contrib in its git repository and included a shell script to perform compilation and packaging. Apart from being just generally tacky, this actually caused the repository to bloat up by 16MB due to the fact that git is really lousy at storing binary files.

Kids these days have it so easy—Leiningen is generally used for managing Clojure projects now. I'm not going to go into detail about this here since it's covered well elsewhere, namely in the readme as well as the Full Disclojure screencast on project management. (Warning: flash video; a downloadable version is available if you create a Vimeo account.) There are other alternatives, but this is certainly the most straightforward. Leiningen gives you a basic skeleton to work from (lein new), handles dependencies specified in your project file, (lein deps) and creates jar files for you, (lein jar) among other things.

Commit 1326451b: The player and rooms namespaces: alter-var-root

The main thing that's going on here is replacing a non-toplevel def with a call to alter-var-root. It's never a good idea to call def from within a function. I tried to justify it by the fact that in this case it was a function that was only meant to be called at startup time as the docstring emphasized, (to initialize the rooms map) but it still felt wrong.

The problem is that the rooms map must be loaded from a bunch of files on disk, but the directory to load from isn't known until the -main function is called. So some mutability is called for here, but it's not really enough to justify a ref or an atom since once the server starts it will never change. In the updated version, alter-var-root replaces the def. It takes a var (mire.rooms/rooms in this case) along with a function to apply to the current value of the var and uses the return value as the new root value of the var. It's also possible to simulate change to a var using binding, but this only affects the current thread, and in this case we want the changes to be available to all threads.

Justifiable use of alter-var-root is rare, but startup-time mutability is one of those places it makes sense.

Commit 0dbd3402: The commands namespace: contrib shuffle

This one is pretty basic; it's mostly just adjusting to the new layout of Clojure Contrib. A number of namespaces got moved: clojure.contrib.duck-streams became clojure.contrib.io, clojure.contrib.str-utils became clojure.contrib.string, etc. In this case we switched from calling clojure.contrib.str-utils/str-join to clojure.contrib.string/join. Some of these namespaces (most of io and some of seq) will be promoted out of Contrib and into Clojure itself before the final 1.2.0 release.

The other thing worth noting here is that in the original version of Mire there were a lot of unqualified calls to use, which bring in all the vars from the specified namespace. It's a lot more idiomatic now to either switch to require with :as to alias the namespace to a short name or to stick with use but to limit the list of vars using the :only qualifier to avoid pulling every single thing in, which is the approach taken here. This may seem like a bit of up-front busywork, but makes it easier to track down dependencies between namespaces and fix them in cases like the Contrib upgrade where things get switched around.

Commit 0a0fa0fa: Upgrade server namespace: the resources directory

Here we see more careful use usage along with moving the room data files from data/rooms/ to resources/rooms/ following Leiningen conventions. The resources dir is meant for files that aren't code but are still used by the project, like HTML templates or data files like the rooms that Mire uses. They will get included in the jar file when the project is packaged.

Commit 43ce1f4e: The test suite: clojure.test and use-fixtures

In the Clojure 1.1 release the test-is library got promoted from Contrib into Clojure itself, so that's reflected here. We also move them to a separate test directory to reflect Leiningen convention. The tests for mire.rooms now uses the use-fixtures function, which is a great way to abstract out common setup to be shared among tests.

While OOP test frameworks use setup/teardown methods, the use-fixtures feature of clojure.test takes advantage of the fact that tests themselves are functions. A fixture is simply a function that takes a function argument. In our case the fixture just runs the function inside some bindings, but other common uses of fixtures are to create data on disk in a try/finally block and clean up when it finishes or to conditionally run the tests only if a given network service is accessible. There's a lot of flexibility with clojure.test fixtures.

What Isn't Here

There have been a lot more new features introduced to Clojure since Mire was released. We haven't covered transients, protocols, deftype, or the thrush combinators mostly because these aren't introductory-level topics, but also because they're the trendy new exciting topics and have been covered well elsewhere. But I hope this has been enough to modernize the Mire project and help extend the relevance of the screencast and associated codebase. Thanks for tuning in!

in which the maintainer's perspective is considered

2010-03-29T19:58:00Z

In the past few months I've had a number of contributions pour in for some of my projects. It's been really exciting to see this level of community involvement, and I try to do everything I can to encourage users to get involved in development. But I don't have a lot of free time these days—with two small kids I'm lucky to have a few hours a week to address incoming patches, much less work on new features.

There are a few things that could streamline the process a bit, so I thought I'd take some time to explain how these projects look from the maintainer's side and the things that contributors could do to make their patches easy to apply.

Coordinating

For simple stuff it's great just to get patches out of the blue. For more involved work, however, it's important to communicate so that parallel feature work doesn't interfere. For my projects, there are few enough people working on them that usually mentioning it on the mailing list is plenty. Larger projects will often track in-progress features using a ticket system, so in that case you'll probably want to do both. Then you can fork/checkout the project and get started.

Branches

The first thing to remember is to keep all your work in topic branches. Most people just commit directly to the master branch, but then it's trickier for them to keep in sync with upstream. Every time you pull in the latest changes it causes a merge commit. This needlessly clutters up the history and makes it difficult for maintainers to find the exact changes they are looking for. It gets worse if there are multiple unrelated features committed directly to master. Even once your changes are merged, it's likely that pulling changes after that point will still result in merge commits rather than fast-forwards simply because the same set of changes were applied in a different order. So keeping your work in a separate branch per feature/fix allows the master history to be kept clean.

When it's time for a maintainer to merge your branch, they have a choice to either rebase it on top of the current master and then commit it, maintaining a linear history, or merge it in to master with a single merge commit. The latter is usually done in the case of longer-lived branches where you want to be able to group together a whole series of commits under a single feature. It's a bit silly/noisy to leave a merge commit lying around from merging a branch with only one or two commits.

Cleanliness

I'm a little more picky about the silly little things than most maintainers; I prefer my projects to be free of trailing whitespace and to use 80-column width, standard indentation, and no tabs. It's easy enough to enforce this in Emacs:

(setq whitespace-style '(trailing lines space-before-tab
                          indentation space-after-tab)
      whitespace-line-column 80)

  ;; add hooks for every major mode you use
(add-hook 'clojure-mode-hook (lambda () (whitespace-mode 1)))

In an ideal world, projects would have great test coverage and bugfix patches would always include first a test that fails in order to highlight the problem the patch addresses, then the implementation that causes the test to pass. Many of my projects unfortunately fall into the tricky-to-test zone either by way of complex asynchronous I/O or UI-heavy pieces of code, making this ideal difficult to achieve. In any case, it gives maintainers much more confidence when applying a patch if they can use tests to verify it's behaving as intended. It just takes a big burden off the person performing the merge if they only have to manually verify that the test passes and makes sense rather than checking by hand that the implementation is behaving as expected.

I've found that in Clojure it's good to aim for function bodies of ten lines or less, modulo exception handling and logging. Naturally there will be places where this isn't practical, but it's good to keep this in mind and carefully examine longer functions to see if there would be a cleaner way to break things up. It should go without saying that using pure functions as much as possible and consolidating all I/O and mutation into a few places will also help a great deal to make things easier to read.

Sending it off

Personally I find it easiest to work with changes that are published to a remote git repository. Generally this means people create forks on Github, but there's no reason other options like Gitorious or even a self-hosted git repo wouldn't work. Github pull requests work well, although please do not send them without a message explaining what the changes do. Github has a "feature" whereby you can send pull requests to everyone with a fork of a project; this is a terrible, terrible thing that you should never use. Most of the people on that list forked the project months and months ago and forgot about it, so please don't spam them.

Some people prefer to just mail patches or attach patches to bug tracker tickets. It's easy to generate these patches by running git format-patch master from your topic branch.

And it's an unfortunate fact of life that some projects require copyright assignment before they can accept nontrivial patches, so check for that first. In some cases a paper form is even required, which can bring a further unexpected delay into the picture.

Commit Access

In some cases I know I'm just not able to give a project the attention it deserves and I volunteer certain contributors for the post of maintainer. Sometimes this is because I just can't give a project the attention it needs to keep up with incoming patches, and sometimes (as with Hugo Duncan and his excellent work on swank-clojure break) it's because I just don't want to be a bottleneck keeping their awesome work from getting in.

In these cases I like to ask that changes don't get merged to master until they've been reviewed by at least one person other than the author. If you're a committer that's applying patches someone else sent in, then you're doing the review, otherwise just try to get some help from IRC or the project's mailing list to review your patch before it makes it to master.

All that said, keep the patches coming! I love getting them.

in which telephone seems like entirely the wrong word

2010-02-07T21:35:24Z

After years of resisting phone ownership followed by a few years of owning a 2003-era Nokia dumbphone, I finally decided to make the jump when the Nexus One was announced. I've got a strong distaste for systems that place arbitrary restrictions on their users, and while the Android OS itself doesn't have any, many Android phones before the Nexus One have had the carriers interfere with the user's control over their phone, though not to the same offensive degree as Apple. The Nexus One is sold directly through Google without giving the carriers a chance to sully it.

Daily Usage

The screen is just brilliant, and the 800x480 resolution means everything is sharp. The OS is very smooth and responsive. Having spent so long on a system where the keyboard is king and the mouse is only used in exceptional cases, switching to the inverse situation on the phone is a bit odd, but not as disorienting as you'd expect. Like any handheld keyboard, the Nexus's is bad for writing anything longer than a tweet, but it's certainly no worse than the hardware keyboard on the old Zaurus I toy around with occasionally or the one on my Kindle. The built-in apps work great, and if you take the plunge to fully switch to GMail, it pretty much makes syncing your mailbox a solved problem.

There are a few nit picks like the color balance being a bit off on the camera, the way the face buttons don't trigger unless you push the upper half, and the fact that the built-in jabber client only supports a single account. But these are all pretty minor or easy to work around. The only thing that really bugs me about it is the fact that there's no ZeroConf implementation yet for the platform. But there are people working on this, so it's just a matter of time.

Oh, and using it to Talk?

It turns out you can also use the Nexus One to interface with the global legacy telephone system and make calls on that. Supposedly it has a very nice dual-mic noise suppression system for when you do this, but I've only made a handful of test calls so far. I got a data-only plan for half of what the regular voice+data plans go for and had planned to use Sipdroid to make VoIP calls with it, but then I realized I just don't make voice calls any more. So while there's a barely-noticeable delay with SIP calls over the 3G network, it really doesn't bother me. I also have used the Wired Tether app to hook up my laptop on the go and can confirm that calls via Skype sound fine too. So it's nice that T-Mobile isn't blocking that on a network level. They do seem to be the least-user-hostile of all the US carriers.

Hacking It

Of course once you get past the formalities, the question that matters to a hacker is how it feels to hack. I've only really gotten started with this, but my initial report is fairly positive. The official toolsets are either Eclipse or Ant, neither of which give me warm fuzzies, but luckily you can use Ant out of the box without getting exposed to the XML-editing ickiness.

Getting programs onto the device is pretty simple. Once your source is ready, you run ant debug, which produces a .apk package. You can use the adb (android debugger) program to load it up over USB, but since I keep leaving my USB cable various places, I prefer just scping it to my server and pointing my device's browser directly at the .apk. You can also use this to install dev builds of various apps before they have been uploaded to the Market.

The API seems pretty sane. A lot of thought has gone into the notion of supporting a single front-and-center application while allowing others to run in the background without impacting battery life and performance too severely. I've played a bit with the graphics tools, and they remind me a fair bit of Processing, which is a good thing. I haven't done much intricate UI work with a lot of buttons or menus, but that kind of stuff can be tedious even in the nicest environments.

Language of Choice

Since the Android VM is based on the JVM, there's a whole host of languages that can run on it. Unfortunately, Dalvik is no Hotspot—it currently lacks JIT, and the GC is merely serviceable rather than astoundingly good like Hotspot's. The lack of a good GC makes using persistent data structures a real drag since they generate a lot of ephemeral garbage, so Clojure is not a good choice. The lack of JIT coupled with CPUs that are comparatively low-powered means that while JRuby runs, it's not altogether pleasant, especially considering the blitz with which regular apps perform. I've been told there is some low-hanging fruit for improving performance on Android, so this is likely to improve to a degree. Rhino, Python, Lua, Scala, and others work, (including, I'm told, even some legacy languages like Java, if you can imagine that) but I decided to try the less-traveled route with something called Duby.

Duby is a language created by Charles Nutter, the head of the JRuby project. JRuby is an amazing feat in part because Ruby's object model is vastly different from what's natively available on the Java platform. By an astounding effort they've managed to put together a first-class Ruby implementation, but it does beg the question: what would a modern language look like that went with the grain of its host instead of violently against it? Duby is an attempt to answer that question.

The syntax of Duby is nearly identical to that of Ruby; it only adds type declarations to method definitions. Yes, that means it's statically-typed. While it has type inference, it's not Hindley-Milner-style, it's closer to Scala's. Locals get their types inferred, it's only arguments in method definitions that need hints. So far I keep forgetting this nearly every time I write a new method since it looks so close to Ruby otherwise, but I'm sure I'll get the hang of it. Closures are implemented, so you can iterate over collections with blocks. Duby is also unique in that it literally has no runtime—its literals translate directly to ArrayLists and HashMaps, so once you've compiled, the code is more or less identical to what the Java compiler would have output.

Update: Duby has (thankfully) been renamed Mirah.

Progress

So far I've only put together a couple toy apps: Hello World, and a graphics demo with a bouncing ball. Unfortunately, Duby is a very immature language, and it shows. Starting out I had to go to Charlie at nearly every turn with stack traces. Half the time it would be my fault, and half the time it would be something as-yet-unsupported by the compiler. But so far he's been able to turn around and bring in all the features I need, which has been quite amazing. I'm hoping to get a chance to dive into the compiler source myself and get to the point where I can add features I need with minimal guidance.

Adapting the Android build process to Duby was surprisingly easy. You redeclare the compile task to call the Duby compiler instead of javac, tell it to output its bytecode in the right directory, and the rest of it just falls into place.

My next plans are to add more interactivity to my graphics demo; I'd like to play with creating objects and applying motion rules to them; I hope to come up with something my two-year-old would get a kick out of. So far it's been a lot of fun and a great way to explore the capabilities of this remarkable device.

in which, were a title to be summarized from the content, it would be altogether too similar to many of the titles used for past articles, possibly to the point of indistinguishability

2010-01-31T18:10:28Z

Anyone who follows my exploits will have noticed I'm a tireless proponent of ELPA, the Emacs Lisp Package Archive. As a maintainer of several Elisp libraries, ELPA makes my life easier by helping me sidestep the boring problems of distribution and installation. You may not know that package.el, the software behind ELPA, has been submitted for inclusion in the next version of Emacs. I've taken up the task of getting it ready.

Including something like package.el into Emacs is a big job, and it's something that can only happen gradually. Emacs comes with a number of applications such as Org Mode and Gnus that are developed externally to Emacs and merged periodically into the main Emacs source tree. If they were to be redone as packages they could still be distributed with Emacs builds but kept out of the source tree. They could also be upgraded and installed/removed independently of Emacs' historically long release cycles.

If you've submitted packages to ELPA before, you know it's a process that could use some streamlining. Currently it's all done by email, and packages must be manually uploaded by a single maintainer before they appear to users. This has long been the biggest shortcoming of ELPA. I've written some additions (package-maint.el) that allow you to automate the maintenance of a package source. Basically you provide it with a list of git URLs, and it will check out each tagged version and create a package from it. Of course, that wouldn't be useful without giving clients the ability to get packages from multiple sources at once, which I also added to package.el.

If you maintain any Emacs packages of your own, please try out my changes to package.el. If you use any of my packages, try upgrading and adding my package source to your list.

(add-to-list 'package-archives
             '("technomancy" . "http://repo.technomancy.us/emacs/") t)

That way you'll get access to my updates as soon as they're tagged rather than waiting for them to be manually uploaded, though currently the latest versions of all my packages are in ELPA. Next steps are closer integration with Emacs in order to have packages installable on a system-wide level as well as a per-user level, prerelease version number support, and extraction of some built-in Emacs libraries as packages. Suggestions, bug reports, and patches welcome!

in which persistence proves a propitious property

2009-12-22T15:14:16Z

When you read about functional languages, one of the things that frequently comes up is the value of persistent data structures. If you come from an OOP background where persistent means "it gets saved to disk", this is a little confusing until you do a little digging to find out it's a different meaning of the word persistent; you discover that it just means the data structures are immutable. This is technically true; all persistent data structures are immutable. But this understanding is a little bit lacking—it doesn't really get at the meaning of persistent.

The point of persistence in this case is that future versions of the object in question can be created without changing either the value or the performance characteristics of the existing instance. So when you've got a vector that you want to work with, you can create another vector based on the original, but with a few new items added to it. In languages that provide persistent data structures this is done without copying; internally the portions of the vector that are the same use a shared structure. But there are some pseudo-persistent implementations that cheat; as you create more and more versions based on the original vector, the performance of the original degrades even though the value is preserved. This is avoided in true persistent implementations such as Clojure's.

The other important thing about understanding persistence is understanding what it's not. A new feature in Clojure 1.1 is the addition of transient data structures. Transients provide speed boosts in cases where you decide performance is more important than persistence by using mutable data structures in a controlled, thread-safe way. If you don't understand what persistence means then you might see the fact that they are mutable and use them as you would in an imperative language—but that's not what they're meant for! The key to understanding transients is not that they're mutable but that they're not persistent. The fact that they are mutable is an implementation detail; you should treat them like regular immutable data structures, you just shouldn't rely on their persistent qualities.

Focusing on immutability is focusing on the negative: what you can't do. Thinking in terms of persistence is focusing on the positive: there are a certain set of guarantees that we may rely on. If you decide in some cases to give up those guarantees for speed benefits, transients allow you to do that, but you shouldn't think of them as your old imperative friends you can alter as you wish.

in which projects may be more easily compiled and distributed

2009-11-25T22:28:10Z

So build tools have been a long-standing pain point when working on Clojure projects. Most projects have used JVM-centric tools like ant or maven, which are far more complicated than what you need to build Clojure, and on top of that must be configured with XML. It could be made to work, and in the case of projects with significant dependencies it was much better than doing it by hand, but it was a pain.

Last week I launched Leiningen, which is a project that brings a native build tool to Clojure. It seems to have people pretty excited about it, which leads me to believe I'm far from the only one who's been feeling the pain here. It's configured using a Clojure file called project.clj like the sample below.

(defproject leiningen "1.0.0-SNAPSHOT"
  :description "A build tool designed not to set your hair on fire."
  :main leiningen.core
  :dependencies [[org.clojure/clojure "1.1.0-alpha-SNAPSHOT"]
                 [org.clojure/clojure-contrib "1.0-SNAPSHOT"]
                 [ant/ant-launcher "1.6.2"]
                 [org.apache.maven/maven-ant-tasks "2.0.10"]]
  :dev-dependencies [[org.clojure/swank-clojure "1.0"]])

Testing

This one's pretty straightforward: lein test searches the test/ directory for namespaces and runs the tests in each.

Building

Running lein jar will compile your code and package it up as a .jar file. If you specify a :main class in your project config, this will be an executable jar with the class you specified. If you just want to compile your code, there's a compile task for that too. Alternatively if you want to build a standalone jar that includes all your project's dependencies for easier distribution, you can do that with lein uberjar.

Dependency Management

This is the fun part. Rather than tracking down and downloading all your dependency jars manually (or worse: checking them in to your source repository), Leiningen provides a deps task to automatically download them and place them in the lib/ directory. This uses some Maven code under the covers, but don't panic—you won't have to see a bit of XML unless you want to. It pulls from the central Maven repositories by default as well as from Clojure's nightly builds and http://clojars.org. Clojars has built-in search, but if you want to pull a library from the central repository, you can search for it on Jarvana. You can also specify development dependencies with the :dev-dependencies key for things that shouldn't be pulled in at production or included in the standalone jar; the syntax is the same.

Publishing with Clojars

By a stroke of luck I dropped hints about Leiningen the week before I made a public announcement only to find out that Alex Osborne was approaching the same problem from the other direction. He'd been cooking up a web site and service for hosting Clojure libraries independently of my work on Leiningen, and we started talking about ways to integrate the two. So now Leiningen makes it easy to publish jars publicly via Clojars so other projects can depend on them. The day after announcing Clojars, Alex was interviewed on InfoQ about the site and how it interacts with Leiningen, so I was thrilled to see both our projects getting exposure so quickly.

Coming Soon...

Soon after the 0.5.0 release, Alex along with Dan Larkin (of the wonderful Clojure JSON library fame) started diving in and adding features. ~~We've got a few more tasks on the plate targeted for a 1.0 release, but should be right around the corner.~~ Update: Leiningen 1.0 released. If you'd like to say hi or drop a line, there's a #leiningen channel on freenode as well as a mailing list. If you found this useful, you should take a look at the Clojure PeepCode screencast.

Logo by Loren Broach.

in which things are mapped, but also reduced

2009-11-13T20:29:30Z

A while ago Tim Bray had a project called the Wide Finder to collect implementations of a log parsing problem in different languages to see which ones made it easy to take advantage of massively-parallel hardware. The idea is to accept a web server log file and return statistics for which pages have been requested the most. Yesterday he posted a follow-up in Clojure using refs.

While his version is interesting for someone just getting into the language because it uses refs (probably the sexiest piece of the language), I think a map/reduce approach is a little more idiomatic since it can be done with no explicit state change.

I'll step through my implementation piece-by-piece. Note that it is naïve and could be optimized for speed at the expense of straightfowardness, especially with regard to reading from disk; my intent here is simply to explain the functional approach.

Update: Commenters have posted a version that totally smokes my agent-based implementation in terms of performance and simplicity. I'm leaving it up because I think it's a fun romp in the land of higher-order functions; if you think having higher-order functions means "I can pass a block to Array#map in Ruby" then you're in for a treat.

(ns wide-finder
  "A basic map/reduce approach to the wide finder using agents.
  Optimized for being idiomatic and readable rather than speed."
  (:use [clojure.contrib.duck-streams :only [reader]]))

Every Clojure file opens with a call to the ns macro. This defines a namespace (wide-finder) and states any dependencies it may have on other namespaces, in this case the reader function from the duck-streams contrib library. Omitting the :only clause would cause it to make all the duck-streams vars available in this namespace, but it's usually better to be explicit about what you need.

We'll start with the entry point. The find-widely function below takes a filename and a number of agents to work with. Agents can be thought of as independent asynchronous workers that share a thread pool and keep a single state value. They are initialized with the agent function that takes a starting state. We'll be using each agent to keep a map of pages to hit counts, so we map this function over a list of n empty maps generated by repeat to get our list of initialized agents.

(defn find-widely [filename n]
  (let [agents (map agent (repeat n {}))]
    (dorun (map #(send %1 count-line %2)
                (cycle agents)
                (line-seq (reader filename))))
    (apply await agents)
    (apply merge-with + (map deref agents))))

Once the agents are initialized, we send them work. Our line-seq sets up a lazy sequence of lines from the file. We map over this sequence together with an infinite loop of the agents we construct using cycle. The map function can loop over multiple sequences in parallel and will stop when the shorter one runs out, so this is just a way of pairing each line in the file with an agent in a round-robin fashion.

(dorun (map #(send %1 count-line %2)
                (cycle agents)
                (line-seq (reader filename))))

The code that does the mapping is the anonymous function #(send %1 count-line %2), which adds a call to the count-line function to the agent's internal queue. This could be written in more traditional lambda form as (fn [a line] (send a count-line line); the #() form is simply shorthand. The count-line function will be called with two arguments: the agent's current value and the next line in the seq. That function will return a new value for the agent.

(doseq [a agents] (await a))

Once all the work has been queued up, it's necessary to call await on each agent to make sure it has a chance to finish its queue. If we used map here, it would be a no-op, since map merely creates a lazy sequence; it does not actually perform the function calls until the value is needed. Since it's not in the tail-call position, the value would simply be discarded. Note that the same problem would occur in the previous call to map, but wrapping it in a dorun call forces the lazy seq to be evaluated.

(apply merge-with + (map deref agents))

Once that's done we can call deref on each agent to get its value. But this gives us n maps rather than a list of totals, so we need to merge the values. merge-with is a special case of reduce that assumes it works on a sequence of maps and merges key collisions using the provided function, in this case +. This gives us a map of page names to hit counts.

(defn count-line [counts line]
  (if-let [[_ hit] (re-find #"GET /(\d+) " line)]
    (update-in counts [hit] inc-or-init)
    counts))

Finally we have the function that actually performs the counting. As mentioned, it takes an agent's state (which is the current map of pages to hits) and a line from the log file. If the line matches the regex #"GET /(\d+) ", then we return an updated version of the counts map that increments the entry corresponding to the hit. The other interesting thing here is that if-let uses destructuring: by binding the return value of re-find to a vector form, it splits the value in two. The first element (the full matched string) is bound to the unused _ local, and the second element (the match group corresponding to the actual page path) is bound to hit.

(defn inc-or-init [i]
  (if i (inc i) 1))

The only piece left is the tiny inc-or-init function that increments the counter given it but treats nil as zero. This would be unnecessary if we could construct hash-maps with custom default values, which a perusal of the implementation of PersistentHashMap.java seems to indicate is supported, though it's not exposed anywhere through a Clojure function. This is the piece of the puzzle that I'm the least happy with, but it may be possible to eliminate it.

In any case, this is a simple example of how to break up a commonplace problem using a classic map/reduce strategy and immutable data structures. I have no idea how it compares in terms of performance to the other Wide Finder implementations since the logs I have for this site are not exactly the hundred-megabyte blockbuster kind of logs that ongoing enjoys, but the fact that it can be parallelized in twelve lines is a testament to the expressiveness of the language.

in which the effects of distance are reduced

2009-10-03T15:37:47Z

Since joining Sonian in April, I've been part of a distributed team that has grown to four Clojure programmers all living in different states of the US. While it's been great to be able to draw from a wide pool of talent, working with a distributed team has its challenges. While we had everyone in Seattle for a week we got to do lot of pair programming, but after everyone went back home we were left wondering how we could emulate that over-the-shoulder atmosphere without all being physically in one place.

We tried VNC, but it was simply too laggy. Then we tried shared screen sessions. This worked out pretty well. The defining characteristic here is that both participants see exactly the same thing. Only one can edit at a time, which is usually not a big deal when you're pairing. Since it's shared on a per-session basis rather than a per-document basis, it means you can share things like the woefully under-appreciated M-x erevision feature, which is fantastic for code reviews. But it still feels a bit restrictive and out of the regular groove. We also use 3G a lot, and unfortunately port forwarding is not permitted by AT&T, our ISP.

Then I discovered Rudel, an Emacs client for the Obby protocol. Years ago I'd started my own implementation of this called Ebby, but I wasn't aware of the more nuanced implications of the protocol, so my implementation was never very robust. Rudel, on the other hand, does a great job of allowing us to share documents among collaborators in real-time.

Now we use Rudel for our main solution and then fall back to screen whenever we want to do something that involves synchronizing the whole screen. While you can run shell or SLIME sessions in Rudel, it's a bit awkward since only the original publisher of the document can actually interact with the subprocess. The other person can enter commands, but they can't run them, since pressing enter will simply insert a newline. I have been playing with the idea of using the Obby chat functionality as a side-channel to perform these kinds of things, but I haven't written any code for it yet.

One thing you've really got to be careful with in this situation is communicating who's responsible for committing. Otherwise you could easily get conflicting commits. SIP works great for talking through this vocally, though unfortunately mostly at work we're using Skype, which has a fairly lousy client for GNU/Linux.

I'm working with Jan, the Rudel author, towards an 0.2 release (quite soon, it looks like) which should improve stability pretty significantly. At that point I should be able to bundle it up for ELPA, though if you're running an Emacs that's older than October 2009 you will still need to install the eieio library by hand.

Anyway, if you work on a distributed team, you should give it a shot. Until 0.2 is released, you'll have best luck with the development version:

bzr branch bzr://rudel.bzr.sourceforge.net/bzrroot/rudel/trunk rudel

Then in your Emacs config:

(load-file "/PATH/TO/RUDEL/rudel-loaddefs.el")

Once you've got a Gobby server running, M-x rudel-join-session should get you connected, and C-c c s will let you subscribe to a new document. Have fun!

in which it is time for something that is almost completely different

2009-08-31T21:24:34Z

Remember that funky leaf shape you used to draw with your ruler in math class in high school when you got bored? I can't be the only one who did this...

Right before _why the lucky stiff took himself off the grid I started reading up on his experiments with code and art, and he spoke pretty highly of the Processing language as a very approachable way to get into programming. As someone who got started coding in QBasic, I've always felt that the amount of effort needed to get started in most modern languages is quite a shame. Sure you can get going easily with text, but when you're new to it you want to see pictures drawn and colors fly around the screen. Processing restores that level of immediacy; if you want a line, you call the line function; there's no fussing with setting up an event loop or a menu bar or any of that nonsense.

The problem is that while Processing is a great toolkit for drawing, it's is actually a pretty poor language when it comes to computational expressivity. It's statically-typed without any sort of inference, and there are no first-class functions or associative data type literals. Iteration is only possible using the most primitive of C-style for loops, and there's a shameful level of statefulness in much of the core functionality... You're not going to miss these things if you're just starting out with programming, but as a seasoned hacker, going back to a language like that feels quite limiting.

Luckily Processing is implemented on the JVM, so any other JVM-hosted languages can call out to it trivially. My weapon of choice right now is Clojure, and there's an adapter library that fits the bill nicely. (A ruby bridge exists too.) I've been playing around with it and have started a sketchbook project containing each of the programs I've written so far.

The first thing you notice when you read up on Processing and take a look at some of the shorter sketches is that very simple ideas often yield interesting and subtle results. This image simply bounces a circle around the window while varying its opacity in a sine pattern. It's around thirty lines of Clojure, but you can see emergent weave patterns in it. This is a fairly common occurrence among Processing projects.

The canonical handbook by Reas and Fry has proven to an excellent introduction. It's geared towards beginner programmers, but the sections are broken up well so that it's easy to skip those on control and iteration while still getting a lot out of the more art-focused parts if you're coming from a programming background.

A nice bonus of being JVM-hosted is that it's possible to run Processing programs as applets (remember applets?) in the browser. I haven't gotten mine exported yet, but having the option to share easily makes it a lot more fun; I hope to have a sketchbook page uploaded soon. In the mean time it's on Github if you want to try it yourself.

Update: I've compiled all my sketches so far together for download. Depending on your OS you may be able to double-click on it; otherwise just launch it with java -jar sketchbook.jar. The chu sketch has trouble launching, and there's one bug where the sketches' window size is wrong upon launch, but you can adjust that.

in which a brotherhood gathers

2009-06-06T22:27:32Z

This week I had the good fortune to be in San Francisco for a meeting of the Bay Area Clojure Meetup. This meeting was a special event since Clojure's creator, Rich Hickey, was in town for the big JavaOne conference. We ended up packing out the room with sixty people, which made it probably the largest gathering of Clojure programmers ever.

The meeting began with a handful of so-called "lightning talks" (of various lengths) on all sorts of topics. A few highlights were Amit Rathore's Swarmiji clustering library and George Jahad's talk on decompiling Clojure with jdb, which showed tools to get a new perspective on how your code gets compiled.

After these Rich gave a talk on the recent work he's been doing on chunked sequences. Basically when a seq wraps a vector or vector-like collection, rather than every first/rest invocation breaking the vector down piece by piece, the seq can break a chunk of the vector off to work on all at once and keep its own internal counter that points to the current position in the chunk. The motivation here was that while you generally can rely on the seq abstraction to map over each element in a vector in a functional manner, sometimes performance concerns force you to use a more imperative-style loop instead. With the chunking in place, the map approach is even more performant than the less-idiomatic loop, so there's one fewer reason to stray from the functional approach. It's really encouraging to see that Rich is making this a priority—he wants to make sure that you never get penalized for doing things the elegant way.

After this Rich hosted a Q&A session, from which two main points stuck in my mind. He mentioned that before the 1.0 release, he asked what people thought would be a good set of short-term goals for the project, to which the response was overwhelmingly "move off SVN to git". Since the project was still fresh of a jump from SourceForge to Google Code, he was reluctant at first, but he seems to have been convinced that the distributed approach is what's best for the community. He promised us that the move will happen. He's currently experimenting with GitHub and making sure for himself that tool support is good enough.

The other question of particular interest was that of tests. Rich doesn't write tests for Clojure. The community has contributed a test suite, but it's kept in a separate repository and doesn't offer full coverage. Rich's opinion is that tests are for finding bugs, and that he is able to avoid bugs by reasoning out the problem up front, so it's not a wise use of his time for him to write his own tests, though he does appreciate and run the community-provided suite. This made me very nervous at first, but the next day I read Out of the Tarpit [PDF], a paper he recommended at the meeting. The paper makes a strong case for informal reasoning being a suitable substitute for test cases in codebases that exhibit referential transparency. While I still prefer TDD for its help in the design phase of coding, I'm much more ready to accept the notion that thinking through the problem up front and verifying it with manual tests is actually a feasible approach to getting working code in functional languages, though of course in situations where mutability is the norm it leads to disastrous results.

Update: Videos are up! (Unfortunately flash-only; sorry!)

in which are found tricks of the trade concerning clojure authorship

2009-05-27T17:35:46Z

So it turns out getting set up to write Clojure code can be a little tricky. There are a lot of disconnected tidbits about how folks have figured out how to configure things, but it can be a bit tricky to tell the difference between, "hey, this is how I finally got it to work" and "this is how you really should be doing it". I figure I know about as much about using Clojure with Emacs as anybody, so here's a run-through of how I've done my setup. There are a lot of moving parts, but bear with me; most of the installation is automated.

Spoiler Alert: Using SLIME with Clojure is now much easier than the instructions detailed below indicate. This page is left up as it provides some background that may be helpful, but if you just want to get going with SLIME, use ELPA.

The Pieces

ELPA: The Emacs Lisp Package Archive functions as a centralized store for Emacs libraries and provides automated installation and upgrades.
clojure-mode: This gets you syntax highlighting, indentation, and other basic goodies for editing .clj files.
SLIME: The Superior Lisp Interaction Mode for Emacs was originally written to support interacting with Common Lisp subprocesses from within Emacs, but it's been extended to work with other lisps.
swank-clojure: An adapter for SLIME that allows it to work with Clojure.
clojure-test-mode: This provides support for running Clojure tests from within Emacs buffers and seeing the results displayed inline.
paredit: Paredit auto-balances parentheses and other matched chars to make sure you don't end up with structurally invalid expressions.

Installation

The easiest way to get started is to grab ELPA. If you use Emacs Starter Kit you've already got ELPA. (If you're new to Emacs, you might want to use the Starter Kit anyway as a base for your own customizations; also check out the PeepCode screencast.) Use M-x package-list-packages to pull up the package list. Move down to clojure-mode and press i to mark it for installation, then press x to go.

Once it's installed you should be able to work with .clj files, and you may be happy with just this. It has rudimentary subprocess support with M-x run-lisp, which is good enough for many, including Rich Hickey, the creator of Clojure. But most of us find it much more convenient to interact more richly with a running Clojure process as we code.

Pressing M-x clojure-install will kick off the Clojure installation process. Once you choose a download location, it will download a number of repositories and compile Clojure itself, so it will take a few minutes. (It requires having git, Java 1.5+, and ant installed.) When it's done, it will configure SLIME and swank-clojure, and it will give you instructions on a few lines to add to your personal config (usually found in $HOME/.emacs.d/init.el) so it will work for future sessions. Deprecated in favour of similar functionality in swank-clojure.

Usage

Hitting M-x slime will launch a new Clojure session in a *slime-repl* buffer. You can also interact with the *inferior-lisp* buffer, but the slime-repl buffer provides a higher-level interface with a few extra niceties. The REPL works as you'd expect, but you can hit , to activate some shortcuts, the most useful being i to change the current namespace (with tab-completion) and restart.

Back in your .clj buffers, C-x C-e has been rebound to execute the form under the point in Clojure instead of Elisp. This is handy, but you won't get accurate line numbers from stack traces involving functions loaded this way. Pressing C-c C-k will load the entire file and ensure stack traces come through accurately.

As you type out function calls, you should see their argument list show in the minibuffer. This is called eldoc, and it's a great way to get a quick refresher about what a function expects. For full documentation lookup you'll need to get handy with C-c C-d d though. Finally you can use M-. to jump to the definition of any given function.

Projects

Of course, after a while you'll be done with just playing at the REPL and want to hack on a real project. Since the JVM doesn't allow you to modify the classpath at runtime, you need to specify up front where it should look for code. The simplest thing to do is add src/, test/, lib/, and classes/ (for AOT compilation, if desired) directories in your project root to the classpath. Then you place your dependency jars in the lib directory. ~~If you've got complicated dependencies, you could use maven to manage them, but if you've only got a couple it's not hard to do by hand.~~ Use Leiningen for dependency management and other build needs.

Invoking M-x swank-clojure-project will prompt you for a project root and start SLIME with the classpath configured appropriately.

Tests

If you've written automated tests for your project using the clojure.test library (which you should), you can use clojure-test-mode to run them. Install it via M-x package-list-packages, and then you can use C-c C-, to run the tests in the current buffer. Failures and errors get highlighted, so if you want to see details about a failure, move the point to the red region and press C-c C-'.

Happy Hacking

I hope this is helpful and clears up some confusion. Now get out there and write some code.

Update: If you are using Slime with both Clojure and Common Lisp, refer to the instructions at http://felipero.posterous.com/1446961.

in which a tool helps with relaxation

2009-05-19T17:20:29Z

I've been experimenting a bit with CouchDB recently. I've been really impressed with how quickly it lets you get started; interacting with it always feels obvious. I've spent enough time doing web apps to get tired of how everyone always jumps straight to the unholy combination of a relational database system and object-relational mapper whenever they need any persistence. (When all you have is a hammer...) So using CouchDB with a functional language is a breath of fresh air.

Putting together a CouchDB adapter in Clojure is really a matter of no more than a page worth of code once you've got an HTTP client, so it's left as an exercise to the reader. Since functional languages encourage working closer to your core data types (which have clear mappings to JSON) there's very little cognitive overhead when it comes to using Couch for persistence.

Last week while I was in Portland I put together a client written in Elisp called relax.el. This lets you browse Couch databases from the comfort of Emacs. It also lets you edit documents straight from an Emacs buffer, which is much more convenient than using the browser's text tools from Futon.

There are a few quirks with regard to pagination, (just press g if it gets messed up) but it's pretty useful as it is. It should be in ELPA soon, so M-x knock-yourself-out.

in which codes are peeped at, in clojure

2009-04-24T21:30:08Z

I'm very proud to announce the release of my latest project, Functional Programming with Clojure, a PeepCode screencast created with Geoffrey Grosenbach. We were lucky to have Rich Hickey, the creator of Clojure do the technical review.

I've blogged about Clojure before. It's becoming clear that with the way multi-core systems are becoming the norm, functional programming provides tools to handle concurrency in a straightforward way. Languages like Haskell and Erlang are experiencing surges in popularity, but Clojure brings some great new tricks of its own to the table. Part of this is its heritage as part of the Lisp family of languages, part of it is being built on a really solid virtual machine, and part of it is Rich's ingenuity that shines in its design.

If you've been wanting to dive into Clojure, this is a great way to get started quickly. The screencast steps through the development of Mire, a multiplayer text adventure. I found this to be an fitting project to showcase Clojure's strengths since it involves concurrent processing while still requiring state to be shared between threads, and it's simple enough to cover in an hour.

So take a look! I think you'll like it.