in which interactive development saves the day

When I was writing EXO_encounter 667 in Fennel, I benefited immensely from the ability to do live reloads. Instead of having to restart the whole process, I could run a single key command from my editor and have the game see the new code immediately. This isn't particularly difficult to do in Fennel, but it's not immediately obvious at a glance either.

Before you understand how reloading works in Fennel, you need a little background regarding Lua's module system, since Fennel is just a compiler that emits Lua code. Older versions of Lua had a module function which would declare the whole rest of the file as being part of a specific module and register that with the module system, and all functions that would normally be declared as global within that file would be exported as part of the module instead. But in version 5.1, that system was recognized as redundant: nowadays a module is just a file that returns a table1 with closures and other values in it. This is reflects the relentless simplicity behind the design of Lua; why have modules as their own concept when tables and closures can do just as good a job?

So that's all well and good; you can just write code that uses functions written in other files by just calling dofile on the filename and putting that value in a local. And that works, but every time you use the module from another place it loads a fresh copy, which is wasteful. Enter the require function. It takes a module name which maps to a filename (by searching the entries of package.path) and gives you the value returned by that file, but it also caches subsequent calls. So every time you require a module, you're getting the exact same table2 in the exact same memory location.

Valley near Mt. Saint Helens

We can take a little detour here from Lua land and back into Fennel, because dofile only works on Lua code. Fennel provides its own fennel.dofile function which works just like the built-in one, but on .fnl files instead. But what about require? Well it turns out require is implemented in a pretty clever way that allows us to teach it new tricks. The way require works is that it looks at the package.searchers table, (it's package.loaders on Lua 5.1) which contains a list of searcher functions. It iterates over the list, calling each searcher with the module name. If that returns nil, it indicates that searcher can't find the module and it moves on, but a searcher which can load the module will return a function which allows require to get (and cache) the value for the module in question. So simply by adding fennel.searcher to package.searchers, we can make it so that require works seamlessly on modules whether they are written in Fennel or Lua:

(local fennel (require "fennel"))
(table.insert package.searchers fennel.searcher)

Now this seems somewhat academic; after all, you have a lot of memory; why do you care if modules are duplicated in memory? But using require for modules proved invaluable during the development of my game because it allowed me to do all my local hacking using .fnl files I was constantly editing, but when I prepared a release, I precompiled it all into .lua files and didn't have to change a line of my code to reflect that.

Well that's wonderful, but if require caches the value of each module, doesn't that interfere with live reloading? Indeed it does; simply re-requiring a module has no effect. You can call fennel.dofile to get a copy of the updated module. But that's no help to the existing code which has the old version of the module. What to do?

To understand the solution it's helpful to make a distinction between the identity of the table and the values it contains. The identity of a table is what makes it truly unique; it can be thought of in terms of that table's particular location in memory. When you pass a table to a function, that function has access to the exact same table, and changes made to it inside the function of course are visible to any other function that has access to the table3. The value of a table refers to what it contains; in the case of a module it's usually about what functions are present under what keys. Since the tables are mutable, the value can change over time but the identity cannot. When you call dofile on a module you get a table that might have the same values as last time you called dofile, (if the file on disk hasn't changed) but it will never have the same identity. When you call require you're guaranteed to get the exact same identical table every time.4

With that background maybe you can see now how this might work. All the existing code has access to the original module table. We can't swap out that table for a new one without reloading all the modules that use it, and that can be disruptive. But we can grab that original table, load a fresh copy of its module from disk, then go in and replace its contents with the values from the new one.

(defun fennel-reload-form (module-keyword)
  "Return a string of the code to reload the module-keyword module."
  (format "%s\n" (let [old (require ,module-keyword)
                            _ (tset package.loaded ,module-keyword nil)
                            new (require ,module-keyword)]
                    ;; if the module isnt a table then we can't make
                    ;; changes which affect already-loaded code, but if
                    ;; it is then we should splice new values into the
                    ;; existing table and remove values that are gone.
                    (when (= (type new) :table)
                      (each [k v (pairs new)]
                            (tset old k v))
                      (each [k (pairs old)]
                            ;; the elisp reader is picky about where . can be
                            (when (not (,"." new k))
                              (tset old k nil)))
                      (tset package.loaded ,module-keyword old)))))

The code above looks like Fennel, but it's actually Fennel embedded inside Emacs Lisp code; because they're both just made up of s-expressions, you can write Fennel code as Elisp code and quote it, then send it to the Fennel repl subprocess which is launched with M-x run-lisp. My recent changes to fennel-mode.el allow this to work out of the box, but they could easily be adapted to any other editor that supports communicating with an integrated repl subprocess.

Of course, all this background really isn't necessary; you can just hit reload now and have it work with no fuss. But sometimes it's interesting to understand why it works, and especially I think in this case the design decisions that went into the module system are noteworthy for allowing this kind of thing to be done in a graceful way, so that's worth appreciating and hopefully learning from.

[1] Technically a module can return any value, not just a table. But if you return a non-table, then the reloading features described don't work, because only tables can have their contents replaced while retaining their same object identity.

[2] Yep; this means you can abuse the module system to do terrible things like share application state across other modules. Please resist the temptation.

[3] Oddly enough in some languages this is not true and data structures default to being copied implicitly every time you pass them to a function, which can be very confusing. To muddle things even more, this behavior is referred to as "pass by value" instead of "we make copies of everything for you even when you don't ask". That doesn't happen here.

[4] For a fascinating discussion of the difference between value and identity and how it relates to equality I strongly recommend reading the very insightful Equal Rights for Functional Objects which goes into much more depth on this subject. Notably Lua's (and Fennel's) equality semantics are consistent with its recommendations despite Lua being an imperative language.