Building Clojure Apps on Google App Engine with boot-gae

It's relatively easy to get a Clojure application running on GAE's devserver; you just need to use gen-class to AOT compile a servlet. See for example Clojure in the cloud. Part 1: Google App Engine. The problem is that you then need to restart the devserver whenever you want to exercise code changes, which is way too slow.

One way around this limitation is to run Jetty or some other Java servlet container rather that the devserver.  See for example:

• appengine-magic
• Interactive Programming with Clojure, Compojure, Google App Engine and Emacs
• Clojure on Google App Engine

The problem with this strategy is exactly that it does not use the official development server from Google. That server is a modified version of Jetty, with strict security constraints, providing a near-exact emulation of the production environment (which also runs a version of Jetty). If you develop with some other servlet container, you won't know if your code is going to run in production until you actually deploy to the cloud.

So there are two problems to be addressed if we want to use the official devserver.  One is that Java servlets must be compiled, since the servlet container will search for byte-code on disk when it comes time to load a servlet; most solutions I've seen end up AOT-compiling the entire app. The other problem is that GAE's security constraints will prevent your app from accessing anything outside of the webapp's directories. That means, for example, that any jar dependencies should be installed in WEB-INF/lib. If you want to load Clojure source files at runtime, they must be on the classpath, e.g. in WEB-INF/classes.

boot-gae is a new set of tools that solves these problems. Using it, you can easily develop Clojure apps with REPL-like interactivity in the devserver environment. It automates just about everything, so building and running an application is as simple as:

$ boot gae/build gae/run

To develop interactively, switch to another terminal session and run

$ boot gae/monitor

It's that simple. Now changes in your source tree will be propagated to the output tree, where they will be reloaded on page refresh.

The gae/build task is a convenience task that composes a number of core tasks that take care of everything:

• installing jar dependencies in WEB-INF/lib
• generating the config files WEB-INF/appengine-web.xml and WEB-INF/web.xml
• generating one stub .class file for each servlet and filter
• copying Clojure source files from the source tree to WEB-INF/classes for runtime reloading
• copying static web assets (html, js, css, jpeg, etc.) from the source tree to the appropriate output directory
• generating and installing a reloader filter, which will be used to detect and reload changed namespaces at runtime

The process is controlled via simple *.edn files. For example, servlets are specified in servlets.edn, which looks like this:

{:servlets [{:ns greetings.hello
             :name "hello-servlet"
             :display {:name "Awesome Hello Servlet"}
             :desc {:text "blah blah"}
             :urls ["/hello/*" "/foo/*"]
             :params [{:name "greeting" :val "Hello"}]
             :load-on-startup {:order 3}}
            {:ns greetings.goodbye
             :name "goodbye-servlet"
             :urls ["/goodbye/*" "/bar/*"]
             :params [{:name "op" :val "+"}
                      {:name "arg1" :val 3}

                      {:name "arg2" :val 2}]}]}

Here two servlets are specified. One task - gae/servlets - will use this data to generate a "servlets generator" source file that looks like this:

;; TRANSIENT SERVLET GENERATOR
;; DO NOT EDIT - GENERATED BY servlets TASK
(ns servletsgen2258)

(gen-class :name greetings.hello
           :extends javax.servlet.http.HttpServlet
           :impl-ns greetings.hello)

(gen-class :name greetings.goodbye
           :extends javax.servlet.http.HttpServlet
           :impl-ns greetings.goodbye)


This file is then AOT-compiled to produce the two class files, WEB-INF/classes/greetings/hello.class and WEB-INF/classes/greetings/goodbye.class. The programmer then need only supply an implementation for the service method of HttpServlet, in an appropriately named Clojure file - in this case, in the source tree, greetings/hello.clj and greetings/goodbye.clj will both contain something like (defn -service ...) or (ring/defservice ...).

Another task, gae/webxml, will use the same information to generate WEB-INF/web.xml.

Thus with boot-gae, only minimal servlet and filter stubs are AOT-compiled. The gen-class source code is itself automatically generated, then AOT-compiled to produce the corresponding class files, and discarded. The programmer never even sees this code (but can keep it for inspection by passing a -k parameter).

boot-gae is available at https://github.com/migae/boot-gae.  A companion repository,  https://github.com/migae/boot-gae-examples contains sample code with commentary.

Next article: boot-gae: building and assembling service-based apps