Tuesday, September 12, 2017

Bazel: Building a JNI Lib

Here's how I managed to use Bazel to compile a JNI wrapper for a C library.  To understand it you'll need to understand the basics of Bazel; you should go through Java and C++ tutorials on the Bazel site, and understand workspaces, packages, targets, and labels. You will also need to understand Working with external dependencies.  That documentation is a little thin, but in this article I'll try to explain how it works, at least as I understand it.

To start, a JNI wrapper for a C/C++ library - let's call it libfoo - will involve three things (native file extensions omitted since they are platform-dependent):
  1. The Java jar that exposes the API - we'll call it libfooapi.jar in this case
  2. The JNI layer (written in C in this case, C++ also works) that wraps your C library (translates between it and the API jar) - we'll call this libjnifoo
  3. Your original C library - libfoo
So the task of your build is to build the first two.

Code Organization

My C library (libfoo) is built (using Bazel) as a separate project on the local filesystem. This makes it an "external Bazel dependency"; we'll see below how to express this in our JNI workspace/build files.

The source for the JNI lib project looks something like this (on OS X):
$ tree foo
foo
├── BUILD
├── README.adoc
├── WORKSPACE
├── src
│   ├── c
...
│   │   ├── jni_init.c
│   │   ├── jni_init.h
... other jni layer sources ...
│   │   └── z.h
│   └── main
│       ├── java
│       │   └── org
│       │       └── foo
│       │           ├── A.java
...
│       │           ├── FOO.java
...
Note that WORKSPACE establishes the project root; you will execute Bazel commands within the foo directory. Note also that we only have one "package" (i.e. directory containing a BUILD file). We're going to build the jar and the jni lib as targets in the same package.

The third thing we need is the JDK, since our JNI code has a compile-time dependency on "jni.h"; this is a little bit tricky, since this too is an external dependency, and you cannot brute-force it by giving an absolute path the the JDK include directories - Bazel rejects such paths. We'll see how to deal with such "external non-Bazel dependencies" below.

WORKSPACE

My WORKSPACE file defines my libfoo library as a local, external, Bazel project:

local_repository(
    name = "libfoo",
    path = "/path/to/libfoo/bazel/project",
)

In this example, /path/to/libfoo/bazel/project will be a Bazel project - it will contain a WORKSPACE file and one or more BUILD files.  Defining a local "repository" like this in the workspace puts a (local) name on the "project" referenced by the path.  Note that projects do not have proper names; a "project" is really just a repository of workspaces, packages, and targets defined by WORKSPACE and BUILD files - hence the name "local_repository".

The "external" output dirs

When you define an external repo like this, Bazel will create (or link) the necessary resources in subdirectories of the output directories, named appropriately; in this case, "external/libfoo".

In other words, if you define @foo_bar, you will get "external/foo_bar" in the output dirs.

TODO: explain - does Bazel just create soft links to the referenced repo?

JDK Dependency

Our JNI library will have a compile-time dependency on the header files of the local JDK. Many more traditional build systems would allow you to express this by add the absolute file path to those include directories; Bazel disallows this.  Instead, we need to define such resources as an external repository; in this case, a non-Bazel external repository.

You could do this yourself for JDK resources, at least in principle (I tried and failed).  Fortunately Bazel predefines the repository, packages and targets you need.  The external repo you need is named "@local_jdk" (note: underscore, not dash); the targets are defined in https://github.com/bazelbuild/bazel/blob/117da7a947b4f497dffd6859b9769d7c8765443d/src/main/java/com/google/devtools/build/lib/bazel/rules/java/jdk.WORKSPACE.  Frankly I do not completely understand how local_jdk and etc. definitions work, but they do. In this example, we will use:
  • "@local_jdk//:jni_header"
  • "@local_jdk//:jni_md_header-darwin"
(If you look at the source, you will see that these are labels for "filegroup" targets - a convenient way to reference a group of files.)

(See also https://github.com/bazelbuild/bazel/blob/master/src/main/tools/jdk.BUILD - I don't know how this is related to the other file.)

BUILD

The local name for the external repository (project) specified in your WORKSPACE makes it possible to refer to it using Bazel labels.  In your BUILD files you use the @ prefix to refer to such external repositories; in this case, "@libfoo//src:bar" would refer to the bar target of the src package of the libfoo repository, as defined in the WORKSPACE above. Note that because of the way Bazel labels are defined, we can expect to find target "bar" defined in file "src/BUILD" in the libfoo repo.

My buildfile has two targets, one for the Java, one for the C.  The Java target is easy:

java_library(
    name = "fooapi",  # will produce libfooapi.jar
    srcs = glob(["src/main/**/*.java"]))

The C target is a little more complicated:

cc_library(
    name = "jni",
    srcs = glob(["src/c/*.c"]) + glob(["src/c/*.h"])
    + ["@local_jdk//:jni_header",
       "@local_jdk//:jni_md_header-darwin"],
    deps = ["@libfoo//src/ocf"],
    includes = ["external/libfoo/src",
                "external/libfoo/src/bar",
                "external/local_jdk/include",
                "external/local_jdk/include/darwin"],
)

Important: note that we put the @local_jdk labels in srcs, not deps.  That's because (I think) they are filegroups, and the labels you put in deps should be "rule" targets rather than file targes.

Exposing the headers

Note that it is not sufficient to express the dependencies on local_jdk; you must also specify the include directories with that repo in order to expose jni.h etc.  That's what the "includes" attribute is for.  You must list all the (external) directories needed by your sources.