The build tool: Jam

Requirements for a build tool

Building a complete operating system is a somewhat complex task. In the case of Haiku, the build involves up to 3 different compilers: for the host system, and in the case of hybrid builds, two different target compilers which are different versions of gcc. Multiple host systems are supported, with a compatibility layer to be able to use some Haiku specific features like extended filesystem attributes even on operating systems that don’t support them. Various target architectures also require us to generate versions of our bootloader that run on a wide variety of environments (BIOS, EFI, OpenFirmware, …) and uses several executable formats. The work doesn’t stop there, as the build system also includes support for downloading pre-built packages, generating complete filesystem images, and also allows a large amount of customization.

Typical build systems usually fall in two categories: low-level systems like make or ninja focus on handling dependencies and running jobs in the correct order, while high-level build systems like cmake or meson provide pre-made rules for most common tasks, resulting in very simple builds for typical projects.

Neither of these are quite satisfactory for Haiku: the scale of the project would make it difficult to handle everything with low-level makefiles, while tools like cmake don’t offer nearly enough control on the compilation process for our needs.

We can summarize these requirements (this isn’t a complete list) for an ideal build system: - Automatic parsing of C++ source files to determine dependencies to header files, and rebuild automatically if those have changed, - Ability to build using multiple compilers, and different set of compiler flags for each target, - Factorization of common commands in a single place, to build many similar targets using the same process, - Handling of tasks other than building code, - Support for multiple steps to build a single target (for example, compiling an executable and then adding resources to it), ideally with correct dependency handling (add the resources only if the executable was rebuilt, or the resource file changed) - Some level of configurability to allow modifying the exact content of the generated filesystem image.

This resulted in the choice of a less known, but more appropriate tool in the form of Jam. This was originally developped by Perforce, but today they have abandoned it. Just like Make, there are various flavors of Jam now available, the most known ones being Boost Build and Freetype Jam. Haiku provides its own variant, as we needed to extend Jam in various ways to make it work for our needs.

Short overview of how Jam operates

The core idea of Jam is to provide generic rules (for example “how to build an application from a set of source files”) and then apply these rules several times. The Haiku build system defines a number of custom rules, allowing to build code both for Haiku and for the host operating system (to be run during the compiling process).

For example, to build an application, one may use:

Application MyExampleApp                      # Name of the application
    : MyExampleSource.cpp AnotherSource.cpp   # Source code of the app
    : be translation                          # Libraries needed
    : MyExampleApp.rdef                       # Resource file
    ;

This will invoke the “Application” rule which looks something like this (this is a simplified version of it):

rule Application
{
    AddResources $(1) : $(4) ;
    Main $(1) : $(2) ;
    LinkAgainst $(1) : $(3) ;
    LINKFLAGS on $(1) = [ on $(1) return $(LINKFLAGS) ]
            -Xlinker -soname=_APP_ ;
}

This in turns invokes more rules: AddResources, Main and LinkAgainst, and also tweaks LINKFLAGS variable for the target. You can see how Jam allows to set variables for a target, which allows us to decide which compiler and flags to use in each case.

The low level rules will take care of calling the Depends rule to let Jam know how dependencies are organized and what needs to be built first.

Rules can call each other until eventually we get to a rule that includes an “action” part. This is where we finally get to actually running commands to do something. For example here is (again, a simplified version for this example) the Cc rule that runs a C compiler:

rule Cc
{
    # Make sure the target depends on the sources
    Depends $(<) : $(>) ;

    # Set variables on the target that will be used by the action
    on $(1) {
            local flags ;

            # Enable optimizations when not building in debug
            if $(DEBUG) = 0 {
                    flags += -O3 ;
            } else {
                    flags += -O0 ;
            }

            # Select a compiler (the actual Cc rule will pick a different one depening on what is being
            # built)
            CC on $(1) = gcc ;

            # Prepare the command line for building the target
            CCFLAGS on $(<) = $(flags) ;
            CCHDRS on $(<) = [ FIncludes $(HDRS) ] ;
            CCDEFS on $(<) = [ FDefines $(DEFINES) ] ;
    }
}

actions Cc
{
    # Actually invoke the compiler with the command line flags prepared by the rule
    $(CC) $(CCFLAGS) -c "$(2)" $(CCDEFS) $(CCHDRS) -o "$(1)"
}

For building Haiku, Jam is combined with a simple “configure” shell script to do the initial setup. There is no need for something as complex as autotools, because of the relatively limited number and diversirty of supported host operating systems (we can assume a reasonably modern UNIX style system) and the fact that a large part of the build is made with our own compiler and libraries, which we have full control on.

Jam rules used for building Haiku

The rules are defined in the build/jam directory. They are spread accross multiple files, each handling a specific part of the build.

Here is an overview of some of these files and what they are used for.

OverriddenJamRules

Jam provides a default set of rules for building simple software. Unfortunately, Haiku isn’t so simple, and a lot of these rules need to be redefined, in particular to handle our setup with multiple compilers.

BeOSRules

Rules specific to BeOS-like operating systems, mainly management of extended attributes and executable resources.

MainBuildRules

Rules for building Haiku applications. This file defines rules like Application, Addon, StaticLibrary. It also contains rules for building on the host: BuildPlatformSharedLibrary, BuildPlatformMain, etc.

ArchitectureRules

Management of compiler flags for different CPU architectures and “hybrid builds” where two compilers are used.

BootRules

Rules related to building the bootloaders.

KernelRules

Rules for building the kernel and kernel add-ons.