The Art of Jamming - All Parts

build, jam

Body

Part 1

Before the musicians in the audience get too excited, I'm not going to talk about getting together with buddies to crank out some tunes. Nor will the cooks in the audience find instructions on making the perfect fruit preserves. No, when I say Jamming, I refer to the act of using the Jam build tool.

In this first part of my series on the Jam build tool, I'm going to provide a high level overview as well as show the product of some of my recent labors with Jam: the Jamfile-engine. For those who have developed on BeOS for a decent amount of time, this may sound familiar. It should, because I have essentially taken the functionality of the Be makefile-engine and "ported" it to Jam.

What Sort of Engine?

For those not familiar with the makefile-engine, take a look at your /system/develop/etc directory some time. In a nutshell, this tool provides a nicely commented makefile which you fill out like a form to use in building your application. You provide the name for the application, the source files it is compiled from, libraries to link to, extra include directories, etc. At the bottom of the makefile template is a command that includes the makefile-engine, which does all the work to get your project built. The same makefile template can also be used to build libraries and drivers.

So as you can see I think the makefile-engine is yet another great idea, originally from Be, Inc., and decided to create a Jam-based tribute to it. Of course there are good reasons to do this besides paying a tribute. One is that it just doesn't seem right to have Jam the official build tool for Haiku, yet leave our application developers using a make-based build system.

Jam From 10,000 Feet

From a high-level Jam is essentially an interpreted scripting language with a specialty in building things. I consider it to be a nice evolution of make, in that it is more powerful and in general easier to use. Of course people who are used to make probably won't like Jam at first because it is so different, and requires a different kind of thinking when creating a build script.

So how does one create a Jam build script? First you define any variables you might need or that are used by rules you will call, and then you call those rules. Rules are somewhat like functions or methods in programming languages: they have parameters and perform some useful function, which for Jam usually involves creating one thing from another. To ease the creation of build scripts, Jam has a capable language syntax, and quite a few built-in rules that perform common build-tool functions, like compiling C or C++ files and linking applications. I will go into more detail concerning the Jam language and built-in rules in my next article, but must explain one thing in detail now: Jam's use of whitespace. It is important to learn this early and well: Jam interprets build scripts as a list of tokens separated by whitespace, and to have the files you create interpreted properly, you must have whitespace everywhere. This includes putting whitespace before the semi-colon (;) that terminates a line. This is important to learn now because you must do this when using the Jamfile-engine described below. If you forget to put whitespace in important areas, Jam will produce errors which can be hard to figure out (believe me, I know.)

Jamming Things Up

So now that you have some basic Jam knowledge, let me explain how to set up the Jamfile-engine. It used to be more complicated - download and install 'jam' and an archive of the Jamfile-engine - but it's dead easy now: 'jam' is already preinstalled with Haiku and the Jamfile-engine can be installed with HaikuDepot or a quick pkgman install jamfile_engine in Terminal.

Find a good BeOS/Haiku application that you have the source code to, so that you can create a new Jamfile with which to build it. It would be a plus if the application you choose already had a makefile which uses the makefile-engine. A makefile of this sort can be spotted by an initial line like this:
```
## BeOS/Haiku Generic Makefile v2.0 ##
```
The version number may be different, but if it looks like that, you have a generic makefile which uses the makefile-engine.

Once you have done the above, you can proceed to fill out the Jamfile template for this application:

Copy the Jamfile from the location that the Jamfile-engine files were installed to (which by default is /system/develop/etc.)
Open the copied Jamfile and the old makefile, if there is one.
Fill out the Jamfile as specified in the comments, or by copying from the makefile. Remember the above discussion of whitespace.

Once you have the template filled out to your satisfaction, open up a Terminal, cd to your application directory and type jam. If you have provided the correct Jam incantations in your template you should find a new obj.X86 directory in your application directory that contains the object files and final application for the project. If you get any errors from the Jamfile-engine, they should be fairly friendly and point you in the right direction. If you get some obscure, hard-to-understand Jam error, look over your template again and make sure you have all the necessary spaces!

One final note to any PowerPC owners in the audience: since I used the Be makefile-engine as a sort of template for my Jamfile-engine, I have tried to include support for PPC machines similar to what was in the makefile-engine. But since I only have an x86 machine I was unable to test this functionality. If anyone would like to test out the Jamfile-engine on a PPC machine (on BeOS) and let me know how it goes I would appreciate it!

In my next article I will walk through the implementation of the Jamfile-engine and give more details about how Jam works and how you can put it to more advanced use in any projects you may be working on.

Part 2

If you open up the Jamfile-engine text file, you will notice it begins with a fairly standard comment section at the beginning:

##  Haiku Generic Jamfile Engine v1.0.2

##  Does all the hard work for the Generic Jamfile

##  which simply defines the project parameters.



##  Most of the real work is done in the Jambase

##  embedded into the jam executable.

##

##  Inspired by the Be Makefile Engine

##

##  Supports Generic Jamfile v1.0.1

##

##  Copyright (c) 2002-2010 Ryan Leavengood

##  Copyright (c) 2011 Peter Poláčik

##  Released under the Terms of the MIT License, see


##  http://www.opensource.org/licenses/mit-license.html

From this you will realize that comments in Jam begin with a hash symbol (#) and they continue until the end of the line. In this case two hashes are used to make this comment stand out more and to indicate that it describes the file as a whole and not just some implementation detail.

After the initial comment the real code begins. In the first part of the Jamfile-engine, several utility rules are defined that can be used and re-used later in the file. The first few rules are simple:

# AddResources <Application Name> : <Resource Files> ;


#   Adds the given resources to the given application.

rule AddResources
{
    Depends $(<) : $(>) ;
}

actions AddResources
{
    $(XRES) -o "$(<)" $(>)
}

# MimeSet <Application Name>;
#   Sets the mime type of the given application to be an application.
actions MimeSet
{
    $(MIMESET) -f "$(<)"
}

Jam Rules: Procedure and Actions

After looking at the template above, many readers may be confused. Why is AddResources defined twice, once using the word rule and then with the word actions? The reason is that Jam rules are created in two parts:

The procedure (defined using the keyword rule):
A set of Jam language statements that are run when the rule is invoked and which usually set up variables that will be used in the actions.
The actions:
Shell commands that get run when a target needs updating (and make use of the variables configured in the procedure.)

In the above case of AddResources, the procedure sets up a dependency between the first parameter and the second parameter:

$(<) depends on $(>)

meaning that when $(>) changes, $(<) must be updated.

Rule Parameters

The variables used here, $(<) and $(>) are aliases for $(1) and $(2), which are the first and second parameters passed to the rule, respectively. Up to 9 parameters, $(1) - $(9), may be passed to rules. While all can be used in the procedure, only the first two can be used in the actions.

In the case of the actions for AddResources, the command xres is called using the variable $(XRES), with the output (-o) being the quoted value of $(<) (the quotes allow for spaces in the application name), and with the input resource files being the value of $(>).

When defining updating rules, $(1) or $(<) is generally considered to be the target (what will be created by the rule) and $(2) or $(>) is the source(s) (what the target will be created from.)

Though mentioned briefly, it should be evident that $(XRES) is just a reference to a global variable defined elsewhere in the Jamfile-engine (which just has the name of the xres command.) This shows that any variables within scope can be used in the actions for a rule.

After AddResources, the MimeSet rule is created, but in this case there is no procedure because there are no variables needed or dependencies for this rule. In fact, this rule only has one parameter: the name of the application that the mimeset command should be run against.

After the above rules a more complicated rule is defined:

# ProcessLibs <List of Library Names> ;

#   Prepends -l to any library names that aren't _APP_ or _KERNEL_ or
#   that don't have .a or .so file extensions. The result will be given
#   to the linker so that it links to the right libraries.
rule ProcessLibs
{
    local result ;
    
    for i in $(1)
    {
        if ( ( $(i) in _APP_ _KERNEL_ ) || ( $(i:S) in .so .a ) )
        {
            result += $(i) ;
        }
        else
        {
            result += -l$(i) ;
        }
    }
    
    return $(result) ;
}

There is a lot going on in the rule, but the first thing that should be mentioned is that, in contrast to the MimeSet rule, this rule only has a procedure but no associated actions. In this case the rule is just a string processing rule that iterates over a list of library names and adds a -l prefix to those that aren't _APP_ or _KERNEL_ or that don't end in .so or .a. To implement this functionality, a lot of the built-in Jam language is used.

Basic Syntax

The first thing done in this rule is the declaration a local variable called result. The local keyword provides for dynamic scoping (as in C or C++): if another variable named result exists when this rule is called, that old value will be saved, the new value will have nothing to do with the old one, and then after the rule is finished, the new value is discarded and the old one restored.

After the declaration of the local result variable, there is another new piece of syntax: the for loop. In this case the for keyword is used to iterate over the items in the list $(1) (the first and only parameter to this rule), setting the variable i to each item in turn.

After the for loop statement is its associated block, which contains the statements that should be run on each loop iteration. This block contains a single if..else statement, which is very similar to the if..else construct in C. The condition that this if statement checks is fairly complex, but when broken down it is simple: in the first set of parenthesis, the in keyword is used to see if the current value of $(i) is in the list [_APP_ _KERNEL_].

List Handling

The in keyword, as can probably be guessed, returns true if the first parameter is a subset of the second. Note the use of the term subset: both "parameters" to the keyword in are lists, and the result is only true if every element in the first list is in the second list. (In this case only a single-element list is used for the first parameter.)

Also note that literal lists in the Jam language (such as _APP__KERNEL_ in this case) do not need any delimiters (where C syntax would require curly brackets ({ and }), commas (,), and double quotes (") to create a list of strings.) This is one of the advantages of the syntax of Jam (and one of the reasons for all the whitespace.) This really begins to illustrate that at its heart Jam is just a list processing language (hello LISP fans!) In fact, changing the Jam syntax to be like LISP probably wouldn't be too hard, but that shall be left as an exercise for the reader.

Operators

After the first condition in the if is an or (||) disjunction, which works just like the C equivalent (the result is only false if both sides are false.) The second side of the or is a condition similar to the first: it checks to see if something is in a list. In this case, though, one of Jam's variable modifiers, :S, is used.

What :S does is return the last filename suffix of the given variable -- in other words -- the file name extension. If the file name extension is in the list ".so.a", then this statement will be true.

Whenever the if condition (which you will note has whitespace separating all tokens) returns true, the += expression is used to add the unmodified value of $(i) to the result list. The operator += works the same as in C: the value of result is set to the old value of result plus the value of $(i). But since Jam is a list-oriented language, this addition is not mathematical, but is a list addition: the new value is added as a new element to the end of the list. In fact, Jam does not have any syntax for doing math at all.

In the case that the if condition is false, the block under the else clause will be run. This block adds the value of $(i) with the prefix -l to the result list.

Variable Expansion

Though it doesn't really come into play here, now is a good time to mention how Jam "variable expansion" works. When you concatenate several variables or a variable with one or more literals, the result is a list that is a product of the components of the variables being combined.

For example, in the simple case of the -l$(i) statement above, the result will be the value of $(i) with -l prepended to it. Since the for loop insures that $(i) is a single element list, the result is simple, but if $(i) had more than one element (such as [be media midi]) the result would be:

[-lbe -lmedia -lmidi].

Given that value of $(i), the result of $(i)$(i) would be:

[bebe bemedia bemidi mediabe mediamedia mediamidi midibe midimedia midimidi].

Try saying that three times fast.

Two final notes regarding variable expansion: if a list contains the null string (""), the result of expansion is still a product, but only of non-null elements. For example, if a variable $(x) was the list [A ""] and $(y) was ["" 1], the expansion of *$(x)$(y)* would be [*A* *A1* ** *1*]. The other note is that any expansion that uses an undefined variable results in an empty list.

After ProcessLibs is a similar string-processing rule:

# MkObjectDirs <List of Source Files> ;
#   Makes the necessary sub-directories in the object target directory based
#   on the sub-directories used for the source files.
rule MkObjectDirs
{
    local dir ;
    
    for i in $(1)
    {
        dir = [ FDirName $(LOCATE_TARGET) $(i:D) ] ;
        Depends $(i:S=$(SUFOBJ)) : $(dir) ;
        MkDir $(dir) ;
    }
}

This rule is used to create sub-directories for the object files that mirror the directory structure of the project source files. Similar to ProcessLibs, a local variable is declared, and a for loop is used, which in this case iterates over source file names. In the loop body the variable dir is set to be the result of a call to the built-in FDirName rule, which takes the items in the list passed to it and concatenates them with directory separators in between each item.

The parameter to FDirName is a list containing the target directory into which everything is built (defined later in the Jamfile-engine) and the directory of the source file (that is what the :D variable-modifier returns.) Then a dependency is set up between the object file for the source file and the created directory name.

The :S= modifier used with $(i) replaces the file extension of the variable with the given suffix, in this case the variable $(SUFOBJ), which is .o on BeOS. By creating this dependency between the object file and the directory it is created in, we can ensure the directory is properly created before the object file.

After the dependency is set up, the actual MkDir rule is called with the given directory. This rule creates the given directory if it doesn't already exist, including any needed parent directories, like the GNU mkdir command with the -p option.

After the MkObjectDirs rule are a few more simple rules:

# RmApp <Pseudotarget Name> : <Application Name> ;

#   Removes the given application file
#   when the given pseudotarget is specified.
rule RmApp
{
    Depends $(<) : $(>) ;
}

actions RmApp
{
    rm -rf "$(>)"
}

# RunApp <Pseudotarget Name> : <Application Name> ;
#   Runs the given application in the background

#   when the given pseudotarget is specified.
rule RunApp
{
    Depends $(<) : $(>) ;
}

actions RunApp
{
    "$(>)" &
}

Pseudotargets

These rules look very similar to AddResources, and their function in the Jamfile-engine should be obvious. One thing that may not be obvious is what the parameters are, particularly the first, pseudotarget name.

A pseudotarget is a name that defines a target which can be specified on the command-line to jam, but that is not really a file system target that can be created. This distinction is specified using certain Jam rules, which will be described later. Suffice it to say that when the pseudotargets passed into the above rules are specified on the jam command-line, the actions for those rules will be run.

For example, if RunApp test : $(APP) ; was specified later in the Jamfile- engine (which it is), running "jam test" on the command-line would run the application in the background (after creating it if need be.)

Now for the next set of rules:

# InstallDriver1 <Pseudotarget Name> : <Driver File> ;
#   Installs the given driver in the correct location
#   when the given pseudotarget is specified.
rule InstallDriver1
{
    Depends $(<) : $(>) ;
    USER_BIN_PATH = /boot/home/config/add-ons/kernel/drivers/bin ;
    USER_DEV_PATH = /boot/home/config/add-ons/kernel/drivers/dev ;
}

actions InstallDriver1
{
    copyattr --data "$(>)" "$(USER_BIN_PATH)/$(>:B)"
    mkdir -p $(USER_DEV_PATH)/$(DRIVER_PATH)
    ln -sf "$(USER_BIN_PATH)/$(>:B)" "$(USER_DEV_PATH)/$(DRIVER_PATH)/$(>:B)"
}


# InstallDriver <Pseudotarget Name> : <Driver File> ;
#   Installs the given driver in the correct location
#   when the given pseudotarget is specified
#   (after making sure that this is actually a driver)
rule InstallDriver
{
    if ( $(TYPE) = DRIVER )
    {
        InstallDriver1 $(<) : $(>) ;
    }
}

These rules, as the names imply, are used for installing drivers. The commands in the actions are taken almost verbatim from the Be makefile-engine (as they say: if it ain't broke, don't fix it.)

The reason that there is an InstallDriver1 and InstallDriver rule is due to the need to check the $(TYPE) variable before actually performing the action. If this isn't actually a driver, it does not make sense to try to install it in the driver directories. So the rule that should be called by users of this rule-set is InstallDriver, which will do the right thing based on the given type of BeOS project. This style of naming (appending 1 to the rule name for the worker rule) is used in the Jambase file, which is why it is used in the Jamfile-engine as well.

Finally, the last two rules defined are:

# Link <Application Name> : <List of Object Files> ;

#   Replaces the actions for the default Jam Link rule with one that
#   handles spaces in application names.
actions Link bind NEEDLIBS
{
    $(LINK) $(LINKFLAGS) -o "$(<)" $(UNDEFS) $(>) $(NEEDLIBS) $(LINKLIBS)
}

# BeMain <Application Name> : <List of Source Files> ;
#   This is the main rule that builds the project.

rule BeMain
{
    MkObjectDirs $(>) ;
    
    if ( $(TYPE) = STATIC )
    {
        Library $(<) : $(>) ;
    }
    else
    {
        Main $(<) : $(>) ;
    }
    
    if ( $(RSRCS) )
    {
        AddResources $(<) : $(RSRCS) ;
    }
    
    MimeSet $(<) ;
}

As the comment illustrates, the first "rule" is really just a re-definition of the actions for the built-in Link rule. The actions from the Link rule in the Jambase have been changed by adding quotes around the application name $(<).

This shows that any of the built-in Jam rules can be modified freely in any Jamfiles you create. In fact, the built-in Jambase can be completely replaced by specifying the -f option to Jam on the command-line (though this may not be too useful since most of Jam's usefulness comes from the rules defined in the built-in Jambase.)

The Main Rule

The final rule, BeMain, is the real work-horse in the Jamfile-engine: this is really what builds the project. Because of all the work done in the rest of the Jamfile- engine, however, this really is quite simple.

First, the MkObjectDirs rule is called with the project source files, which will create the needed object directories as described above. Then an if statement is used to determine if the type of project is a static library. If it is, the built-in Jam rule Library is called, which compiles the given source files and then archives them into a static library of the given name. Otherwise the built-in Jam rule Main is called, which compiles the given source files and then links them as the given application name.

At this point it should be mentioned that both Library and Main make use of the Objects rule, which uses the Object rule, which is smart in that it looks at the file extension of the given source file and then calls the appropriate rule to compile it. Files that end with .cpp or .cc or .C are compiled with the C++ rule, while .c files are compiled with the Cc rule, and .l files are compiled with Lex, etc. Thus sources can be a mixed list of any Jam-supported files and they will all be compiled correctly and linked into one application or library. (There is also a fairly easy way to add support for compiling other types of files, such as Pascal or ASM files, for instance, which will be explained in the next article.)

The rest of the Jamfile-engine is mostly definitions of various variables, and only a few parts of it use any Jam concepts that have not already been explained. Those are the only parts that will be explained here, starting with this:

# Set the directory where object files and binaries will be created.
# The pre-defined Jam variable OSPLAT will indicate what platform we

# are on (X86 vs PPC, etc.)
LOCATE_TARGET = obj.$(OSPLAT) ;

As described briefly in the explanation of the MkObjectDirs rule, this variable defines where targets should be located, i.e. where they are created. In this case it is set to be "obj." with the platform appended. The variable $(OSPLAT) is one of the few variables actually compiled into Jam (and therefore not set in Jambase), and it is an all capital description of the CPU type (such as X86, PPC or SPARC.)

Jam runs on many platforms, and a properly written Jamfile should be able to work on many of them, unmodified. For example, the Jamfile-engine should theoretically run any Haiku platforms as it is today (the term theoretically is used here since no one has tested the Jamfile-engine on a non-x86 machine yet.)

Also, though it is used in MkObjectsDir, LOCATE_TARGET is actually a variable from the Jambase that is used extensively in all the built-in Jam rules. Of course one thing that a helpful Jamfile-engine user discovered is that despite the setting of this variable, source files that exist in subdirectories are created in similar subdirectories under LOCATE_TARGET, not directly in it. This is why the MkObjectDirs rule was created, because otherwise the compiler complains when it tries to put object files into non-existent directories.

After the LOCATE_TARGET definition comes a few more definitions:

# Set some defaults

if ( ! $(NAME) )
{
    ECHO "No NAME defined!" ;
    NAME = NameThisApp ;
}
if ( ! $(TYPE) )
{
    ECHO "No TYPE defined...defaulting to APP" ;
    TYPE = APP ;
}
if ( ! $(SRCS) )
{
    ECHO "NO SRCS defined...defaulting to *.cpp in current directory" ;
    SRCS = [ GLOB . : *.cpp ] ;
}
if ( ! $(DRIVER_PATH) )
{
    DRIVER_PATH = misc ;
}

These are all probably pretty obvious. The few interesting points are:

( ! $(SOME_VAR) ) will be true for an undefined or empty variable.
The GLOB rule returns any files that match the given criteria in the given directory. The pattern rules will be explained more fully in the next article.
The syntax of the square brackets ([]) around a rule invocation expands the results of that rule into a list which can then be assigned to a variable.

Following the definitions above is a large section that defines variables based on the CPU type. Again, most of this is based on the Be makefile-engine, with a few tweaks because of Jam's more capable syntax. One of those tweaks is the use of the Jam switch statement instead of a series of if statements. Since that is the only new piece of Jam syntax, that is all that will be described from this section of the Jamfile-engine:

switch $(OPTIMIZE)
{
    case FULL : OPTIMIZER = -O3 ;
    case SOME : OPTIMIZER = -O1 ;
    case NONE : OPTIMIZER = -O0 ;
    
    # Default to FULL
    case * : OPTIMIZER = -O3 ;
}

The Jam switch statement probably looks familiar to C or C++ programmers. Overall it works the same, but has a few nicer features. For instance, it does not do matching based on simple numeric equivalence, but on string matching.

In the above case, if $(OPTIMIZE) is set to any of the explicitly listed cases, the matching statement gets run. There is no need for a break statement as in C, only the matching statement gets run. There also is no default branch, though the same functionality can be had by using * as the matching criteria, as done above. In fact, the GLOB statement described above actually uses the matching syntax from the switch statement (switch was implemented before GLOB), and again the matching syntax will be more fully described in the next article.

There is one more new Jam rule that is part of this processor-dependent section of the Jamfile-engine, right at the end:

else
{
    EXIT "Your platform is unsupported" ;
}

The EXIT rule prints out the given statement and then halts the execution of Jam. This is best used in cases of serious error, as done above when the $(OSPLAT) is not X86 or PPC.

The next series of statements in the Jamfile-engine are platform-independent settings. The only thing really new here is the definition of the various pseudotargets used by the Jamfile-engine:

# Set up the driverinstall target...this makes it easy to install drivers
# for testing
Always driverinstall ;
NotFile driverinstall ;
InstallDriver driverinstall : $(NAME) ;

# Set up the rmapp target...this removes only the application
Always rmapp ;
NotFile rmapp ;
RmApp rmapp : $(NAME) ;

# Set up the test target...this runs the application in the background
#Always test ;
NotFile test ;
RunApp test : $(NAME) ;

As mentioned above when describing the RmApp, RunApp, and InstallDriver rules, a pseudotarget is defined and then passed to each rule to act as the target that can be passed to Jam on the command-line to perform the given action. In the case of RunApp, "test" is used, which is set up as a pseudotarget by the calls to the built-in Jam rules Always and NotFile.

The Always rule marks a target so that it is always updated, even if it exists. This rule can be used with real file-based targets as well as pseudotargets, though in general it is most useful with pseudotargets. If this rule is not used, the pseudotarget will only work the first time (generally when the target it depends on is first created.)

The NotFile rule is the rule that actually makes a target a pseudotarget, by informing Jam that it isn't really a file, so it cannot be built. When combined with Always, this allows convenient targets to be specified when calling Jam, so that for instance jam rmapp will remove an application created by the Jamfile-engine, "jam test" will run that application, and jam driverinstall will install a driver in the correct place.

Finally, the last statement in the Jamfile-engine is a call to the previously-described BeMain rule:

##-------------------------------------------------------------------
## OK, let's build
##-------------------------------------------------------------------

BeMain $(NAME) : $(SRCS) ;

So by now you should understand quite a bit more about Jam and also how the mysterious Jamfile-engine works. As you can see, the Jamfile-engine really isn't that complicated, and overall is quite a bit simpler than the Be makefile-engine (though with all my comments they are about the same length.) Also it should be evident that though Jam is probably more complicated than make, the extra functionality is very useful, and the platform-independence of most Jamfiles alleviates the need for complicated configure scripts.

The next article in this series will be a Jam cookbook that will describe how various build problems and challenges can be solved with Jam. The author already has a few ideas for some "recipes", but I would ask that anyone reading this who has other challenges, please e-mail them to me. Especially things that you think "cannot be done with Jam." You may be right, but I'll try my best to show how it can be done.

I'll end with a small anecdote: I frequently see people working on Haiku complain that "I would write a Jamfile, but I don't know how." My hope is that by the end of this series of articles, no one has to say that again.