A week or so ago, I needed to look into a program that was partially generated by the tools bison and flex (yacc/lex). The program was built with a makefile, since the BeIDE can't handle bison/flex as part of the build...or can it? I figured that since gcc and mwcc are actually handled as add-ons in the BeIDE, you should be able to handle the limited needs of bison/flex the same way.

The BeIDE comes with a shell tool add-on that lets you specify shell scripts to be run during steps of the build. This could actually be used for running bison/flex, but the shell script would be executed on every build. Since flex and bison produce files that I'll use as part of my source, I didn't want them running on every build. I could write the shell script to check whether flex/bison actually need to run, but it would still be less than optimal...and besides, it wouldn't give me enough to write about in this article.

As I was thinking about writing an add-on for flex and then for bison, I realized that I didn't want to write multiple add-ons at all. In fact, I wanted an add-on that would work for both flex and bison, and that could also be easily extended for mwbres, rez, or a host of other tools of this sort. But how to do this? The BeIDE add-on architecture allows a single add-on to specify more than one builder. Given that, I decided that I should be able to factor out a minimal amount of specific code for each tool. I went ahead and built a simple generic translator add-on that I then modified to handle these multiple tools. It can also be easily extended as I encounter more tools of these types that I would like to use.

The BeIDE supports editor and builder add-ons. I'll leave the editor add-on architecture for another article. The generic translator project can be found at ftp://ftp.be.com/pub/samples/beide/translator.zip. It would be more instructive to look at the source as you're reading the article. Unfortunately, R4 did not include the headers needed for writing BeIDE add-ons. Until you get your copy of R4.1, the headers and textual documentation for the add-on API can be found at

ftp://ftp.be.com/pub/samples/beide/Plugin_API.zip.

As the BeIDE starts up, it looks at all the add-ons in a known location and calls two important functions:

extern "C"
status_t MakePlugInView(int32 inIndex, BRect inRect, MPlugInPrefsView*&
outView)

extern "C"
status_t MakePlugInBuilder(int32 inIndex, MPlugInBuilder*& outBuilder)

These two functions return the MPlugInBuilder and MPlugInPrefsView that do the work and communicate with the BeIDE. The MPlugInBuilder is a wrapper for the external tool. It is used to specify what tool will be run, at what stage of the build the tool is used, what parameters to pass to the tool, and finally, how to interpret output from the tool. The MPlugInPrefsView provides the user interface for getting and setting options for running the tool. In some cases an add-on will provide multiple MPlugInPrefsViews.

In the functions above, the inIndex parameter allows me to create more than one view and more than one builder. Both functions will be called with an incrementing inIndex until they return something other than B_NO_ERROR. To make this example simple, I don't even support preference settings. That will be left as an exercise for the reader. Don't you hate it when authors write that? Now I know why they do. They don't want to do the work themselves.

Writing an MPlugInBuilder is a straightforward process. (especially when you don't worry about user preferences). There are about 16 member functions that must be overridden to create a concrete MPlugInBuilder class. Many of them can be written in just a few lines of code. Remember that I decided my add-on would support a variety of translation tools. This means that I need some way to customize a shared body of code for each individual builder. We are using C++, right? Then why not a class? The main work of the MPlugInBuilder is handled by GenericBuilder, which does as much work as possible before delegating specific needs to a BuildHelper it owns. MakePlugInBuilder wants to create a MPlugInBuilder object; I create a GenericBuilder in each case but pass each one a specific helper class. Here is my MakePlugInBuilder:

extern "C"
status_t
MakePlugInBuilder(int32 inIndex, MPlugInBuilder*& outBuilder)
{
  long result;

  switch (inIndex) {
    case 0:
      outBuilder = new GenericBuilder(new FlexHelper);
      result = B_OK;
      break;

    case 1:
      outBuilder = new GenericBuilder(new BisonHelper);
      result = B_OK;
      break;

    case 2:
      outBuilder = new GenericBuilder(new ResHelper);
      result = B_OK;
      break;

    default:
      result = B_ERROR;
      break;
  }
  return result;
}

I support three tools: flex, bison, and mwbres. But you can easily see how to extend this to other tools by writing a new BuildHelper and adding an additional case. To get a feel for writing an MPlugInBuilder, it would be best to go look at GenericBuilder.cpp.

The concrete BuildHelper classes (FlexHelper, BisonHelper, and ResHelper) override six member functions that deal with actions or information specific to each individual tool. I'll talk about just three of these methods here.

One of the main jobs of the MPlugInBuilder is to create the argument list for running the tool. If I had a MPlugInViewer, this is where I would retrieve and interpret the options in the BMessage that was created with the MPlugInViewer. The BeIDE asks the add-on to fill in a BList with a char* for each option. The Plugin_API documentation says that each string must be copied, since the BList adopts them and later will free them. Since free is used, I use strdup rather than new char[].

status_t FlexHelper::BuildArgv(BList &inArgv, const char *filePath)
{
  BString outputName;
  this->MakeOutputFileName(filePath, outputName);

  outputName.Prepend("-o");

  inArgv.AddItem(strdup(outputName.String()));
  inArgv.AddItem(strdup(filePath));

  return B_OK;
}

Notice that BuildArgv uses another helper method, MakeOutputFileName. Each BuildHelper class creates an output file name based on its input file name, with a different extension. For example:

flex: .l -> .cpp
bison: .y -> .cpp
mwbres: .r -> .rsrc

The new class BString can really be helpful for string manipulations. To create the output file we just take the input file and replace the extension:

FlexHelper::MakeOutputFileName(const char *filePath, BString &
outputName)
{
  outputName = filePath;
  outputName.ReplaceLast(".l", ".cpp");
}

What happens when tools have errors? All output is captured by the BeIDE and then sent to the add-on by calling

MPlugInBuilder::ParseMessageText(const char *inText, BList &outList)

ParseMessageText has the job of figuring out what is in the message and creating ErrorMessages that are then added to outList. Any ErrorMessages in outList, will be displayed in the Message Window. It's very simple to create "text only" error messages. Just take each line of text and stuff it in a new ErrorMessage. However, it's much nicer to communicate the file and line information so that the user can navigate from that error message to the spot in their source with the problem.

Looking at the type of tools that I wanted to support, I saw they had a similar style for error messages. Here is an error from flex, bison, and mwbres:

"test.l", line 123: EOF encountered inside an action
("test.y", line 84) error: invalid $ value
File test.r Line 6 # parse error near 'include'

Since I didn't want to write N different parsers, I had GenericBuilder::ParseMessageText be in charge of breaking the message up into individual lines and passing each individual line to the BuildHelper. (If a tool has multiple line errors, GenericBuilder::ParseMessageText will have to be overridden.) The base class BuildHelper provides a few methods to parse lines that follow a format like this:

something <filename> something <linenumber>
something message

(Here I am talking about flex and bison and I write the grammar like that...sheesh!) This means that the different concrete BuildHelper classes can implement the parsing in one line. All they have to do is specify the tokens (the "something" stuff) before and after the file name and line number.

Once the add-on is built, how do you use it? The add-on needs to live in the BeIDE directory under plugins/Prefs_add_ons. Once installed, use the Settings/Target preferences to connect file types and extensions to the tools. To build my project that used both flex and bison I added the following targets:

Extension: l
Tool: flex
Flags: Precompile Stage

Extension: y
Tool: bison
Flags: Precompile Stage

I then added the foo.yy.l and foo.tab.y files to the project. Since flex and bison are used to produce files that are then compiled as part of the project, I also added the files they produce: foo.yy.cpp and foo.tab.cpp to the project. The BeIDE has now been successfully enhanced to work with flex and bison. Once I wrote and debugged the GenericBuilder and FlexHelper, it took me about 5 minutes to write and test the BisonHelper and ResHelper. I'm sure many developers will think up lots of creative uses for the BeIDE add-on API.

My father used to tell me stories about how he tried to create systems that took care of themselves so well that no human intervention was needed. He's had at least one job that, according to him, he succeeded getting to run itself so well that he had nothing to do and left out of boredom.

I try to do the same, although even when I think I have something taking care of itself, there are always more changes to fold in, new features to test, new conditions to take into account. So far, I can't foresee any time in the future when what I do could get boring.

Baron Arnold told you that a newer, better release of the InputRecorder would be available in March with my newsletter article Be Engineering Insights: Worn Out Rhymes. This is so. He also mentioned that it would contain source code. This is also true.

The source has been cleaned up a lot since last month, along with the user interface. A memory leak in the input filter was fixed. And now you may even be able to run multiple recorders at the same time, though that hasn't really been tested much.

While the original author used BResources to store the record data, I chose another format—a simple, plain file structure. I implemented all three forms in my MessageWriter class, so you can explore and play with all the options. (The attribute- and resource-based methods need to be modified slightly and recompiled, but they are implemented.)

I decided not to use attributes to store the input records for several reasons. The first one is that I don't really need to, since I don't need to randomly locate input events in the file (I treat it as a simple stream). The second is that there's currently no easy way to tell how large a file's attributes are (with InputRecorder the files can be large). Finally, a simpler file format is easier to transfer. (Zip files are limited to 64k of attributes per file, which can easily be exceeded when recording input. The Zap format doesn't have this restriction, but isn't as common.)

A reminder: We do not recommend using BResources as general purpose data storage; we recommend using attributes instead. BResources were designed specifically for storing application resources.

One reason for the delay in getting the InputRecorder (especially as source) to you has been that some input server-related parts aren't quite finished. Some device and filter elements may be changing, breaking compatibility, at some future date, in order to improve them. Some of these things may still be subject to change, although I've convinced Those On High that it would be great let developers have access to the InputRecorder.

The source this week is amazingly straightforward. The headers for the input filter stuff are in /boot/develop/headers/be/add-ons/input_server/, where there are three files: InputServerDevice.h, InputServerFilter.h, and InputServerMethod.h.

Input scripting uses only two of the three input_server add-ons. It uses a device to write input messages to the input_server, and a filter to read input_messages from the input_server. Both use ports to talk to the user-level program.

It should be clear to some of you that the design of the InputRecorderDevice is such that it will pass along any message its port receives directly to the input_server. This means, for example, that you could create plug-ins for your favorite scripting language that let you arbitrarily script input.

Here in the QA Lab we use a simple, straightforward little tool to do just that—I'll tell you about it in a later article. It contains within it the details of the kinds of messages devices can send. It contains many arcane facts that in some cases I just guessed about, based on how our devices appear do it and what did and didn't work. That's why it needs to be looked at carefully before it's released, to ensure that all the information it contains is accurate.

As an example, one of my favorite scripts that use this tool is called 'normalactions' because it contains things most everyone does at the same time. It utilizes four workspaces: one to send/receive e-mail to/from itself; one to compile and run the sample code; one to browse the web with NetPositive; and one to periodically run audio, which continues playing while everything else is going on. It uses a couple of helper programs to do assist in the synchronization issues—things like ensuring that the active application is what I expect it to be.

So you see, the long trek toward my dream of a fully automated QA, where all the QA engineers sit back drinking coffee and watching subprocesses kill machines and fetch engineers is on the way. Not in the near future, but it's coming slowly. Maybe one day QA will become so boring that I'll start looking for a new line of work... though somehow that doesn't seem likely.

I heard a rapidly approaching Canyonero, the deadliest of sport utility vehicles, in the parking lot yesterday, and once again recognized the sound of doom. And sad to say, I was a convenient target for it.

"You know, those audio sample apps you're coming up with are pretty lame," spat Morgan le Be, BeOS hacker and mistress of the black arts, bringing her smoking 'Ero to a halt against the red curb. "Who plays only one sound at a time these days?"

"Well, you can hook up your sounds to the system mixer..."

"Uh-uh, buster—I want a mixer, just like the system mixer, but for my own private use."

The gauntlet was down, and for the first time, I knew how to parry Ms. le Be's attack. Just a few minutes' explanation, and she was satisfied.

And so I say to you, O, True Believers: Heed the dark magick of the Media Kit. Somewhere in the bowels of the Media Add-On Server dwells a dormant mixer node, as awesome in its slumber as a hibernating polar bear, waiting for you to harness its power.

status_t err;
dormant_node_info mixer_dormant_info;
int32 mixer_count = 1;

err = BMediaRoster::Roster()->GetDormantNodes(
    &mixer_dormant_info, &mixer_count, 0, 0, 0,
    B_SYSTEM_MIXER, 0);

media_node mixer_node;
if (err == B_OK) {
    err = BMediaRoster::Roster()->InstantiateDormantNode(
        mixer_dormant_info, &mixer_node);
}

About a year ago, at our developers' conference in Santa Clara, we formally unveiled the first version of the BeOS for the Intel architecture. This year, on April 9-10, we'll have our first Media Kit conference—an even more momentous event than our 1998 entry into the world of x-86 hardware.

Not that we didn't enjoy the occasion. For me, developer conferences are always a happy time. I get to meet the people who, in a very real sense, "make" the platform. The people who commit an incredible amount of time, energy, and creativity to our product. We get vigorous feedback, encouragement, and great ideas.

I've often thought that, in our business, you know you've done something right when your platform is perverted. By which I mean—before other interpretations are suggested—that programmers or normal people use your product in ways you hadn't thought of. One reason you hadn't thought of them is the myopia you get from seeing your product all the time. Another reason is that developers represent so many different areas of interest and expertise—whereas, to a hammer everything looks like a nail.

The best part of having your platform "perverted" is seeing that the product is rich enough for others to extract more uses than you'd initially foreseen. For example, the creators of the Macintosh didn't think of it as the engine of what became Desktop Publishing, DTP. The concept had no meaning at the time, just as home computing was ridiculed as a fad. In retrospect it all seems logical and preordained: if you have a great user interface, good fonts, and an easy-to-use laser printer, you'll create DTP.

Ah, but it wasn't so obvious beforehand, and retroactively amused witnesses remember contorted positioning statements suggested by the experts of the day. One of the best was "Graphics Based Business Systems." I trust that I'll be equally amused and humbled someday by a few of my own suggestions for our little company.

So, we had a good time last year, complete with an enthusiastic speech from Intel's Claude L'Eglise and a quasi slip-up from yours truly recounting a recent meeting with Andy Grove. At the time, we had an agreement with Intel to keep their investment in Be confidential—it was disclosed last November at Comdex. As I proceeded to describe a meeting where I came to thank Andy for his support and to give him a couple of black Be T-shirts—and photocopies of Intel's stock certificates fresh from the closing one hour earlier, my colleagues grew a little agitated. I realized the blunder I was about to make and abruptly moved to another topic.

This year, we'll reveal the secrets of the Media Kit. OS platforms are sometimes compared to musical instruments, with rendering power as well as expressive power. In our case, to stretch the metaphor a bit, we could say that basic BeOS features such as symmetric multiprocessing, pervasive multithreading, and a 64-bit journaled file system give it rendering power. The Media Kit, on the other hand, gives the BeOS expressive power, the ability to write new music—that is, new applications that aren't necessarily as easy to write, or perhaps impossible, on other sets of APIs.

We've often referred to the BeOS as the Media OS. The Media Kit is the means by which what was the promise of new OS technology can now be expressed in new applications that address both the creation and the consumption of digital media.

I look forward to meeting you at this conference and to seeing surprising new applications down the road.