Seems as if every other week the Engineer Insight article discusses byte swapping. Must be that time because that's my topic today. Up until now we've always written about what one should do, in theory. Finishing up the first Intel release has given us all more practical experience and I'd like to share some of it with you.

There are several sections of the API, including the BMessage class and file attributes, where dealing with endian issues might effect your coding habits. I'll discuss the BMessage class in this article and leave the file attributes case for a later article.

The BMessage class does lots of work for you in terms of byte ordering. If you are just reading and writing standard data types to BMessages then you needn't fret about byte ordering. By "standard" I mean all the data types (with three exceptions) defined in support/TypeConstants.h.

For example, if you have code like the following:

// code that writes your data into a message
msg->AddString("label", text);
msg->AddInt32("weight", weight);
msg->AddMessenger("return_ref", some_messenger);

// let's save this message to a file
msg->Flatten(&some_file);

...
// That file can now be moved to another platform. What
// happens isn't under your program's control.
...

// let's read that message out of a file
msg->Unflatten(&some_file);

// code that reads your data out of a message
msg->FindString("label", &text);
msg->FindInt32("weight", &weight);
msg->FindMessenger("return_ref", &some_messenger);

In the above code you don't have to worry about byte ordering. If the file is moved across platforms the system (the Flatten/Unflatten calls themselves) will handle any byte ordering issues. The same is true if the message is flattened into any sort of buffer and then moved to another platform. An example of this would be if you flattened a message and saved it in an attribute of some file.

The three exceptions mentioned above are B_ANY_TYPE, B_RAW_TYPE, and B_OBJECT_TYPE. The first type, B_ANY_TYPE, isn't really a type. The latter two types have arbitrary and undefined formats. They can be anything, so there is no way that the system can properly swap bytes between Intel and PPC platforms.

Also, it is the developers responsibility to swap any custom data types that you add to a message. Here's an example:

struct my_info {
    int32  weight;
    char   bx[2];
    float  percent_done;
};

...
my_info  info;
// info gets filled in with data

// now you add data of this type to a message
msg->AddData("data", 'myin', &info, sizeof(my_info));

Now if msg is flattened and moved to platform with differing endianness you'll have to deal with byte swapping:

my_info  info;
void     *raw;
ssize_t  size;

msg->FindData("data", 'myin', &raw, &size);
// raw points to raw/unswapped data, not necessarily a
// valid my_info structure. If endianness of this platform
// differs from source platform then work needs to be done.

One option is to add a field to the my_info struct specifying the endianness of the data as written. The other option is to redo the data structure so that it is endian insensitive. For more details on dealing with vanilla C structures such as my_info see Brad Taylor's Engineering Insights articles in issue 2-#9 and issue 2-#45 of the Newsletter:

Another alternative that I'll discuss in detail is using the BFlattenable class. Using this class can make handling custom data structures somewhat easier. Let's redo the above example of a C structure (my_info) using the BFlattenable class:

struct TMyData : public BFlattenable {
    // here's my data
    int32  weight;
    char   bx[2];
    float  percent_done;

virtual status_t Flatten(void *buffer, ssize_t size) const;
virtual status_t Unflatten(type_code c, constvoid *buf,
                           ssize_t size);

// overrides for other functions are omitted to keep example small
};

The BFlattenable class gives you nice bottlenecks for moving data in and out of BMessages, files, or file attributes. These bottlenecks give you a framework for dealing with byte ordering. As always you have two options for dealing with this issue:

• Write out data in natural order and swap when needed on read.

• Write out data in canonical format and swap when needed on read.

status_t TMyData::Flatten(void *buffer, ssize_t size) const
{
  char *p = (char *) buffer;
  // need to save a flag indicating the current platform. So
  // we remember if this platform is little endian. The
  // FlattenSize function must account for this extra space.
  *p++ = B_HOST_IS_LENDIAN;

  // now save the rest of the data in natural format
  *((int32 *)p) = weight;
  p += sizeof(int32);

  *p++ = bx[0];
  *p++ = bx[1];

  *((float *)p) = percent_done;

  return B_OK;
}

status_t TMyData::Unflatten(type_code c, constvoid *buf,
  ssize_t size)
{
  const char  *p = (const char *) buf;
  uint8       endian;
  bool        must_swap;

  // read the endian flag saved by the Flatten call
  endian = *((uint8 *) p++);

  // compared the saved value with current value
  must_swap = (endian == B_HOST_IS_LENDIAN);

  // must_swap will only be true in the source and
  // destination have different byte ordering

  // now simply read out the data
  weight = *((int32 *) p);
  p += sizeof(int32);

  bx[0] = *p++;
  bx[1] = *p++;

  percent_done = *((float *) p);

  // now swap the data if needed
  if (must_swap) {
    weight = B_SWAP_INT32(weight);
    percent_done = B_SWAP_FLOAT(percent_done);
    // don't need to swap the bx chars.
  }
  return B_OK;
}

status_t TMyData::Flatten(void *buffer, ssize_t size) const
{
  char *p = (char *) buffer;

  // Decided to always write the data in little endian
  // format. So we'll write data using the HOST_TO_LENDIAN
  // macros.

  *((int32 *)p) = B_HOST_TO_LENDIAN_INT32(weight);
  p += sizeof(int32);

  *p++ = bx[0];
  *p++ = bx[1];

  *((float *)p) = B_HOST_TO_LENDIAN_FLOAT(percent_done);

  return B_OK;
}

status_t TMyData::Unflatten(type_code c, constvoid *buf,
  ssize_t size)
{
  const char  *p = (const char *) buf;
  uint8      endian;
  bool        must_swap;

  // We know that the data was saved in little endian
  // format. So read in the data using the appropriate
  // macros from support/ByteOrder.h

  weight = B_LENDIAN_TO_HOST_INT32(*((int32 *) p));
  p += sizeof(int32);

  bx[0] = *p++;
  bx[1] = *p++;

  percent_done = B_LENDIAN_TO_HOST_INT32(*((float *) p));
  return B_OK;
}

And there you have it. Hopefully this helps clarify how to use BMessages in a multi-endian world. It's the PC thing to do.

One of the great features of the BeOS is the ability to query information about files' attributes. This is both useful in programming and for the general user wanting to find files that meet certain criteria. The mechanism behind queries is a series of attribute indices that list the files that can be searched given a particular attribute and value.

Indices are volume-specific, meaning that different volumes connected to your machine can have different indexed attributes. Furthermore, an attribute index is only updated whenever a matching attribute is written. So any files on the volume before the index is created are not listed. The BeOS does not step through and examine all of the files on the volume every time an index is created (that could be a lengthy process on a large volume.)

This week's sample code allows you to selectively re-index files to make sure that all of their attributes are up-to-date in the attribute indices. Indexer is comprised of a BApplication class that examines entry_refs, various InfoView classes that process and display information about these refs, and some global indexing functions. There is an InfoView for each type of entry: File, SymLink, Directory and Volume. These classes combine to give you detailed information about each dropped ref in addition to re-indexing any referenced files that need it.

When a reference to a file, directory, symbolic link, or volume enters Indexer, more often than not it arrives in the form of an entry_ref. Full details about entry_refs can be found in Entry.h, but quickly an entry_ref specifies the location of a potential file on disk by specifying a device number, a directory number, and a name. Paths also specify location in a similar manner, and throughout Indexer you will see various translations between these two entry formats. One thing to note, however, is that entry_refs should not be stored on disk, as the device number can change between boots. entry_refs saved to disk, and then copied in a file to another disk will just not work when unpacked on the other side. Use paths if you need to save the location of an entry to disk.

When items are selected and dropped on Indexer's icon or one of its windows from the Tracker, the refs come bundled in either a B_REFS_RECEIVED or B_SIMPLE_DATA message. Both of these messages are similar: The real meat of the message is bundled into the "refs" members. Parsing the message to get the references is the first task.

void Indexer::RefsReceived(BMessage *msg)
{
  uint32     type;
  int32      count;
  entry_ref  ref;

  msg->GetInfo("refs", &type, &count);
  if (type != B_REF_TYPE)
    return;
  for (int32 i = --count; i >= 0; i--) {
    if (msg->FindRef("refs", i, &ref) == B_OK) {
      EvaluateRef(ref);
    }
  }
}

Each entry_ref pulled out of the message is passed along to EvaluateRef() to start our real work. Indexer also accepts arguments from the command-line which are processed in its ArgvReceived() function. Any paths present are translated into entry_refs with the get_ref_for_path() function, and the resulting refs are also passed to EvaluateRef().

status_t Indexer::EvaluateRef(entry_ref &ref)
{
  struct stat st;
  BEntry entry;

  if (entry.SetTo(&ref, false) != B_OK)
    return B_ERROR;
  if (entry.GetStat(&st) != B_OK)
    return B_ERROR;
  if (S_ISLNK(st.st_mode))
    return HandleLink(ref, st);
  else if (S_ISREG(st.st_mode))
    return HandleFile(ref, st);
  else if (S_ISDIR(st.st_mode)) {
    BDirectory dir;
    if (dir.SetTo(&ref) != B_OK)
      return B_ERROR;
    if (dir.IsRootDirectory())
      return HandleVolume(ref, st, dir);
    else
      return HandleDirectory(ref, st, dir);
  }
}

The first thing we want to do is determine exactly what type of entry_ref we have gotten. To do this we need to get a stat structure. The stat structure provides a host of information about entries, including the flavor of the node, and creation and modification times. To get the stat we need an instance of the BStatable class, and BEntry is an easy one to get. As we want to be able to tell if we have a symbolic link, we inform BEntry::SetTo() to not traverse any links it finds.

We can then proceed to get the stat and discover what type of entry we have. If we discover a file or a symlink, we simple start to handle them. If we find a directory we need to do a little bit more examination to determine whether we have a normal directory or a volume. Volumes are represented as directories that live in the root of the file system. To determine which type we have we create a BDirectory and ask it if it is a root directory. If it is, we handle it as a volume, if not we handle it as a directory.

As Indexer is a fairly large bit of sample code, I won't go into all of the details of processing each entry type. Instead I'll simply point out some of the more interesting bits, and leave you to examine the rest of the sample code in more depth later.

status_t Indexer::HandleVolume(entry_ref &ref,
  struct stat &st, BDirectory &dir)
{
  BVolumeRoster vol_roster;
  BVolume       vol;
  BDirectory    root_dir;
  dev_t         device;

  while (vol_roster.GetNextVolume(&vol) == B_NO_ERROR) {
    vol.GetRootDirectory(&root_dir);
    if (root_dir == dir)
      break;
  }
    // build the info window and the like
  ...
}

extern status_t
  get_attribute_indices(dev_t device, BList &index_list)
{
  DIR           *index_dir;
  struct dirent *index_ent;

  index_dir = fs_open_index_dir(device);
  if (!index_dir)
    return B_ERROR;
  while (index_ent = fs_read_index_dir(index_dir)) {
    char *text = strdup(index_ent->d_name);
    index_list.AddItem(text);
  }
  fs_close_index_dir(index_dir);
  return B_OK;
}

void dInfoView::TraverseDirectory(BDirectory &dir)
{
  entry_ref ref;

  while(dir.GetNextRef(&ref) != B_ENTRY_NOT_FOUND) {
    EvaluateRef(ref);
  }
}

void dInfoView::EvaluateRef(entry_ref &ref)
{
  struct stat st;
  BEntry entry;

  if (entry.SetTo(&ref, false) != B_OK)
    return;
  fEntryCount++;
  entry.GetStat(&st);
  if (S_ISLNK(st.st_mode))
    fLinkCount++;
  else if (S_ISREG(st.st_mode)) {
    if (ref.device != fRef.device) {
      fInvalidCount++;
      return;
    }
    fFileCount++;
    BNode node;
    if (node.SetTo(&ref) != B_NO_ERROR) {
      fInvalidCount++;
      return;
    }
    status_t status = B_OK;
    status = reindex_node(node, *fIndexList);
    if (status == INDEXED)
      fIndexed++;
    else if (status == PARTIAL_INDEXED)
      fPartialIndexed++;
    else
      fNotIndexed++;
  }
  else if (S_ISDIR(st.st_mode)) {
    BDirectory dir;
    if (dir.SetTo(&ref) != B_OK) {
      fInvalidCount++;
      return;
    }
    if (dir.IsRootDirectory())
      fInvalidCount++;
    else {
      fSubDirCount++;
      TraverseDirectory(dir);
    }
  }
}

extern status_t reindex_node(BNode &node, BList &index_list)
{
  attr_info info;
  status_t  status = B_OK;
  int32     to_be_indexed = 0;
  int32     indexed = 0;
  int32     not_indexed = 0;
  int32     size = 1024;
  char      *value = (char *) malloc(size * sizeof(char));

  //rewrite all of the appropriate attributes
  for (int32 i = 0; i < index_list.CountItems(); i++) {
    char *attr = (char *) index_list.ItemAt(i);
    if (node.GetAttrInfo(attr, &info) == B_OK) {
      to_be_indexed++;

      // adjust the size of our static buffer if necessary
      if (info.size > size) {
        value = (char *) realloc(value, info.size);
        size = info.size;
      }

      if (node.ReadAttr(attr, info.type, 0,
                        value, info.size) > 0) {
        if (node.WriteAttr(attr, info.type, 0,
                           value, info.size) > 0)
          indexed++;
        else not_indexed++;
      }
      else not_indexed++;
    }
  }

  free(value);
  value = NULL;

  if (to_be_indexed > 0) {
    if (indexed > 0) {
      if (not_indexed > 0)
        return PARTIAL_INDEXED;
      else
        return INDEXED;
    }
    else return NOT_INDEXED;
  }
  else return NOT_INDEXED;
}

void fInfoView::GetAttributes(BNode &node, BList &list)
{
  char attr_buf[B_ATTR_NAME_LENGTH];

  node.RewindAttrs();
  while (node.GetNextAttrName(attr_buf) == B_NO_ERROR) {
    char *string = strdup(attr_buf);
    list.AddItem(string);
  }
}

Finally, some last notes, suggestions, caveats and plans. There are a series of volumes that cannot be accessed through the Tracker (such as /dev or /pipe) that information can be gotten about through the command-line. Take a look at a couple of them and see what can be seen (although not much will be seen in the virtual file systems in the way of files, the info windows usually fail.)

I also want to note that the code for displaying the information is probably not how the interface guys would want to see it done. Use the sample code to look at the Storage Kit classes, not as a pristine example of Interface Kit code.

I'll continue to update Indexer in the future, and I would love to hear of any improvements you make. Upcoming features might include code to remove and create indexes, as well as a look into whether passing BEntrys or entry_refs is more efficient.

In the meantime, Indexer can be found at: ftp://ftp.be.com/pub/samples/storage_kit/obsolete/Indexer.zip.

I'll be back next week with Victor Tsou from the Doc Team. We'll be filling you in on many of the things you'll want to know about the upcoming Release 3 release of the BeOS for Intel. Until then...

This isn't about my college-bound son and the education of our auto insurance company. It's an attempt to put some thoughts on electronic paper concerning the sea of driver trouble ahead of us.

Once upon a time, we had totally proprietary hardware. This was at the very beginning of the company, when multiprocessor hardware was the province of high-end servers and workstations, not down to the $2000-$3000 level as it is today (see the Ming specials at http://www.be.com/support/guides/ming-specials.html).

In the beginning, we were in the position of having to design I/O cards and write the drivers. Now though, we are in the retroactively obvious situation where we can benefit from other people's investment in chipsets, motherboards, and I/O devices, and focus our efforts on the OS.

But the abundance of riches comes at a price: how can we get drivers to support this wonderful wacky world of PC add-ons? Put another way, some observers have attributed OS/2's failure to a paucity of drivers. Customers eager for an alternative to Windows were disappointed to find that their interface cards and peripherals weren't always supported by IBM or by the hardware add-on manufacturer. Are we going to disappoint hopeful BeOS users in the same way? Are we going to bleed to death attempting to write drivers or cajoling vendors for support?

The hidden pivot in the argument is the comparison to OS/2. If indeed, we come out of nowhere representing ourselves as a general-purpose OS ready for mainstream consumption, then we face an impossible challenge. Put another way, OS/2 positioned itself as an alternative to Windows and failed to deliver what turned out to be an impossible proposition, better DOS than DOS, better Windows than Windows.

The BeOS is not a general-purpose OS, it is a complement or a supplement to Windows. The BeOS coexists with the general-purpose Windows as a specialized OS, just as Linux does. Fine, but what does it mean for the sea of drivers issue?

First, it means we have to keep setting expectations, just as we did in earlier days with tongue-in-cheek but serious surgeon general warnings: this product is unfit for consumption by normal humans. It still is and, just like Linux, will be for a long time—if not for ever.

As an industry such as ours mature, it will diversify, segment and specialize, and the thought that any OS has to be or do "everything" doesn't have to apply. More specifically, as we represent ourselves as the media OS, best fit for real-time WYSIWYG, the reality-based positioning, a Be, Inc. exclusive, translates into a focus set of hardware targets. This, in turn, translates into a finite set of drivers. That's what we have to define, that's what we have to communicate.

BeDevTalk is an unmonitored discussion group in which technical information is shared by Be developers and interested parties. In this column, we summarize some of the active threads, listed by their subject lines as they appear, verbatim, in the mail.

To subscribe to BeDevTalk, visit the mailing list page on our web site: http://www.be.com/aboutbe/mailinglists.html.