The first thing to understand is the difference between a "file format" and a "codec." A file format is nothing more than a container to store data. A file format contains additional information about the movie (such as its size, the frame rate, etc.) but the file format does not typically specify that the audio or video must be encoded in a specific way. A codec, on the other hand, is a way of encoding audio or video data. How a piece of data is encoded is (typically) not dependent on the file format it's stored in.

As an example, consider an AVI file. An AVI file can store audio and video encoded with numerous different encoders. Another way to say it is that an AVI file is a way to store audio and video in a file, but you can encode that audio/video data any way you wish. Encoding audio or video data transforms it from a very high- bandwidth stream of raw data into a stream of compressed data. While you could do anything with a stream of compressed data, it is typically stored in a common file format such as AVI or QuickTime.

The following table lists the media file formats that BeOS supports:

There are many other file formats (such as FLI/FLC, ASF, etc.) but currently the BeOS does not support them.

Each of the file formats listed in the table (except for MPEG) has a similar structure, although each one uses its own terminology to describe the specifics. The basic idea is that a file is made up of "chunks" of data and if you don't understand a chunk you can skip over it. The way this is typically done is with a 4-byte identifier to describe the type of chunk and a 4-byte (or sometimes 8-byte) size. A diagram helps to illustrate:

file offset   data
------------+---------------------+
          0 |  0x000012b8         |
------------+---------------------+
          4 | 'moov' (0x6d6f6f76) |
----------------------------------+

This represents the first 8 bytes of a QuickTime file. QuickTime stores its chunks with the size first and then the identifier (and as one would expect, AVI does it the opposite way). The identifier is known as a "four-character code" or "fourcc." A fourcc is often a mnemonic or clever name that makes it easy to identify when looking at a hex-dump of a file. In this example the ASCII characters 'moov' have the value 0x6d6f6f76 when treated as a 4-byte integer (on a big-endian system).

The size of this "moov" chunk is 0x12b8 bytes long (4792 in decimal). Because we know the size, even if we don't know anything about a moov chunk we can correctly skip over it and parse the next item (of course, if you skip the moov chunk you won't be able to do very much!).

The concept of a chunk'ed file with an identifier and size for each chunk is a simple but good way to store data that needs to be parsed by many different types of programs. Having well-identified chunks and the ability to skip over them if they are unknown makes for a robust file format.

In addition to a linear series of chunks in a file, both QuickTime and AVI allow a chunk to contain other nested chunks. That is, certain chunk identifiers indicate that the chunk is a container that has other chunks within it. The size of a container chunk is the sum of all the chunks inside it. For example, an AVI file may contain a 'LIST' chunk that contains other chunks. The size of the LIST chunk would allow you to skip over it if you wanted. An example in a QuickTime file would be the 'trak' chunk, which contains many other chunks that describe the media track.

To get a better feel for the layout of a file format I suggest that you fire up DiskProbe and scan through some media files. You'll quickly see how they're laid out. It's also helpful to have the file format spec handy for serious perusal (a good source of specs is http://www.wotsit.org/, or just fire off a search on http://www.google.com/).

The MPEG file format is actually not one but three formats. You can have a raw video stream, a raw audio stream, or a "system" stream (which mixes multiple channels of audio and video together). The MPEG file formats are designed to be a continuous stream that you can start receiving and begin decoding after you find a sync token. With streaming (or broadcasting if you prefer) as a major design goal, MPEG does not have many of the features that other media file formats have.

The different MPEG file formats all have some common deficiencies. The first is that none of them have a good header identifier that is even moderately unique. Instead, MPEG files have header identifiers like 0x1ba (system stream), 0x1b3 (video stream), and 0xfff (audio stream). These values are atrocious identifiers because they are far too easy to encounter in chunks of non-MPEG data.

Next, the structure of MPEG streams is tightly bit-packed and does not lend itself well to easy parsing. For example, to extract the clock value from a video stream, you have to do this:

t = ((((bigtime_t)(stream[0] & 0x0e)) << 30) |
     (((bigtime_t)(stream[1] & 0xff)) << 22) |
     (((bigtime_t)(stream[2] & 0xfe)) << 15) |
     (((bigtime_t)(stream[3] & 0xff)) << 8)  |
     (((bigtime_t)(stream[4] & 0xfe)) << 0));

As Dave Bort would say: Crazy.

Another area that poses big problems when dealing with MPEG is that a lot of MPEG data is not well formatted. There's what the spec says and then there is what's done. This is the kind of problem you come to expect when dealing with data files generated by lots of programs, but MPEG seems to have the problem in spades. The highly varied interpretations of what the spec says make trying to parse MPEG files an interesting job.

Getting beyond my childlike whining, MPEG does have a notion of chunks of data, although the size of each chunk is not usually specified as a 4-byte value in the stream of data. Instead, the size is either implied or determined by continuing to read the data until you find the start of the next chunk.

The inherent streaming nature of MPEG means that there are no global indices of file positions to frames or to where the sync points are in the file. This complicates life for BeOS because a program often wants to know how long a media file is so it can display a properly sized slider or progress bar. To make that work, BMediaFile pre-parses the MPEG video stream so that it knows where sync points are in the file and roughly how many frames there are. The disadvantage of doing this is that it can take a long time to read through all the data first before decoding any of the frames. Our current approach of pre-parsing the entire MPEG stream is not something we're happy with and we plan to rewrite it.

From a media programmer's standpoint the important lesson here is that the number of frames returned from BMediaTrack->CountFrames() is only an approximate number. BMediaTrack->Duration() is also only an approximate value. This may be more or it may be less due to the imprecise nature of file formats like MPEG. Therefore, when you write a piece of code to process all the frames in a file you should write the loop this way:

while(1) {
    status_t err;
     err = track->ReadFrames();
    if (err == B_LAST_BUFFER_ERROR) // all done!
        break;
    else if (err == ...)            // handle other errors
        ....
    else if (err == B_NO_ERROR)
        render the frames...
}

or some variant of that approach (a good example is what the tcode program does). You should NEVER code a for loop that counts up to the number of frames returned by CountFrames().

Another interesting twist is that in QuickTime and some AVI files a single frame may be displayed for longer than the frame rate of the file. That is, a single frame is displayed for several seconds instead of repeating the frame's data in the file. Therefore, if you need to display a scroll bar or slider, the Duration() of the track is a better estimate of how long it really is. The correct thing to do is to use Duration() as an estimate and dynamically adjust scrollbars and sliders until you get a B_LAST_BUFFER_ERROR.

It's important to remember that CountFrames() and Duration() are not guaranteed to be precise and thus should NOT be used as an upper limit in a for or while loop. This imprecision is certainly frustrating for programmers but it's a fact of life that needs to be dealt with.

Now that we understand the concept of chunks, it's time to talk about what's in some of those chunks. In QuickTime and AVI there is a notion of a media track that contains audio or video data. In a QuickTime file the 'trak' chunk indicates a media track. In AVI there is the 'avih' chunk, which contains global information about the entire movie, and the 'vids' chunk that contains details about the video track.

The QuickTime file format is much more flexible than AVI and allows any number of tracks, each with its own frame rate and data type. QuickTime tracks have extensive information about the presentation of the track's contents. Video tracks even have a 3x3 transformation matrix that can be applied to the video before compositing it into the final presentation!

Audio tracks in all file formats have the necessary information for playing the audio: sample rate; size of the samples (8 or 16 bit); mono, stereo, or multichannel, etc.

The track header also specifies how the track's data is encoded. For example, Cinepak-encoded data is identified with a fourcc of "cvid." Indeo-5 encoded data is identified with a fourcc of 'iv50'. You can find a very complete list of fourcc codes from AVI files at:

http://www.fourcc.org/

Audio track information usually uses a single integer to identify which codec to use. In a WAV files for example, raw audio is indicated by the integer value 1, and MS-ADPCM (Microsoft's Adaptive Pulse Code Modulation) is identified by a 2. AVI uses the same identifiers as WAV. QuickTime uses its own set of fourcc's, of course.

Another good reference for AVI/WAV file codec id's is RFC-2361:

http://www.faqs.org/rfcs/rfc2361.html

In addition to the information about a track, a file format will also have information about where in the file the data can be found. Typically this takes the form of a table that indexes frames and file positions. Using this information, you can extract the stream of data that makes up a particular track. In the abstract, most media files look something like this:

+--------+------------+------------+-----+------------+------
| header |   track 1  |   track 2  | ... |  track 1   | ...
|  info  | data chunk | data chunk |     | data chunk |
+--------+------------+------------+-----+------------+------

What you effectively have are interleaved chunks of data for each track. In practice, you usually have a single audio track and a single video track. Because the data rate of audio is far less than video, you often have a single audio chunk followed by several video chunks, followed by another audio chunk, etc.

If we were to split out the data chunks for each track and put them together, we would have a contiguous stream of data for each track. Again, it's important to remember that the file format doesn't care what the data is in a track or how it's encoded, it only cares about the ordering of the chunks of data and what frame (or frames) they correspond to.

Now that we know what a track is, let's discuss how we get from the bits of data that belong to a track to something that we see on screen (or hear). If you're programming on BeOS you simply use the BMediaTrack object to get access to the data that makes up a track. The BMediaTrack object will let you access the decoded data for the track with ReadFrames(), or you can choose to get the encoded data with ReadChunk().

What's going on behind the scenes is that when you want to decode a frame from a media file, you have to first look up in the index where that frame's data lives on disk. Once you know that, you can seek there, read the encoded data into memory and hand it off to a decoder. This process (and doing it efficiently) is the bulk of the work that BMediaFile and BMediaTrack do.

The index that maps frames to on-disk locations can be very simple, as it is in AVI, or much more complex, as it is in QuickTime. In QuickTime each frame can have a different duration, can appear anywhere in the file, and QuickTime index/mapping tables support complex arrangements of the data (presumably to allow it to be optimized for slow devices such as cd-roms). In most cases the QuickTime approach is massive overkill. Aside from the ability to change the duration of individual frames, the extra functionality that QuickTime offers is almost never used.

Above we described conceptually how we get the encoded data out of a file format. Once we've retrieved the encoded data for a particular frame, we hand it off to a decoder. The decoder's job is to convert that encoded video into a decoded format that the client (the person calling BMediaTrack->ReadFrames()) can deal with. Most simple video compression formats read the chunk, decode it into the output buffer, and are done. More complex codecs have state information that they must maintain from frame to frame. Indeo5 and MPEG are two examples of encodings that require lots of state information to be able to decode a frame.

Let's now delve into a few more details about different video and audio codecs. This is the list of encoded media formats supported by the upcoming BeOS Release 5. There are, of course, many more encodings out there—these are just the ones we support:

Before we jump into the details of each video codec, let's step back for a second and discuss some general terminology. A video codec encodes data using an algorithm to transform an uncompressed frame or field of video. The algorithm may operate on each frame independently or it may require information about previous frames. If an encoder encodes each frame independently, then when you want decode a frame you can do it without having to look at any other frames. In this situation each frame is treated as a keyframe (i.e., it can be decoded completely without requiring other data) and, therefore, you can seek to any frame in the file (aka: a perfectly seekable file).

If the encoding algorithm requires information about previous frames to encode the current frame, then when decoding the data you must first decode the prior frames leading up the frame you want. This type of encoder will output a "keyframe" every so often as a sync point in the file. Remember—a keyframe can be decoded independently of any other frames. So for example, the output of an encoder could be this: 1 keyframe, 10 delta frames, 1 keyframe, 10 delta frames, etc. That means that you can't seek to any arbitrary frame; you have to seek to a keyframe and then play forward to reach the frame you want.

Most sophisticated encoding algorithms almost always have keyframes every 10-20 frames. The algorithms take advantage of the temporal cohesion between each frame in video data. Keyframes are the reason that you can not seek to an arbitrary position in most video files. BMediaFile has support for seeking to the closest keyframe (ahead of or behind) to the frame you wanted. This feature did not work very well in Release 4.5 but will work much better in Release 5. If you require seeking to the exact frame you requested, you need to iterate calling ReadFrames() until you get to the frame you want.

Video codecs each have different properties that make them more or less suited to different tasks. Cinepak, for example, is definitely not a real-time capture type of encoder. It can take several seconds to encode a frame. A stream of Cinepak data always starts with a keyframe and is followed by delta frames and then usually another keyframe (then the cycle repeats). Some Cinepak-encoded videos only have a single keyframe, as the first frame and everything else is just deltas. Decoding Cinepak data, though, takes very little time, making it a good playback format (for lower-end machines, etc.).

PJPEG (aka Photo-JPEG) encodes each frame as a JPEG image. This works reasonably well but does not take advantage of the temporal coherence in video. You can find fast (real time, even) PJPEG encoders (such as the one that ships with personalStudio). MJPEG is JPEG encoding of fields of video (remember though—frames are not the same as fields with video!). PJPEG is a good format for editing, because each frame is independent (i.e., a keyframe) so perfect seeking is possible.

One (simplistic) way to describe MPEG-1 is to think of it as PJPEG frames interspersed with deltas between the PJPEG frames. MPEG-1 encodes frames into one of three types: I-frames, P-frames, or B-frames (there are also D frames but these are extremely rare). I-frames are keyframes, P frames are "predicted" frames, and B frames are bi-directional frames. You can only seek to I-frames (and if an MPEG-1 file had only I-frames it would be roughly equivalent to a PJPEG file).

Indeo-5 encoding is similar MPEG in that it uses temporal cohesion to improve the compression rate, but there isn't a lot of information about the specifics of Indeo-5 encoding. Indeo-5 encoding can be done in real time if you have the right encoder (not currently on BeOS). The quality of Indeo-5 is decent, although not as good as a good MPEG-1 encoder. Indeo-5 is a good distribution format but not the best as an editing format.

DV (Digital Video) encoding is the encoding used by DV cameras (duh!) that communicate with a computer over 1394 (aka Sony i.Link, aka Firewire). DV is a good encoding: it's high quality; has constant bit-rate (about 4 megs/sec); and it can be decoded in real time in software. DV is also a good format for editing because every frame is independent.

The Apple Video, MS Video, and MS RLE formats are all old video encodings that don't have good quality compared to the newer codecs. Essentially they're all variants of run-length encoding.

Raw video isn't really an "encoding" but it is an option for storing video if you have the disk bandwidth. Typically, you can capture raw 320x240 size video to a standard IDE hard disk. Capturing 640x480 video at 16 bits per pixel requires about 17.57 megabytes/second bandwidth. That means that you need either a super-fast single drive or a striped disk setup. You can store raw video in a variety of pixel formats (RGB-32, RGB-16, YUV-422, etc.). Both AVI and QuickTime support raw video; MPEG does not. Raw video is perfectly seekable and is the best format for editing if you can handle it.

The MS-ADPCM, CCITT-ADPCM and ima4 encoders are all variants of Adaptive Pulse Code Modulation encoding. Essentially, they encode 16-bit audio into 4-bit chunks in a mostly lossless manner. The encoding process isn't terribly CPU intensive and decoding is quite cheap. You probably wouldn't want to use these formats for editing but for final distribution they work well.

uLaw encoding compresses 12-bit audio into an 8-bit quantity. The quality is usually poor (phone-line quality) but the encode and decode are extremely cheap. Old timers may remember the original SparcStation-1 had a /dev/audio that spat out (and accepted) uLaw-encoded data. And before I get flamed, yes, the NeXT Cube did it too. BeOS only supports decoding this format.

MPEG-1 audio is a sophisticated encoding scheme that uses psycho-acoustic models, DCT transforms, and Huffman encoding to compress audio, typically around 10 to 1. MPEG-1 audio has three layers, called, appropriately enough, layers 1, 2, and 3. MPEG-1 layer 3 audio is commonly known as MP3. There is also a layer 2.5 for low bit rates but it is not terribly common. MPEG-1 audio can be encoded with a wide variety of options (different data rates, stereo/mono, different sample sizes and sample rates). By far the most common form of MPEG-1 audio data is 128kbs data rate 44.1 khz stereo data (it's the format that 90% of your pirated mp3's are in). MPEG-1 audio is typically quite expensive to encode. Decoding doesn't take a lot of CPU time relative to video but it's much more than any of the other audio encodings. MPEG-1 audio is a great format to distribute audio in because of its wide acceptance and excellent compression ratio.

As with video, raw audio provides perfect quality. Raw audio can be stored in a variety of sample rates (11,000 hz, 22050 hz, 32000 hz, 44100 hz, 48000 hz) and in a variety of sample sizes (8-bit, 16-bit, or even 32-bit). Fortunately, raw audio doesn't have the bandwidth requirements of video, so storing it on a standard hard disk is easy. Editing audio is always best when done with raw audio.

That about covers our tour of media file formats and codecs. I know that this is a bit short but the subject really is vast and we don't have the space to write all that should be written about it.

The two most important points that I hope people take away from this article are these:

• MPEG makes life difficult because it's difficult to know exactly how many frames are in a file until you've read them all. Therefore, you must code processing loops to end when they receive the error B_LAST_BUFFER_ERROR (even if you're not planning to deal with MPEG files).

• Most video formats (and some audio formats) use keyframes, which makes seeking imprecise. When seeking is imprecise you may ask to seek to frame 35 but only get frame 30.