One fateful night several years ago, while staring bleary-eyed at my monitor and sipping on weak tea, this semi-epiphany came upon me. The pattern of abuse and enlightenment that accompanied my battles with Visual Basic suddenly seemed eerily familiar. Memories of my earlier struggles with C, C++, and a number of other minor language forays seeped in. Of course! These were not random events -- there's a signal in there. I wrote down the seven stages, forgot my name, then went to sleep.

The Seven Stages

1. Babe in the woods
2. Copycat
3. Daredevil
4. Apprentice
5. Patriot
6. Prisoner
7. Master

Ernie

Let me now introduce our hypothetical subject -- a young, fresh-faced college kid named Ernie J. Whipplemore. Ernie decides that learning C++ is simply one of life's most important tasks. He's read all about it and is enthralled by the power it will give him. He will master the languager or die. So what does Ernie have in store?

Babe in the woods

Dazed and confused. Ernie spends a lot of time reading C++ books and studying examples. It's not much fun to know so little, but learning is still a kick, and a challenge not be backed away from. Simply compiling a sample program is an accomplishment. But a sweet one at that... until an attempted minor change screws up everyting. Ernie, meet compiler diagnostics... compiler diagnostics, meet Ernie.

Ernie will likely send out a few desperate emails during this stage. Hopefully the gurus lurking on the developer mailing lists he has subscribed to have gentle souls, because some of his questions will evoke considerable humor and disbelief. Ernie will learn to delete those emails from his drive.

Copycat

Things are starting to make sense. Ernie finds that he can successfully create a range of programs that run and actually do something. His body of work, however, bears more than a striking resemblance to the work of others. Ernie has become a proficient mimic.

Imitation is the sincerest form of productivity, and Ernie's fingers become quite familiar with his editor's copy and paste keys. He doesn't always fully understand why his programs work, but that they do is a source of comfort. Still, it would be nice to veer off the beaten path once in awhile. Back to the books.

Daredevil

Wheeeeeeee! Programming is fun!

It's wonderful to be so clever. And how convenient that there are so many tricks and techniques for expressing one's cleverness! Operator overloading, virtual base classes, iostreams, polymorphism. Anyone for template instantiated recursion via multiple inheritence?

Ernie has fallen into the perpetual newbie programmer trap -- that of writing programs that are less about getting something done than of displaying the author's genius. Don't ask him if any of this stuff is really needed. Who friggin cares? This sh*t is simply too cool!

Apprentice

Well, the good times can't last forever. At some point, Ernie begins to take critical note of his work. Why does his associate Frank write code that has far fewer bugs, runs 3X quicker, and is written in much less time? For the first time, Ernie begins to ponder the difference between writing a program that works and writing a good program.

"So what does Frank have that I don't have?", ponders Ernie. So begins his quest to find the holy grail. Ernie adopts a mentor and begins to lap up the wisdom, as fast as he can. The mentor may be a workmate, an online acquaintance,... perhaps a hard copy of "Extreme Programming".

Silver bullets left and right... flying in so fast, it's impossible to escape a direct hit. Naturally, there is no end to the stream of (often conflicting) programming prescriptions being proffered. But which advice is right? Ernie will have to choose.

Patriot

At last, success! Having found a system that works for him, Ernie is ready to take on the world. He will begin each coding day by pledging allegiance to the flag of XYZ methodology.

This is perhaps the longest phase of them all. A certain mastery has been achieved and the game rules seem very clear. That the rules are essentially arbitrary is not a valuable concept at this stage.

Patriots are truly productive. Probably 90% or more of all the production code being run in the world is the work of patriots.They write the songs that make the whole world sing (to their tune, of course).

There is an undeniable stamp to patriot code: it's highly structured, complex, and very tightly (over?) engineered. Though the code works and works well, it has a certain rigidity. That rigidity prevents more flexible and elegant designs from being created. One structure requires another. Conversions are then required to translate between them. Functions pile up into libraries. Libraries pile up into frameworks. The hard disk drive and memory chip makers owe a debt of gratitude to the hard work of patriot programmers whose complex systems have no end to their resource requirements.

Prisoner

Paradise lost. As the years pass by, Ernie notices something unpleasant but unmistakable: no amount of care or adherence to the golden rules prevents all problems. Bugs still surface as reliably as mold on bread. Projects still fall behind schedule. Major sections of code that were supposed to be extensible for 1000 years continue to need total rewrites.

What a burden! Isn't there a better way to handle all this mess? Prisoners still get work done, but with little joy. The lure of other programming languages, tools, platforms, and methodologies is strong. The lure of sunny south-sea islands with miles of sandy beaches and not a computer within sight is even stronger.

Many will jump ship at this stage. Buf if Ernie hangs on, he may find a new way.

Master

Time to snatch the pebble from the hand. Ernie has some fresh thoughts: Could it be that good code is produced by talented, conscientious human beings and not by tools and methodologies? Could it be that even the best code is no match for the relentless march of technological fads and the changing whims and follies of users over time?

The best defense is nimbleness. Ernie masters the art of the light touch. A collection of scripts may work as well as a full-blown application. Programming tools can be mixed and matched as needed: a smidge of perl or python when adequate, or a complete custom C++ framework when called for. Whatever works best.

A master is not afraid to break the rules. Bad code is sometimes good code... but not necessarily. He understands the reasons behind the rules and thus knows when it makes sense to bypass them. Rules are less important than understanding the problem. Passing on that understanding is even more important.

Masters define the environment in which they work instead of the other way around. Don't like the language? Write your own. Don't like the OS? Write your own. But this may not be necessary or even desirable. Nothing is ultimately required or excluded.

Footnote:

So, did Ernie make it to this heightened awareness? You bet! Ernie is now a master and has achieved bliss. He earns a high salary, lives in a beautiful home, and is married to a wonderful woman. Unfortunately, his standard routine of sitting at a desk all day has led him to become overweight, carpal tunnel has invaded his hands, and his wife just left him for another man. But hey, don't feel too bad -- he's only a hypothetical guy anyway.

Requirements - Exported Functionality in BeOS

	status_t watch_node(const node_ref *node, 
		uint32 flags, 
		BMessenger target);

	status_t watch_node(const node_ref *node, 
		uint32 flags, 
		const BHandler *handler,
		const BLooper *looper = NULL);

	status_t stop_watching(BMessenger target);

	status_t stop_watching(const BHandler *handler, 
		const BLooper *looper = NULL);
	

	int notify_listener(int op, nspace_id nsid,
		vnode_id vnida,	vnode_id vnidb,
		vnode_id vnidc, const char *name);
	
	int send_notification(port_id port, long token,
		ulong what, long op, nspace_id nsida,
		nspace_id nsidb, vnode_id vnida,
		vnode_id vnidb, vnode_id vnidc,
		const char *name);
	

The latter is only used for live query updates, but is obviously called by the former. The port/token pair identify a unique BLooper/BHandler pair, and it used internally to address those high-level objects from the kernel.

When a file system calls the notify_listener() function, it will have a look if there are monitors for that node which meet the specified constraints - and it will call send_notification() for every single message to be send.

Each of the parameters vnida - vnidc has a dedicated meaning:

• vnida: the parent directory of the "main" node
• vnidb: the target parent directory for a move
• vnidc: the node that has triggered the notification to be send

The flags parameter in watch_node() understands the following constants:

• B_STOP_WATCHING
  watch_node() will stop to watch the specified node.
• B_WATCH_NAME
  name changes are notified through a B_ENTRY_MOVED opcode.
• B_WATCH_STAT
  changes to the node's stat structure are notified with a B_STAT_CHANGED code.
• B_WATCH_ATTR
  attribute changes will cause a B_ATTR_CHANGED to be send.
• B_WATCH_DIRECTORY
  notifies on changes made to the specified directory, i.e. B_ENTRY_REMOVED, B_ENTRY_CREATED
• B_WATCH_ALL
  is a short-hand for the flags above.
• B_WATCH_MOUNT
  causes B_DEVICE_MOUNTED and B_DEVICE_UNMOUNTED to be send.

Node monitors are maintained per team - every team can have up to 4096 monitors, although there exists a private kernel call to raise this limit (for example, Tracker is using it intensively).

The kernel is able to send the BMessages directly to the specified BLooper and BHandler; it achieves this using the application kit's token mechanism. The message is constructed manually in the kernel, it doesn't use any application kit services.

Meeting the Requirements in an Optimal Way - Implementation in OpenBeOS

If you assume that every file operation could trigger a notification message to be send, it's clear that the node monitoring system must be optimized for sending messages. For every call to notify_listener(), the kernel must check if there are any monitors for the node that was updated.

Those monitors are put into a hash table which has the device number and the vnode ID as keys. Each of the monitors maintains a list of listeners which specify which port/token pair should be notified for what change. Since the vnodes are created/deleted as needed from the kernel, the node monitor is maintained independently from them; a simple pointer from a vnode to its monitor is not possible.

The main structures that are involved in providing the node monitoring functionality look like this:

	struct monitor_listener {
		monitor_listener	*next;
		monitor_listener	*prev;
		list_link		monitor_link;
		port_id			port;
		int32			token;
		uint32			flags;
		node_monitor		*monitor;
	};

	struct node_monitor {
		node_monitor		*next;
		mount_id		device;
		vnode_id		node;
		struct list		listeners;
	};
	

The relevant part of the I/O context structure is this:

	struct io_context {
		...
		struct list		node_monitors;
		uint32			num_monitors;
		uint32			max_monitors;
	};
	

If you call watch_node() on a file with a flags parameter unequal to B_STOP_WATCHING, the following will happen in the node monitor:

1. The add_node_monitor() function does a hash lookup for the device/vnode pair. If there is no node_monitor yet for this pair, a new one will be created.
2. The list of listeners is scanned for the provided port/token pair (the BLooper/BHandler pointer will already be translated in user-space), and the new flag is or'd to the old field, or a new monitor_listener is created if necessary - in the latter case, the team's node monitor counter is incremented.

If it's called with B_STOP_WATCHING defined, the reverse operation take effect, and the monitor field is used to see if this monitor don't have any listeners anymore, in which case it will be removed.

Note the presence of the max_monitors - there is no hard limit the kernel exposes to userland applications; the listeners are maintained in a doubly-linked list.

If a team is shut down, all listeners from its I/O context will be removed - since every listener stores a pointer to its monitor, determining the monitors that can be removed because of this operation is very cheap.

The notify_listener() also only does a hash lookup for the device/node pair it got from the file system, and sends out as many notifications as specified by the listeners of the monitor that belong to that node.

If a node is deleted from the disk, the corresponding node_monitor and its listeners will be removed as well, to prevent watching a new file that accidently happen to have the same device/node pair (as is possible with BFS, for example).

Differences Between Both Implementations

Although the aim was to create a completely compatible monitoring implementation, there are some notable differences between the two.

BeOS reserves a certain number of slots for calls to watch_node() - each call to that function will use one slot, even if you call it twice for the same node. OpenBeOS, however, will always use one slot per node - you could call watch_node() several times, but you would waste only one slot.

While this is an implementational detail, it also causes a change in behaviour for applications; in BeOS, applications will get one message for every watch_node() call, in OpenBeOS, you'll get only one message per node. If an application relies on this strange behaviour of the BeOS kernel, it will no longer work correctly.

The other difference is that OpenBeOS exports its node monitoring functionality to kernel modules as well, and provides an extra plain C API for them to use.

And Beyond?

The current implementation directly iterates over all listeners and sends out notifications as required synchronously in the context of the thread that triggered the notification to be sent.

If a node monitor needs to send out several messages, this could theoretically greatly decrease file system performance. To optimize for this case, the required data of the notification could be put into a queue and be sent by a dedicated worker thread. Since this requires an additional copy operation and a reserved address space for this queue, this optimization could be more expensive than the current implementation, depending on the usage pattern of the node monitoring mechanism.

With BFS, it would be possible to introduce the possibility to automatically watch all files in a specified directory. While this would be very convenient at application level, it comes with several disadvantages:

1. This feature might not be easily accomplishable for many file systems; a file system must be able to retrieve a node by ID only - it might not be feasible to find out about the parent directory for many file systems.
2. Although it could potentially safe node monitors, it might cause the kernel to send out a lot more messages to the application than it needs. With the restriction the kernel imposes to the number of watched nodes for a team, the application's designer might try to be much stricter with the number of monitors his application will consume.

While 1) might be a real show stopper, 2) is almost invalidated because of Tracker's usage of node monitors; it consumes a monitor for every entry it displays, which might be several thousands. Implementing this feature would not only greatly speed up maintaining this massive need of monitors, and cut down memory usage, but also ease the implementation at application level.

Even 1) could be solved if the kernel could query a file system if it can support this particular feature; it could then automatically monitor all files in that directory without adding complexity to the application using this feature. Of course, the effort to provide this functionality is much larger then - but for applications like Tracker, the complexity would be removed from the application without extra cost.

However, none of the discussed feature extensions have been implemented for the currently developed version R1 of OpenBeOS.

I heard a friend of mine on the phone recently speaking with his wife, or perhaps one of his children. I don't know what the conversation was about, but he said "I'll take care of it". It struck me profoundly--that is exactly a father's job. Whether it's the water heater breaking down, or broken hearts, or even just a bad day of school, a father is called on to be the ultimate cure-all for whatever issues there are in family. That he, too, is human doesn't enter into the equation. It doesn't matter if he had a bad day at work, or if he is depressed, or even if he is just tired: It is still his job to take care of it. When everyone else needs him, he is there. He is the core, the nucleus, and the center of the family.

Leadership is much the same whether it is leadership of the family or leadership of an open source project. When there are hard decisions to be made, or tedious work to be done, the brunt of the burden often falls on the leader to ensure that it happens. Leaders accept that as part of their job. When bad decisions are made, responsibility should fall only on the shoulders of the leader. One example of this in our community was when Be announced the discontinuation of support for the BeBox (the first time). There was a loud cry from the community. JLG personally responded that a mistake had been made and that support would be continued for three years. This is a great example of how leadership should work--people make mistakes and sometimes make bad assumptions, but when they do, they own up to them and correct them personally.

Which brings me to ask: What is leadership?

Many people believe that leadership is the process of telling people what to do, clearing off your desk, putting your feet up, and allowing others to do the work. The same people, when faced with difficult situations, oftentimes wonder why their employees won't back them up. They hold only the whip over their employees. They believe that great vision statements, glorious five-year plans, and profound press releases will inspire their troops. Their employees don't see them as part of the team but oftentimes as part of the problem. One of the most useful things that I have learned in my career is that no one person in a business or in any organization is more important than any other. If you don't believe that, try firing your janitor. Try hiring a surly temp as your receptionist. Other people believe that people are replaceable, interchangeable like bricks. Oftentimes (in business especially) management fails to realize the value of longevity. The people who have been around your business the longest are the people who know the most about it and are the most valuable. One of the surest signs of bad leadership is an exodus of experienced employees. One of the surest signs of good leadership conversely is high employee retention.

My definition of leadership is "to influence, guide, protect, encourage, and assist through example, planning, clear communication, and kindness."

• influence -- instead of ordering people (which should be a last resort) to do what needs to be done, persuading them is a far better choice
• guide -- sharing your experience, perspective, knowledge, and viewpoint to help others grow
• protect -- whether this is standing up for your team members, or defending your family there is no more powerful way to show whose side you're on
• encourage -- everyone gets discouraged sometimes and needs someone who has been in their shoes to let them know that things will be better
• assist -- when the team member needs help, their leader is the first person they should be able to turn to for assistance
• example -- the primary way that a leader should lead is by example--being the personification of what they want their team members to be
• planning -- a team without a vision wanders aimlessly
• clear communication -- if the leader has a plan but the team doesn't know it they are no better off than if they had no plan at all
• kindness -- it may seem obvious but the golden rule still stands--do unto others as you would have them do unto you

I believe that there is no more difficult position than leading volunteers. Unlike a job or a family there is little formal structure holding a team member in place. There is a personal cost if you quit your job or if you leave your family. There is also no more rewarding position than leading volunteers for the same reason--if you keep the team together, it is a validation of your leadership.