My first Month of Contract Work

As some of you know, I’ve started my contract work on Haiku pretty exactly one month ago. During that time I’ve been working on various things that I’d like to summarize in this post. In the future I plan on posting more but shorter entries, but since much has happened in this month this one is going to be a bit more elaborate.

The wpa_supplicant

The more or less obvious first task was to finally finish the port of the wpa_supplicant, a software component that is responsible for handling the authentication and key management in wireless networks. It was the missing component keeping Haiku from accessing WPA/WPA2 encrypted networks.

The wpa_supplicant has gone through many hands already. I am not sure if it was even ported multiple times. Axel Dörfler did the original version of what I received in the end as part of his contract work over a year ago. He did implement most of the needed parts and had it pretty close to working already. Due to lack of time it was sitting in that state for some months until it was eventually handed over to Fredrik Holmqvist who continued work on it. Again due to lack of time it didn’t get much further. I’ve got the code in early July and managed to track down an issue in Haiku that prevented it from working as intended. Since it was basically usable at that point already I went ahead and made a few packages available that could be installed manually. However these versions were not integrated at all and due to that weren’t really stable either. History repeated itself and due to lack of time on my part it sat there unmodified for a couple of months…

Now that the generous donations of our dedicated fans has enabled Haiku Inc. to sponsor my contract work, the lack of time issue is finally out of the question. I’d like to take this opportunity to thank all of you who have made this possible!

So it was more or less obvious to continue work on wpa_supplicant as a first priority. As described above, it was basically working at that point already, however it was in now way integrated. To understand why this is so important and why it couldn’t be a stable component of Haiku without integrating it, one needs to know that wpa_supplicant is a rather traditional UNIX application. In this context that means that the whole program is written in a way that it relies on one single process running at any given time, running a classic event loop that does work as soon as work is indicated either by signals or incoming queued data. Under this assumption it is safe to just modify pretty much all runtime data directly, because you don’t have to worry about anyone else messing with it at the same time. For example you can change the list of configured networks or the state of the currently active network directly, without having to synchronize those modifications with anyone else. Now in case of Haiku the basic concept is quite different. Most things are multi-threaded and/or event based. Pretty much everything is therefore aware of multiple threads, shared data and how to properly handle modification to such data.

You can already guess what’d happen to the wpa_supplicant when we just started firing network notifications and association requests from various threads at code that really wasn’t built with these concepts in mind. Since the wpa_supplicant is waiting for events most of the time, it would actually work most of the time, as no critical data is modified most of the time. And that is the state the code basically was in when I created the standalone packages in July. Well, most of the time it’d work, but if things were timed right (and it wasn’t so hard to trigger) the wpa_supplicant would either outright crash or stop working. To make it stable it was therefore necessary to integrate it in some way.

There essentially were two options to go from here. Either “assimilate” the wpa_supplicant code base and restructure it so that it can work in a multi-threaded environment, or make the Haiku specific part wrap and serialize everything so that the wpa_supplicant code can function in the way it does on other platforms. While the first option would be nice to have it would be a huge amount of work (the wpa_supplicant code base is quite extensive). It’d also mean that we’d basically loose the ability to use the original code and just add Haiku specifics. We’d essentially have to maintain a fork of the code and backport all the changes made to the original wpa_supplicant manually. In the long run this is quite time consuming and error prone.

So instead I went with the second option. Basically we now feed everything that comes in, network notifications, join/leave requests and other events, into a single queue of “work to be done”, therefore serializing it. The wpa_supplicant event loop is then notified of the presence of requests so that it moves on and processes all of them in the single-threaded manner it was written for. While this sounds pretty simple now it has its own set of problems that needed quite a bit of work. For one, we can’t make any synchronous requests to the wpa_supplicant, i.e. doing a request and waiting for a reply, from network kit classes or the net_server. This is because the wpa_supplicant may need to access other functions of the network kit (and indirectly the net_server) that themselves do synchronous requests, locking the whole thing up in a deadlock… Eventually most of that was solved and by now we have a pretty stable component to work with (at least it never crashed on me since then).

Another part that the wpa_supplicant now provides is the GUI for entering network details such as network name, authentication method used and possibly the password/key. This also needed to be implemented to get away from the configuration file based model. The idea is that the wpa_supplicant is used to gather that information and that it uses native network kit classes to store those details for later. On the next connection the net_server will then be able to provide those details to the wpa_supplicant already so that it can connect without having to ask for any more details. This concept basically allows the network configuration to be stored in a Haiku native way, abstracted from the supplicant in use. Therefore it’d be feasible to replace the use of the wpa_supplicant with a different supplicant if there was a need to do so. It also allows native tools to create/modify/use such configurations (a wlan scanner that allows to auto-configure a network for example).

One of the central points of storing such network configurations is obviously storing the network credentials (WEP key, pre-shared key, certificate, username/password combo, …). Of course these could simply be written out into the (plain text) configuration file, but doing so has obvious security concerns. Ideally these credentials are stored in a central, secure facility that is at the same time generic enough to do credentials management for other applications as well. Haiku currently still lacks such functionality, known on other platforms as Keychains, Wallets, credential managers, etc.. Implementing this in a useful and extendable way is one of the tasks that is yet to be done and stands pretty much at the top of my regular TODO list now. I don’t have to start from scratch here, as Axel Dörfler who faced the same situation, already wrote down an API outline that combines many of the concepts of the aforementioned implementations on other platforms.

So all in all, the wpa_supplicant is pretty stable by now and should work on most of the wireless network cards where we have a FreeBSD based driver for (with a few exceptions where the hardware needs to be forced off due to bugs). The code modifications for the Haiku version of the wpa_supplicant are now available on Haiku-Ports and an optional package ready for use is available now as well. It is a bit rough around the usability edges (escape to cancel, default input field, default button, etc.) but that’s to come separately once the UI is actually finished and includes all further needed options. Since the eventual goal is to not require any specific use instructions (and since they would change with each further step in fully integrating it), I won’t put anything on that matter in this post. There are resources on the web that document the current workings. I have a relatively simple pre-shared key over here, so I simply manually connect and enter it each time for now.

The Thinkpad Distraction

Looking at my progress, the last wpa_supplicant related changes are back quite a bit already. What happened there, you might ask. And the answer would be that I’ve bought a new laptop, a Thinkpad X1, for my Haiku contract work. I used to use a massively underpowered HP laptop for most of my Haiku work. It has an ultra low voltage CPU which is quite enough for normal tasks, but for working on an OS, the turnaround times weren’t really usable. So it was pretty clear to me that once I was on contract I needed a new laptop to work efficiently. I therefore got myself a Thinkpad X1.

This (I’m writing this on that laptop) is a pretty current machine. It has a Core i5 at 2.5GHz, integrated Intel graphics and is SandyBridge based. There are also a few other nice gimmicks, but I didn’t really care all that much about them. The idea was that I wanted hardware that is supportable with an open source operating system while being generic enough so that any driver development would benefit many other systems as well. Going full Intel made sense in that regard as HDA, integrated graphics and Intel wireless chipsets are all pretty common and Intel supports open source drivers and releases the necessary specs.

So I was prepared to do quite a bit of driver work once I got the machine. I figured that it’d boot pretty much for sure, as a blocking Haiku issue on SandyBridge was recently fixed, meaning that the platform was at least partially tested and working. I also expected the wired networking to work out of the box or at least take very little effort to support. The audio should’ve worked as well. What would probably not work, or so I figured, would be the wireless network interface as well as the integrated graphics.

When the machine then finally arrived and after shrinking the Windows 7 partition down to a reasonable size, installing Haiku and the Haiku Bootmanager and verifying that everything worked in that combination I was all set. Haiku obviously booted otherwise I wouldn’t have been able to install it. However it only booted from the USB 2.0 port on the side, not from the USB 3.0 port on the back. While we do not support xHCI yet, I would’ve expected legacy support to kick in and make it behave USB 2.0, but oh well (TODO item added). Upon having Haiku booted I immediately noticed that the touchpad and trackpoint worked, but the “buttons” on the touchpad didn’t (TODO item added). Those aren’t separate buttons, but are implemented by pressing down the whole touchpad on either side for right and left click. Since I’m used to use the trackpoint anyway it didn’t bother me all that much, so I didn’t give it much priority. I basically only use the touchpad for scrolling and was happy to see that it worked out of the box (two finger scrolling as well as the scroll area on the right side).

More obvious, and expected, was the use of VESA for graphics. The native resolution (1366x768) unsurprisingly wasn’t present in the VESA mode list, so the image I got was a blurry, stretched out 1024x768. This obviously was a no-go for full time working on this laptop, so that was my highest priority TODO item to add. I then looked at the network status and, much to my surprise, found both the wired and the wireless adapter showing up right there, including a list of found wireless networks. It turns out that the iprowifi4965 driver actually supports a much wider range, up to and including the 6000 series of which this laptop sports one. I tried connecting and the wpa_supplicant showed up promptly asking me for credentials and eventually connected without any problem. I honestly have no idea if the wired network actually works, I haven’t had the need for a cable so far…

The integrated HD audio is currently only detected, but produces no sound. This is the same for many other HDA based systems and our bug tracker has a lot of tickets concerning that. Once I’m done with the graphics driver I’ll probably come around and look into that (TODO item added).

So basically I was stuck with a non-native resolution and had to go and implement support for SandyBridge based integrated graphics into intel_extreme. Researching other drivers, specifically the changesets needed to get this model supported, and using the open specs I figured out that the SandyBridge architecture is pretty much the IronLake architecture used in previous generations of CPU integrated graphics. Both of which we didn’t support. Indeed the move of the GPU into the CPU package and the move from a two-chip chipset into a one-chip “chipset” (the Platform Control Hub, PCH) meant that quite a few things were shuffled around. It was nowhere near as bad as I initially feared though. The basic structure is thankfully still pretty much the same as with more traditional chipset based integrated graphics. That means a lot less work, as the new architecture can be integrated into the existing intel_extreme driver instead of having to write a whole new one from scratch.

And that’s pretty much what I did then. Extracting some parts, using different register locations and adding SandyBridge specifics here and there. I got to the native resolution pretty quickly actually and was all happy. But the situation is pretty much the same as with the standalone version of the wpa_supplicant I outlined above. Basically it worked, but it wasn’t really the real deal. It worked mostly because the BIOS already programmed the graphics hardware and the panel to the right values. So what I did was mostly “just” setting the mode (it’s obviously a little more complex, but you get the idea). It also only happened to work with the panel because the driver mistook there to be an analog output so it programmed the “right” pipe.

What that means is that while I now have the native resolution working, it is really all dependent on the right setup happening before Haiku starts. The HDMI port isn’t programmed at all for example, so it’s not usable from Haiku, as are most DVI ports on other hardware using intel_extreme currently. So what I am currently working on is re-structuring the intel_extreme driver so that it can more properly support different ports and do more initial setup for them to become usable under Haiku. Since this puts the driver into a pretty “messy” state while working on, it is very difficult to commit things in-between. This is one of the reasons why you haven’t seen many more commits in that regard.

The Sickness Distraction

An unfortunate reason why I’ve been less productive during the last two weeks was/is a bacterial infection and the accompanying medication. It basically meant that I had to work in shorter bursts than I would have liked. How does this affect the contract? Well, I’m keeping pretty precise track of my working hours and in the end I am only going to bill for actual hours spent. So rest assured that everything is going to be fair. I’d really be uncomfortable wasting any of that precious donation money (back in the day I was a donor myself), so I’m trying really hard to ensure that the investment in this contract pays off.

The (presumed) Low Hanging Fruits

When working on more complex tasks that tend to take a long time to get to any rewarding milestone (as is the case with the intel_extreme work), I sometimes need to take a break so that I don’t get frustrated. In such cases I start looking into issues that have annoyed me during my personal Haiku use or I look at the bug tracker to see if I can take on a lighter task for a while.

In this case I’ve stumbled upon an interesting looking bug that causes content of some windows to be drawn on the wrong workspace when moving windows across workspaces. I figured it can’t be all that hard to track that down, so I went ahead and took a look, setting up a app_server test environment and reproducing the issue. While looking over that code, nothing obvious came up, so I read a little further. One thing lead to another and I noticed how the workspace size seemed to “jump” in the Workspaces applet depending on what workspace one is on. It turns out that various screen configurations are (rightfully) stored per workspace, so that a workspace keeps the correct configuration even when attaching different monitors. Apparently though, this mechanism isn’t yet fully taken into account and the workspace config you see for the current workspace is the only one being correct, while the others always use the first available configuration…

That’s how far I’ve come on that front. I know that I’ll have to look into that sooner than later anyway, because this quite probably needs adjustments as soon as we implement native multi-monitor support. The intel_extreme work is basically ground work to get that supported in that driver.

Various Other Stuff

Like most other devs, I do get all the Haiku commits via the commit mailing list, all the ticket updates via the bugs mailing list and the discussions on various other mailing lists. This takes up quite a bit of time to process (and most often I actually do that while not working on contract, i.e. during my normal private email processing). Sometimes these inputs spark an interest and lead to a quick fix for an issue here and there completely unrelated to what I am actually working on. So if you see out of context commits from time to time that’s usually where they are coming from.

During my contract work I’m also pretty much constantly connected to the Haiku IRC channels. On the one hand this is one of the most direct and easy ways to get in contact with me and on the other it is a good way to get quick feedback. When I’m knowledgeable in some area I usually try to help out when asked and the same is true for many others in these channels, so this generally works out both ways.

Conclusion

I’d like to conclude this month of contract work by again saying thank you to all the donors and also to the Haiku Inc. members who made this all possible. Not only for the money spent, but also for the trust put in my work. Only a few months back I wouldn’t have guessed that I could actually do Haiku work as my main job and it feels really rewarding to be in this situation.

Now I’m first and foremost looking forward to the upcoming BeGeistert, where I will be present during the main event as well as the coding sprint following it. My work schedule during this time will probably consist mostly of many discussions and various tasks that can better be tackled together, so my normal TODO list will likely take a back seat during this time. After or during BeGeistert I’ll continue working on the things outlined in here.