'Packaging Infrastructure' Contract Weekly Report #1

By the time I got home on Monday and finished up some other things I had to do, it was the evening, so rather than start my contract then I spent the remaining hours of the day getting caught-up on the backlog of patches and small bugs that accumulated since I last had time to spend on Haiku a month or so ago. I committed a number of patches that had been waiting in Trac, such as a fix to an off-by-one-error in BMessage, a usability fix in Expander, and some miscellaneous fixes to GCC warnings.

I started the actual work on Tuesday morning. I'd already been discussing with Axel and Oliver what the buildserver needed to do exactly, and the best ways to implement it. As previously mentioned, Python had already been selected as the language the client should be written in for maximum portability across architectures (and stability, as Ruby crashed all the time at that point -- but Hamish Morrison has since fixed the problem in our Ruby port).

What the server should be written in was still up in the air, though. Alexander von Gluck already had a prototype one written in Ruby, called HaiKeuken and a client to go with it, but I dislike Ruby and thought HaiKeuken was far too bulky and complex for the task it had to do (~7500 lines of code for the server & client combined, not counting CSS -- and all it does is display the contents of the HaikuPorts repository with lint status, essentially). I initially toyed with writing the server in Python, so that the client and the server would be written in the same language, but after some experiments with CherryPy and the Python Packaging Infrastructure (which I'd never used before), I quickly gave up and went back to my original idea of writing the server in node.js.

For those unfamiliar with Node, it's essentially "JavaScript for the server" -- it's designed for building event-driven web applications. Its package host, npm, is decidedly more sane than Pip+VirtualEnv, and despite only being a few years old (in contrast to Python, which is nearly 25), it's got a rich featureset and a large number of packages available.

Once I'd picked what I was working with, writing the first parts of the system was a breeze. I wrote a thin Python client that connects using a TLS socket to the buildhost, and waits for JSON messages instructing it to return information or run shell commands. The client is thin on purpose; it is essentially blind to what it's doing, so that the server can be updated to run more tasks while the client doesn't require updates (or if it does, it's technically possible for the server to use the client to update it).

The server is also simple and modular. I reused the web interface's layout and styling that HaiKeuken had (which saved a lot of time), and recreated pretty much all of the functionality of HaiKeuken between Tuesday and Wednesday, as well as enhancing the web interface to also be "thin" (the server just feeds it JSON, and all the HTML page generation happens in your web browser -- the HaikuDepot Web Interface works in much the same way). As of right now, the server is pretty basic -- it only clones and indexes the HaikuPorts tree, and keeps track of what recipes change with each call to git pull, as well as keeping track of builders, but it already does more than HaiKeuken ever did -- in just over 500 lines of code, or less than 1/10 of HaiKeuken's 7500 lines of code.

All of my code is now online on my GitHub account, where it'll live for at least the next few weeks -- it's unclear where its final home is (on git.haiku-os.org, or at HaikuPorts on BitBucket, or on Haiku's GitHub organization, etc.), but that's a minor detail that'll be straightened out eventually. Over the next week, I plan to start implementing the bulk of the "package rebuild" logic, which includes calculating the least-expensive route to rebuild a specific set of packages, and then working on the logic that actually builds and uploads packages.

See you all next week!