Services Kit features overview

Blog post by shisui on Thu, 2010-08-19 09:45
, , , , , ,

The Kit.

First of all, let's talk about the good news. Among others things, I introduced most of the classes required to make HTTP requests over the network, just like the cURL library allows you to do. The Services Kit should be viewed just like an onion : the core of the kit is mainly responsible of managing the various protocols (HTTP, HTTPS, FTP, LDAP, ..).

In its simplest shape, the kit takes an URL, decide what protocol he should instantiate, make the request to retrieve a resource, ensure that the request is well completed and return an object containing all the resulting data. Advanced uses of the kit includes many protocol-defined options that can be sent to the protocol class to refine some of its behavior. For example, one can tell the HTTP protocol to follow every redirection it will encounter (which is the classic behavior of a browser), specify to store cookies or not, add an form which will be sent among the request, ...

I wanted to be sure that at least the most important protocol was implemented, and here it is : the HTTP protocol is almost fully handled in Services Kit. In fact, the only missing thing is a proxy manager, which is not introduced yet in the kit. The actually implemented features in the HTTP protocol are :

  • HTTP/1.1 requests as well as old HTTP/1.0 if the user wants it.
  • GET/HEAD/POST/PUT requests
  • Raw POST data upload (for basic forms)
  • Multipart MIME POST data upload (for complex forms with file upload)
  • Chunked file uploads, allowing to upload a memory buffer just like it was a file and without knowing its size by advance.
  • HTTP authentication (basic and digest)
  • Optionally following redirection, with a max amount of redirections
  • Custom headers (in order to add and/or modify those who were generated by the kit)
  • Custom referer
  • Custom user-agent string
  • Full cookie management

Thanks to Stephan, my mentor, and his remarks, the cookie management classes has been refactored in order to have several performance improvements. They are now much more efficient (at least a 60% gain when working on big cookie database) than before when looking for the cookies of a specific domain thanks to a hash-map based storage.

Every protocol request lives in a separate thread from the main application, thus an app can do his own stuff until the request ends. The kit offers a few ways to send notifications to the application. The first one is to subclass a BUrlProtocolListener, register it as the protocol hook class, and it will receive various synchronous notifications coming from the protocol :

  • ConnectionOpened(), when the network socket is ready
  • HostnameResolved(), when the remote host name has been resolved
  • ResponseStarted(), when the remote host begins to respond to our request
  • HeadersReceived(), when metadata are received (like HTTP headers)
  • DataReceived(), when raw data is received
  • DownloadProgress(), periodically instructs the listener on the download state (number of bytes received and number of expected bytes)
  • UploadProgress(), periodically instructs the listener on the upload state
  • RequestCompleted(), when the request ends successfully or fails

      • An application can also be asynchronously notified of the protocol behavior directly by registering a BHandler which will received the same messages as above, but through the messaging system. Eventually, one can decided to make a synchronous request by using a dedicated wrapper including the url request, and the listener, which will wait until the request ends.

      Webkit and WebPositive

      An other aspect of the second half of the summer of code has been the integration of the Services Kit within the WebKit port of Haiku, and WebPositive. This proposal was issued by my mentor in order to see how the kit would fit in a real application. Crawling through the WebKit code base was quite painful since compiling a full debug version of the required webkit libraries takes about two hours and half on my development machine.

      Eventually, and after a lot of bug fixes :-), I have come to a well working WebPositive no longer lying on cURL libraries, and using only Services Kit! There is good news and bad news.

      Let's begin by the bad one : even if it works, it's still limited by Services Kit features, and WebPositive can only access pages over HTTP. That is to say no webmail at the moment, since almost all of them uses secure SSL connections with HTTPS.

      On the other hand, Services Kit seems faster than the cURL backend. A little test of a huge file download over my local network led me to this conclusion : Web+ with cURL was downloading at ~250kB/s whereas Web+ with Services Kit was around 1.5MB/s. The core of the Kit is not yet optimized with deferred loading and other features like that, but we could expect much more good things !

      Next steps

      I'm not letting Services Kit orphan at the end of GSoC and I will continue the development of the kit. Many things are yet to be done and I'm currently moving away the HTTP protocol from the kit in order to make an add-on for each protocol. Such a mechanism will allow much more protocols to be handled by Services Kit, but leds to many refactoring in the actual code in order to have a clean implementation of these add-ons, of optional protocol features, and so on.


shisui's blog