Sometimes it is plain frustrating to see how people try to be smarter than you and hard-code functionality which is (almost) impossible to override.

The rant (skip this section or don't complain!)

The one single most-frustrating feature of my HTC Dream/G1 phone looks like an attempt to be smart and save batteries: Whenever the phone connects to a WLAN, the 3G/mobile data connection is terminated.

"What's the problem?" you might retort. Now, most Internet applications are using TCP as their protocol of choice, and TCP maintains a connection bound to an IP address. Whenever you change your Internet access, you switch IP addresses and all existing TCP connections vanish. Your downloads are aborted, your SSH connections are closed, your IM session is terminated (or, even worse, it looks like online but is not).

The smart G* engineers of course have provided a way to detect the change of connectivity, using a NetworkConnectivityListener. They also probably implemented some really smart synchronization protocol into their binary-only applications, to improve the user experience.

However, they did not provide a way to prevent the deactivation of 3G data service. They added in some complicated code to keep a 3G data connection open to the MMS service, but the "normal" data session is just terminated whenever a WLAN is found. This would not be as bad as it sounds if such a state change would only happen twice a day (WLAN → 3G when you leave home; 3G → WLAN when you come back). However, WLAN is eating your batteries really really fast. Thus, the smart G* engineers made the phone automatically switch WLAN off one minute after the display backlight is disabled. That means: you look at the phone clock, it finds a WLAN, terminates all your connections, goes to sleep, turns off WLAN, terminates all your connections, ... GOTO 10

To add insult to injury, they added ConnectivityManager.requestRouteToHost(int networkType, int hostAddress), which looks like it would set up a route to your destination using the specified network interface. Ha-ha! Fail! That function only works if the requested interface is already up!

For application developers, this basically means that they have to catch the CONNECTIVITY_ACTION events, terminate the stale connection and open a new connection, synchronizing all of the state between the client and server. This of course implies that the application protocol must support re-synchronization. HTTP for example provides the Range: x- header to continue a partial download. For Jabber, there is XEP-0198 (which is still missing in most implementations). Other protocols, like SSH, are basically screwed.

For developers working at a mobile carrier, this is also bad news - there is no way to access the 3G data network when the user is surfing via WLAN.

Compared to this, the Symbian way of presenting the user a list of available networks when an application opens a socket is just a heavenly dream. Sorry, smart G* developers, you f'ed up this one.

The hack

After following the PdpConnectionConnectivityManagerConnectivityService twine of bloat, I saw some really fascinating code in ConnectivityService:

if (!mTestMode && deadnet != null) {
    if (DBG) Log.v(TAG, "Policy requires " +
      deadnet.getNetworkInfo().getTypeName() + " teardown");
    toredown = teardown(deadnet);
    if (DBG && !toredown) {
    Log.d(TAG, "Network declined teardown request");
    }
}

Now, what it means is basically that if mTestMode is enabled, the old connection is not terminated when a new one is established. mTestMode is set as:

mTestMode = SystemProperties.get("cm.test.mode").equals("true")
    && SystemProperties.get("ro.build.type").equals("eng");

On a rooted phone, all we need to get it is to change /system/build.props, reboot, and call requestRouteToHost().

Fortunately, the smart G* engineers fixed this evil exploit for the 2.0 release! GOTO 10 again!

One-and-a-half Xbox360 hackers

Yesterday, I was sitting around and pondering whether to implement some kind of NAND filesystem support in XeLL. After all, there are people out there who would like to boot a Linux kernel without having to attach a USB storage device or insert a CD-R.

On the other hand, XeLL should actually be ported to libxenon, a hardware abstraction library for the Xbox360 hardware, which is desperately lacking developer attention as well. Now, I am only one person, but there are so many missing features. And by missing features I actually mean "things one could rip port from other OSS projects".

The idea of libhomebrew

And that is where the idea enlightened me. Every homebrew-on-$HARDWARE project so far was redoing the same things:

1. hack the hardware (optional)

2. write drivers for hardware components

3. create a library containing/wrapping all the drivers and a libc

4. port libraries with additional functionality (like MP3 playback and JPEG decoding)

5. port libSDL ;-)

6. write apps!

Now, steps 1. and 2. are of course specific to $HARDWARE. Step 3. is often based on how step 2. was performed, but it does not have to. The following steps however could be easily abstracted away from the actual hardware, even though currently, they are redone countless times.

But do we really have to redo them for every new platform?

One lib to rule them all

Instead, we could just create the one homebrew library, libhomebrew. It would contain a basic set of functionality, ports of commonly-used libraries and of course HAL backends for all supported platforms.

Everybody in the homebrew scene would profit from this:

• homebrew authors could do write-once deploy-everywhere development.

• platform hackers would profit too: instead of porting SDHC drivers or libmad to yet another console, they could just add a libhomebrew HAL backend for their hardware, automagically gaining all the libs (and many apps).

• library maintainers could integrate libhomebrew support into their libs, without the fear of creating a forest of hardware adaptations.

*yawn* This is all old news!

Of course, this is nothing I could file a patent for (I would not blog it if it was ;-)). The basic idea already exists for decades in many different implementations, however not in such a homebrew-centric way so far.

The three projects most similar to the presented idea are:

1. The Linux kernel is exactly that, plus a huge pile of bloat completely superfluous for console homebrew.

2. ScummVM is an adventure game "emulator" with a large set of platform backends.

3. devkitPro provides homebrew toolkits for several different platforms, but no common hardware abstraction as far as I could see.

Progress of libhomebrew

So far, all there is is a libhomebrew wiki page on the free60 wiki. After all, I am only one person with a full-time job not related to homebrew in any way.

However, I hope to find some interested developers who are tired of re-writing drivers and porting yet another lib to their favourite platform.

Contribute or at least spread the word!

cgit, a gitweb replacement?

Recently I was made aware of the existence of cgit, a git web frontend written in C. It looked promising, so I tried it out.

It seems to perform better than gitweb, which I kind of expected from the fact that it is a native binary, as opposed to a perl script forking git for the actual work. Also, there is a comparison performed by the author of cgit.

However, I was convinced by the clean, short, nice looking URLs provided by cgit, which allow to checkout files from your default HEAD without nasty hash tags:

/cgit/libmpq.git/tree/Makefile.am

versus the rather ugly gitweb URL:

/git?p=libmpq.git;a=blob;f=Makefile.am;
    h=1f70fdb06cc13ae0338c380139c352383aea7700;hb=HEAD

I think in the good old new days of Web 0.2, this is called RESTful behaviour, which also coincidentally contains the substring "STFU".

Syntax highlighting with cgit

Anyway, once I was at playing with cgit, I wanted to add some syntax highlighting, too. To enable it, cgit already comes with a filter script (you see the elegant URL syntax at work again?). This script uses the highlight command, but it does not support all the file formats of highlight, and might be missing some elegance.

I made an attempt at supporting all the files I have in the different git repositories, which you can find here:

#!/bin/sh
# store filename and extension in local vars
BASENAME="$1"
EXTENSION="${BASENAME##*.}"

# map Makefile and Makefile.* to .mk
[ "${BASENAME%%.*}" == "Makefile" ] && EXTENSION=mk

exec highlight --force -f -I -X -S "$EXTENSION" 2>/dev/null

This variant extracts the file extension of the file to be displayed, contains detection of Makefile.* files and spares one fork by running exec highlight. If the syntax is not known, --force makes the command just pass through the input.

Do not forget to add the CSS required by highlight, as documented in the original syntax-highlight.sh.

What is still missing?

Cgit comes with cache support to further reduce the CPU load. However, I yet have to figure out how to make the caching time long enough to make a difference on this low-traffic server without taking forever to update the cache content whenever I push changes.

A cosmetic issue remains with directories in the tree view. They should get a more decent mark-up for people to notice them. Update: this is easy with a small css tweak (example).

And, last but not least, I am missing a feature to limit the blob-size for generating HTML. This is especially important for binary objects which are served as a hex-dump, generating gazillions of bytes for something you are not going to read anyway. Update: If you want something done, send a patch upstream! (example) :-)

Update 3: All three patches have found their way into the cgit master repository, thanks very much, Lars!

XEP-0198: Stream Management is an XMPP extension adding stanza acknowledgements and stream resumption.

What does that mean in English? If you are using Jabber on your cell phone, you are not going to lose messages anymore whenever you get out of service coverage.

So, why do we need it?

Even though Jabber is using TCP, which is called a reliable protocol, messages can be lost when a user loses connectivity: the server still has an open TCP connection to your client, sends the message, and the message vanishes because you are gone. In the good case, the connection is immediately closed by your network provider, causing further messages to be stored on the server. In the bad case however, there is no reply (because your provider employs paranoid admins), and the server takes the usual TCP timeout (several minutes to hours) before taking you offline.

How can XEP-0198 save our lives?

The new proposal can not magically keep the connection while you are offline. However, it introduces two important elements to provide you with the messages you missed when you come back:

• stanza acknowledgement allows both the server and the client to request a reply (acknowledgement) on every sent message or a group of messages. Once this reply is received, the messages are considered as delivered.

• stream resumption is used when a connection is re-established. By including sequence numbers in the stanza acknowledgements, both client and server can tell each other which messages they received before the interruption, requesting to re-send later messages.

By combining these two enhancements, it is possible to provide a delivery guarantee for every message between you and your server. The same mechanism can also be used on S2S (server to server) links, however these are generally far more reliable.

XEP-0198 also allows to implement throttling of connections on busy servers, however this does not have such a big effect on the subjective reliability of Jabber.

Where can we get it?

Unfortunately, since XEP-0198 has been proposed to Draft Standard on 2009-06-17, not much has happened. So far, the only implementation the author could find is in jabberd2.

However, there is something you can do:

• provide patches for your favourite Jabber client.
• kindly request the authors to implement it.
• support existing requests.

Here you can find related requests for different Jabber software:

• SMACK (a Java client library usable in mobile phones)
• ejabberd (scalable Jabber server written in Erlang)
• Prosody (a lightweight Jabber server written in Lua)

Notes

There are several related extensions, like XEP-0184: Message Receipts or XEP-0199: XMPP Ping. However, these do not provide the reliability and efficiency of XEP-0198.

If you know of a client or server that supports XEP-0198, please feel free to leave a comment or drop an email to <georg@op-co.de>.

Am I actually making a blog? I don't even want to know!

Setting up the CSS for iki is not quite a nice & easy job, but at least it is not the usual Web 0.2 bloat.

Update: ikiwiki has the nasty misfeature of overwriting local.css whenever you regenerate the wiki without --refresh. Because I do not like it and I did not want to dig too deep, I added the following to my .htaccess:

Redirect /blog/local.css /ikiwiki.css

Up-Update: The obvious solution is to put local.css into the source directory instead of the destination directory. Thanks to Spida for this hint.