As a long-time Mutt user I always looked with envy at you Thunderbird and Kmail and what-not fans, as you could spawn new windows for reading and writing e-mails with a mere click (or sometimes a double-click).

It was just too bothersome to have $EDITOR block my inbox until I finish writing or give up and postpone the mail, losing track. As I am using Screen to run Mutt anyway, it seemed like a logical step to make it spawn new screen windows for writing mails and opening other mailboxes. The problem of multiplexing Mutt seems to have appeared to other people before as well, however the solution is not quite easy to spot.

The Idea Behind Mutt Multiplexing

The suggestion for people like me is: replace editor with a command line that spawns a new terminal and opens up mutt -H on a copy of the temp file. The original mutt will then see that the message was not changed and drop it silently, while the new instance allows editing. Fortunately, you can call screen from inside screen, thus creating a new window, so project mutt-screen was born! Unfortunately, not everything was as easy as expected.

The Gory Details, version 1

So, to make the whole thing better manageable, lets split that editor setting line into two parts: a script that is called in the new screen window and what needs to be done in the original mutt instance.

Lets call the script ~/bin/mutt-compose.sh:

#!/bin/sh
mutt -H "$1"
rm "$1"

Now, lets make a copy of the mail draft (%s), and run the composer in a new screen window:

# edit messages in a new screen window
set editor="cp %s %s.2; screen -t composer ~/bin/mutt-compose.sh %s.2"
unset wait_key

Great! We did it! Oh wait, no! It's a fork bomb! The new composer of course evaluates editor and launches... another copy of itself.

Edit the editor, version 2

Consequently, we need to override editor for the editing instance. Because we need to prevent the new Mutt from inserting a second .sig and more headers, lets create a second config file. Then we can call mutt -P ~/.mutt_compose -H "$1" from the script, with ~/.mutt_compose containing:

# read main config
source ~/.muttrc

# remove hooks, headers and sig, they are already in the draft
unhook send-hook
unset signature
unmy_hdr *

# call the right editor immediately
set autoedit=yes
set editor="vim +'set tw=72' +/^$/+1"

But now its going to work, isn't it? Not quite - something happened to postponed messages! Why are there now two edit windows? It seems, Mutt has a different code-path for messages which have been edited already. Here, the first instance ignores that the message was not edited and falls through to the compose window, while the second instance runs the editor (and permanently deletes the message if you do not explicitly save it, oops!).

Forking the editor - or not? Version 3

Our forking-out of the editor causes problems in three cases, as it seems: <recall-message>, <edit> and <resend-message>/<forward-message>. The former can be worked around by using a custom macro:

# override the <recall-message> hotkey
macro index,pager R "<shell-escape>screen -t postponed mutt -F ~/.mutt/compose -p<enter>"
# prevent recall on <mail>
set recall=no

This comes at the additional discomfort of not being asked if you want to resume writing your last mail when pressing m. That might be possible to achieve with some workaround - feel free to leave a comment if you find one.

For <edit>, the only viable workaround seems to be resetting editor to its former state, editing the message in-place and setting editor back to the screen call. Inconvenient, blocking your main Mutt, but possible.

# bonus points for outsourcing the two different editor settings into their own
# source'able one-liners...
macro index,pager e '<enter-command>set editor="vim +set\ tw=72 +/^$/+1"<enter><edit><enter-command>set editor="cp %s %s.2; screen -t composer ~/bin/mutt-compose.sh %s.2"<enter>'

With <resend-message>/<forward-message>, there is no workaround. I tried pushing keys into the buffer to quit the compose window, changing relevant settings, making blood sacrifices to various gods, all without success. So far I have to live with manually quitting the first composer and editing the message in the second one. Yikes.

The resulting config, version 4 (final)

When we put everything together, the following settings files should be in place.

~/.muttrc

# do not forget your own settings...

# edit messages in a new screen window
set editor="cp %s %s.2; screen -t composer ~/bin/mutt-compose.sh %s.2"
unset wait_key

# override the <recall-message> hotkey
macro index,pager R "<shell-escape>screen -t postponed mutt -F ~/.mutt/compose -p<enter>"
# prevent recall on <mail>
set recall=no

# set the editor for for editing messages in-place
macro index,pager e '<enter-command>set editor="vim +set\ tw=72 +/^$/+1"<enter><edit><enter-command>set editor="cp %s %s.2; screen -t composer ~/bin/mutt-compose.sh %s.2"<enter>'

# open mailbox listing in a new window
macro index,pager y "<shell-escape>screen -t mboxes mutt -y<enter>"

~/bin/mutt-compose.sh

#!/bin/sh
# set the screen window title to the message receiver
awk -F 'To: ' '/^To:/ { print "\033k" $2 "\033\\" }' "$1"
mutt -F ~/.mutt_compose -H "$1"
rm "$1"

~/.mutt_compose

# read main config
source ~/.muttrc

# remove hooks, headers and sig, they are already in the draft
unhook send-hook
unset signature
unmy_hdr *

# call the right editor immediately
set autoedit=yes
set editor="vim +'set tw=72' +/^$/+1"

The following .screenrc.mutt file can be used to spawn a screen with your inbox in it and a status bar showing the outsourced instances:

~/.screenrc.mutt

hardstatus alwayslastline "%-Lw%{=b BW} %50>%n%f* %t %{-}%+Lw%< "
sorendition = bG
vbell off

# start mutt directly in a window
screen -t INBOX mutt

Conclusion

This solution is not perfect. In certain corner cases (<resend-message>) it just does not behave. In other situations (caching the PGP passphrase does not work) it has less comfort. Nevertheless, it has increased my Mutt productivity and decreased my frustration a lot. Now all is missing is clean support for multiple profiles (read: without having many scripts bound to clumsy keyboard shortcuts) and proper in-mutt mechanism for message tags.

Sometimes it is plain frustrating to see how people try to be smarter than you and hard-code functionality which is (almost) impossible to override.

The rant (skip this section or don't complain!)

The one single most-frustrating feature of my HTC Dream/G1 phone looks like an attempt to be smart and save batteries: Whenever the phone connects to a WLAN, the 3G/mobile data connection is terminated.

"What's the problem?" you might retort. Now, most Internet applications are using TCP as their protocol of choice, and TCP maintains a connection bound to an IP address. Whenever you change your Internet access, you switch IP addresses and all existing TCP connections vanish. Your downloads are aborted, your SSH connections are closed, your IM session is terminated (or, even worse, it looks like online but is not).

The smart G* engineers of course have provided a way to detect the change of connectivity, using a NetworkConnectivityListener. They also probably implemented some really smart synchronization protocol into their binary-only applications, to improve the user experience.

However, they did not provide a way to prevent the deactivation of 3G data service. They added in some complicated code to keep a 3G data connection open to the MMS service, but the "normal" data session is just terminated whenever a WLAN is found. This would not be as bad as it sounds if such a state change would only happen twice a day (WLAN → 3G when you leave home; 3G → WLAN when you come back). However, WLAN is eating your batteries really really fast. Thus, the smart G* engineers made the phone automatically switch WLAN off one minute after the display backlight is disabled. That means: you look at the phone clock, it finds a WLAN, terminates all your connections, goes to sleep, turns off WLAN, terminates all your connections, ... GOTO 10

To add insult to injury, they added ConnectivityManager.requestRouteToHost(int networkType, int hostAddress), which looks like it would set up a route to your destination using the specified network interface. Ha-ha! Fail! That function only works if the requested interface is already up!

For application developers, this basically means that they have to catch the CONNECTIVITY_ACTION events, terminate the stale connection and open a new connection, synchronizing all of the state between the client and server. This of course implies that the application protocol must support re-synchronization. HTTP for example provides the Range: x- header to continue a partial download. For Jabber, there is XEP-0198 (which is still missing in most implementations). Other protocols, like SSH, are basically screwed.

For developers working at a mobile carrier, this is also bad news - there is no way to access the 3G data network when the user is surfing via WLAN.

Compared to this, the Symbian way of presenting the user a list of available networks when an application opens a socket is just a heavenly dream. Sorry, smart G* developers, you f'ed up this one.

The hack

After following the PdpConnectionConnectivityManagerConnectivityService twine of bloat, I saw some really fascinating code in ConnectivityService:

if (!mTestMode && deadnet != null) {
    if (DBG) Log.v(TAG, "Policy requires " +
      deadnet.getNetworkInfo().getTypeName() + " teardown");
    toredown = teardown(deadnet);
    if (DBG && !toredown) {
    Log.d(TAG, "Network declined teardown request");
    }
}

Now, what it means is basically that if mTestMode is enabled, the old connection is not terminated when a new one is established. mTestMode is set as:

mTestMode = SystemProperties.get("cm.test.mode").equals("true")
    && SystemProperties.get("ro.build.type").equals("eng");

On a rooted phone, all we need to get it is to change /system/build.props, reboot, and call requestRouteToHost().

Fortunately, the smart G* engineers fixed this evil exploit for the 2.0 release! GOTO 10 again!

XEP-0198: Stream Management is an XMPP extension adding stanza acknowledgements and stream resumption.

What does that mean in English? If you are using Jabber on your cell phone, you are not going to lose messages anymore whenever you get out of service coverage.

So, why do we need it?

Even though Jabber is using TCP, which is called a reliable protocol, messages can be lost when a user loses connectivity: the server still has an open TCP connection to your client, sends the message, and the message vanishes because you are gone. In the good case, the connection is immediately closed by your network provider, causing further messages to be stored on the server. In the bad case however, there is no reply (because your provider employs paranoid admins), and the server takes the usual TCP timeout (several minutes to hours) before taking you offline.

How can XEP-0198 save our lives?

The new proposal can not magically keep the connection while you are offline. However, it introduces two important elements to provide you with the messages you missed when you come back:

• stanza acknowledgement allows both the server and the client to request a reply (acknowledgement) on every sent message or a group of messages. Once this reply is received, the messages are considered as delivered.

• stream resumption is used when a connection is re-established. By including sequence numbers in the stanza acknowledgements, both client and server can tell each other which messages they received before the interruption, requesting to re-send later messages.

By combining these two enhancements, it is possible to provide a delivery guarantee for every message between you and your server. The same mechanism can also be used on S2S (server to server) links, however these are generally far more reliable.

XEP-0198 also allows to implement *throttling* of connections on busy servers, however this does not have such a big effect on the subjective reliability of Jabber.

Update, 2012-03-31: Protocol Versions

Over the years, the XEP-0198 specification developed, changing the XML stream format on the way. Fortunately, they introduced a protocol version to check for compatibility. Unfortunately, the authors who bothered enough to implement the proposal, only added one single version of the spec, providing no backward compatibility:

Version	Namespace	Annotations	Implementations
0.5 (2008-09-29)	urn:xmpp:sm:0	First version with an official namespace and protocol versioning.	ejabberd
0.6 (2009-03-19)	urn:xmpp:sm:1	Renamed attributes, added (later deprecated) stanzas attribute.
0.8 (2009-04-09)	urn:xmpp:sm:2	Simplified protocol. Added (later out-specced) throttling element <t/>.	Gajim, Prosody, Psi, Swift(en)/Stroke
1.2 (2011-03-02)	urn:xmpp:sm:3	Further simplified protocol. Removed throttling, per-number acks.

Where can we get it?

Unfortunately, after XEP-0198 has been proposed to Draft Standard on 2009-06-17, not much has happened. Update: The following applications already support the extension:

• Gajim (Gnome-oriented XMPP client): version 0.8
• jabberd2: pre-0.5?
• Prosody (a lightweight Jabber server written in Lua): version 0.8
• Psi (multi-platform XMPP client): forks of Psi and Iris: version 0.8
• Swift ("your friendly chat client", multi-platform), by means of Swiften, its embedded XMPP library: version 0.8
• Stroke (a Java port of Swiften): version 0.8

If your favorite server/client is not in that list, there is something you can do:

• provide patches for your favourite Jabber application.
• kindly request the authors to implement it.
• support existing requests.

Here you can find related requests for different Jabber software:

• SMACK (Updated May 2011, original Forum thread) (a Java client library usable in mobile phones)
• BEEM (Android XMPP client based on SMACK)
• ejabberd (Updated Jan 2012: fork with XEP-0198) (scalable Jabber server written in Erlang)
• Pidgin (multi-platform multi-protocol IM client)

Notes

There are several related extensions, like XEP-0184: Message Receipts or XEP-0199: XMPP Ping. However, these do not provide the reliability and efficiency of XEP-0198.

If you know of a client or server that supports XEP-0198, please feel free to leave a comment or drop an email to <georg@op-co.de>.