Things related to the Extensible Messaging and Presence Protocol (XMPP). Here be rants about the design (and lack of it) in anything related to XMPP and mobile communications.
So I was bored and I set up an XMPP server under 👆️.op-co.de, in addition to the one I already had under ツ.op-co.de. It worked with some clients and failed with others. But both ツ and 👆️ are valid Unicode 1.1 characters, so WTF? Buckle up (or better: put on your 🤿) for the 19-RFC deep dive...
...or skip right to the TL;DR.
Note: despite my generous use of Emojis throughout this post, none of it was generated by a text extruder machine. All words are the product of artisanal typing on my keyboard, and the Emojis were hand-selected from an Emoji-picker widget.
XMPP addresses
XMPP, the eXtensible Messaging and Presence Protocol, formerly known as Jabber®, defines its address format in RFC 7622.
An XMPP address (formerly known as Jabber® ID, or JID, as used in the RFC) has three parts:
jid = [ localpart "@" ] domainpart [ "/" resourcepart ]
The localpart is usually the username, but is not used when addressing a server.
The domainpart is the hostname or domain name, and can be a Unicode DNS
identifier, an IPv6 address in square brackets, or a legacy IP address. This
is the only mandatory part of a JID.
The resourcepart is the internal identifier of an individual client,
allowing a user to have multiple clients connected at the same time; it is
also used for the nickname in
XEP-0045: Multi-User Chat.
Each of these three parts must be valid UTF-8 and can be up to 1023 bytes (not characters!) in length.
Furthermore, there are restrictions for each part, for example:
localpart = 1*1023(userbyte)a "userbyte" is a byte used to represent a UTF-8 encoded Unicode code point that can be contained in a string that conforms to the
UsernameCaseMappedprofile of the PRECISIdentifierClassdefined in RFC 7613 [...]
Come again, please? Okay, let's take this apart, slowly.
a "userbyte" is a byte used to represent a UTF-8 encoded Unicode code point
This is a convoluted way to say that we accept up to 1023 bytes (not characters!) of valid UTF-8.
that can be contained in a string that conforms to the
UsernameCaseMappedprofile of the PRECISIdentifierClassdefined in RFC 7613
In addition to being valid UTF-8, it must also conform to the
IdentifierClass in PRECIS (RFC 7613).
PRECIS: Preparation, Enforcement, and Comparison of Internationalized Strings
PRECIS is the successor to Stringprep (RFC 3454, which we can ignore for now).
However, the PRECIS definition in RFC 7613 is obsoleted by RFC 8265, which we can't ignore and will have to take our character profiles and classes from.
So we need the IdentifierClass for the localpart, and furthermore
the FreeformClass for the resourcepart.
PRECIS classes, profiles and categories
The earlier Stringprep approach explicitly defined its classes as valid ranges of Unicode code points (characters). However, given that Unicode is a living (versioned) standard, new characters (and new Emojis! 💡) get added every year. This left Stringprep in an uncomfortable place, forever hard-coded to the long-superseded 2002 Unicode 3.2 standard.
To allow for future compatibility, PRECIS took a different path. It describes an algorithm that can be applied to an individual Unicode character in order to determine whether it belongs to a certain PRECIS class.
The classes (IdentifierClass and FreeformClass) are defined in
RFC 8264, and the profiles (UsernameCasePreserved,
UsernameCaseMapped, OpaqueString, Stringprep) are defined in
RFC 8265.
Furthermore, PRECIS attempts to retain backward compatibility with earlier standards like IDNA2008, as well as with itself. If a certain character is "valid" under an earlier version of Unicode, PRECIS tries to ensure that it stays "valid" under later versions. The only explicit exception from this is that code points that were "undefined" in earlier Unicode versions can later be assigned and move to "valid" or "disallowed".
Each of these rules is applied to individual characters, or to character categories, as defined in RFC 5892: IDNA code points.
Given this toolset, we can now get back to the individual XMPP address parts.
XMPP address elements
localpart - the user name
As stated in RFC 7622 above, the localpart must be...
a string that conforms to the UsernameCaseMapped profile of the PRECIS IdentifierClass
So we have the profile (UsernameCaseMapped) and the class
(IdentifierClass) to look up.
The UsernameCaseMapped transformation
The UsernameCaseMapped profile in RFC 8265 performs some
normalization steps: it requires
decomposition of certain East Asian
characters, lowercasing, Unicode Normalization Form C, and application of the
Bidi rule.
Later, RFC 8265 § 3.3.2 says:
Ensure that the string consists only of Unicode code points that are explicitly allowed by the PRECIS IdentifierClass defined in Section 4.2 of [RFC8264].
What's allowed by IdentifierClass?
RFC 8264 §4.2.1 defines the valid and disallowed character properties, as well as certain groups that require special treatment.
Valid identifiers contain "Code points traditionally used as letters and numbers in writing systems", the ASCII 7-bit characters U+0021 through U+007E, and a few characters that are only allowed in a certain context, like U+00B7 MIDDLE DOT which is only allowed inside the Catalan ela geminada "ŀl".
The ツ character (U+30C4 KATAKANA LETTER TU) belongs to the "Letter, other" (Lo)
category of Unicode)
and thus is a valid letter character allowed in IdentifierClass. The 👆️
emoji (U+261D WHITE UP POINTING INDEX) belongs to the "Symbol, other" (So)
category with all the other Emojis. The whole "Symbol" category is disallowed
inside of IdentifierClass. Bummer. Sad trombone! 🎶
On the other hand, the "Nonspacing Mark" (Mn) category is allowed, and so ḩ̸̡͇͉̬̓͝e̷͙̪̯̬̬͍͒̂̓̽̀̄ ̵̨̨̪̯̞̠͒̐͘͝c̷͍͆o̸̡̢̥͌̒̌̀͜͜m̶̬̙̙̓̌͘͠ë̷́̉́͜t̴̍̔͜͠h̷̖̭̫̥̖̥͐͂͊͒̓.
Putting localpart together
So essentially, PRECIS only allows the boring regular lowercase letters from any supported language, and none of the fun Emojis.
To add insult to injury, RFC 7622 §3.3.1 imposes further restrictions by disallowing some more fun characters:
" U+0022 (QUOTATION MARK)
& U+0026 (AMPERSAND)
' U+0027 (APOSTROPHE)
/ U+002F (SOLIDUS)
: U+003A (COLON)
< U+003C (LESS-THAN SIGN)
> U+003E (GREATER-THAN SIGN)
@ U+0040 (COMMERCIAL AT)
However, there are still a bunch of "funny" permitted characters left from the ASCII7 block:
!#$%()*+;=?[\]^`{|}
This leaves us with some valid old-school ASCII smiley user name options on the table:
;=)
B*}
And a bunch of Unicode letters that can be abused, with special thanks to Egyptian hieroglyphs:
| Symbol | Code Point | Name |
|---|---|---|
| ۃ | U+06C3 | ARABIC LETTER TEH MARBUTA GOAL |
| ツ | U+30C4 | KATAKANA LETTER TU |
| 𓀐 | U+13010 | EGYPTIAN HIEROGLYPH MAN WITH BLEEDING HEAD WOUND |
| 𓂸 | U+130B8 | EGYPTIAN HIEROGLYPH HUMAN PHALLUS |
| 𓂹 | U+130B9 | EGYPTIAN HIEROGLYPH ERECTILE DYSFUNCTION |
| 𓃂 | U+130C2 | EGYPTIAN HIEROGLYPH LEG SEVERED BY HAND GRENADE |
| 𓄀 | U+13100 | EGYPTIAN HIEROGLYPH ENRAGED YAXIM USER |
| 𓀬 | U+1302C | EGYPTIAN HIEROGLYPH CHUCK NORRIS RIDING ON TWO GIRAFFES |
The domain part
Back to RFC 7622 §3.1:
domainpart = IP-literal / IPv4address / ifqdnthe "IPv4address" and "IP-literal" rules are defined in RFCs 3986 and 6874, respectively, and the first-match-wins (a.k.a. "greedy") algorithm described in Appendix B of RFC 3986 applies to the matching process
We will leave IP literals out... for now. Just a note that the format for IPv6
literals need to be enclosed in brackets and may contain a %zone postfix.
ifqdn = 1*1023(domainbyte)a "domainbyte" is a byte used to represent a UTF-8 encoded Unicode code point that can be contained in a string that conforms to RFC 5890
RFC 5890: IDNA Definitions and Document Framework is a new addition to our list. It is the "Definitions" part of the IDNA2008 ("Internationalized Domain Names for Applications", released in 2008) specification. The RFC 5892 we encountered earlier belongs to the same specification suite.
However, there is not a single "string" that "conforms" to RFC 5890. RFC 7622 §3.2.1 has a more precise requirement:
the string consists only of Unicode code points that are allowed in NR-LDH labels or U-labels as defined in RFC5890.
An NR-LDH (non-reserved letter, digit, hyphen) label is an ASCII label (not containing "special" Unicode characters) according to the "hostname" syntax defined in RFC 952 back in 1982.
IDNA-valid U-labels
The U-label definition can be found in RFC 5890 §2.3.2.1:
[A U-label] is also subject to the constraints about permitted characters that are specified in Section 4.2 of the Protocol document and the rules in the Sections 2 and 3 of the Tables document [...].
Rant about RFC rendering
The links in the quoted paragraph are pointing to the wrong RFC, so I disarmed them in the quote above.
The normative RFC format before RFC 8650 (late 2019) was fixed-width ASCII, 58 lines, 72 characters with manual page breaks, designed to be printed by a 1982 line printer on US Legal (even though the PDF renderings are using US Letter). The markup in the HTML versions linked from this post is auto-generated from a semantic analysis of the normative ASCII documents.
The fixed-width fixed-page format is unreadable on mobile devices, and effectively trips up reflow algorithms. There used to be an alternative ebook rendering of RFCs that was the only useful way for people with bad eyes to read RFCs. It stopped rendering new documents in 2019 and was abandoned in 2022. Nobody cared.
The links above point to sections of RFC 5890, because the string parser saw "Section x.y.z" and assumed it to be a reference to section x.y.z of the current RFC. It was not. The links should go to RFC 5891 §4.2, RFC 5892 §2 and §3.
U-label definition
Let's get back to the U-label definition from RFC 5890 §2.3.2.1. It is a variant of the "IDNA-valid string":
For IDNA-aware applications, the three types of valid labels are "A-labels", "U-labels", and "NR-LDH labels" [...]
A string is "IDNA-valid" if it meets all of the requirements of these specifications for an IDNA label. [...]
[A U-label] is also subject to the constraints about permitted characters that are specified in Section 4.2 of the Protocol document and the rules in the Sections 2 and 3 of the Tables document [...].
So. Uhm. A U-label needs to be IDNA-valid, and an IDNA-valid string is either a U-label, an A-label or an NR-LDH label. This is not a recursive definition!
The referenced RFC 5891 §4.2 Permitted Character and Label Validation further clarifies:
The candidate Unicode string MUST NOT contain characters that appear in the "DISALLOWED" and "UNASSIGNED" lists specified in the Tables document.
Furthermore, it may not begin or end with a "-", and must not contain a "--" at the third position, in order to not be mixed up with A-labels.
An U-label can be up to 252 bytes (not characters!) long (§4.2), but its ASCII-compatible encoding (ACE / A-label) form must not exceed 63 ASCII characters (equal to bytes!). In addition, DNS limits the full hostname to 255 characters.
Valid U-label characters
The set of valid characters is defined by RFC 5892 §2.
A minor detail that we omitted above, when talking about valid localpart
characters, was that RFC 8264 in fact does not define the
character categories, but instead contains references to the respective
subsections of RFC 5892 §2.
Despite of that, the valid character sets for localpart and domainpart are
not equal. ß U+00DF LATIN SMALL LETTER SHARP S is explicitly included for
U-labels (I haven't figured out why it would be disallowed though), as is 〇
U+3007 IDEOGRAPHIC NUMBER ZERO (Nl) (which is in the disallowed "Letter
Number" (Nl) category) and there is a number of other
exceptions.
Korean is restricted to modern Hangul syllable characters.
IDNA is using case folding to normalize the letter case. This matches the lowercase conversion of RFC 8265, except when it doesn't.
Furthermore, DNS Registries are allowed to restrict the valid characters for domain names, probably in order to limit homoglyph attacks.
Putting domainpart together
There is a significant overlap between localpart and domainpart. However,
- U+002D HYPHEN-MINUS is the only special character from the ASCII set
that's still allowed, and it may not appear in all positions.
Lowercase letters (or uppercase Cherokee) and numbers are allowed, ASCII smileys are not. Egyptian hieroglyphs and diacritics are still in the game for subdomains, or if your Registry allows them on the domain name.
To prove a point, this post is reachable via ḧ̴͖́e̷͚̿-̸̧͘c̴͖͌o̴̻̊m̷͕̂e̷͔͊t̷͚̊h̵̦̄.op-co.de and there is an XMPP server, too:

The resource part (a.k.a. chatroom nickname)
RFC 7622 §3.4 is
where the resourcepart gets defined:
The resourcepart of a JID is an instance of the
OpaqueStringprofile of the PRECISFreeformClass, which is specified in RFC7613.
This is actually the same mechanism as with localpart, just with a different
profile and a different class.
Characters in FreeformClass
RFC 8264 §4.3.1
defines the valid FreeformClass code points. This includes all traditional
letters and numbers, printable ASCII (U+0021 through U+007E), punctuation,
spaces, and 🚨 symbols‼️ 🤯 Finally!
On top, OpaqueString will apply
some normalization,
including the conversion of all non-ASCII whitespace into U+0020. Character
case will be retained.
Stripping nicknames
RFC 7622 §3.4.1
also has a note regarding the use of resourcepart for nicknames:
In some contexts, it might be appropriate to apply more restrictive rules to the preparation, enforcement, and comparison of XMPP resourceparts. For example, in XMPP Multi-User Chat [XEP-0045] it might be appropriate to apply the rules specified in [PRECIS-Nickname].
"it might be appropriate" is not normative language, right? The Nickname
profile is derived from FreeformClass and is a mapping that removes leading
and trailing whitespace, and reduces consecutive whitespace into one U+0020.
And it applies the lowercase transformation for nickname comparisons, to
disallow multiple users to have the same case-normalized nickname.
That's it.
So you can have all the Emojis as your nickname, right? RIGHT?
Hysterical raisins
Jabber was born in 1999. The first formal XMPP specification was RFC 3920 in 2004. Over the decades, both the XMPP specification and the Unicode standard evolved, thus also changing what is considered a valid XMPP address. Implementations that we need to interoperate with might be running on some older version of the specification, and accept a different subset of "valid" Unicode characters.
Let's sort this out as well!
2004: The Original Specification
RFC 3920 §3 Addressing Scheme defines the JID syntax:
- A "domain identifier" (later renamed to
domainpart) is an IDNA string according to RFC 3490 (IDNA2003) and must match theNameprepprofile defined in RFC 3491. - A "node identifier" (
localpart) must match theNodeprepprofile defined in Appendix A. - A "resource identifier" (
resourcepart) must match theResourceprepprofile from Appendix B.
Nameprep, Nodeprep and Resourceprep are profiles of Stringprep (from
RFC 3454, which contains
tables with allowed and prohibited characters, as well as character mappings
to perform, based on Unicode 3.2).
Each of the profiles defines the set of tables and steps to apply. For
example, the Nameprep processing consists of three steps:
- Mapping:
- remove ("map to nothing") 27 different hyphenation characters
- apply case mapping and folding to 1676 characters ("A" ➡️"a", "𝛬" ➡️"λ", ...)
- Prohibited Output (based on tables in
RFC 3454 Appendix C):
- disallow ASCII and non-ASCII space characters (but not control characters - those are disallowed by XML 1.0, which is the mandatory foundation of XMPP)
- disallow Private Use and "non-character" ranges from Unicode, as well as surrogate codes
- disallow some inappropriate characters (like
� U+FFFD REPLACEMENT CHARACTER) and orientation markers
- Only allow unassigned code points according to IDNA rules (allowed in queries, not in "stored strings")
The handling of IPv6 literals in RFC3920 assumes that they are inserted
verbatim, with no surrounding [] and no %zone identifier.
Nodeprep is similar to Nameprep, but disallows control characters, as well
as the forbidden characters we know from localpart, namely "&'/:<>@
Resourceprep is also similar to Nameprep but allows ASCII whitespace and
doesn't perform case folding, allowing for uppercase characters.
But on the good side, neither IDNA2003 nor the Stringprep profiles disallow the use of Emojis (that are part of Unicode 3.2) in domain names! 🎉
2008: A New IDNA Hope
However, the experience of operating IDNA2003 in the wild for a few years led to the documentation of 37 pages (measured in 72-character ASCII on US Legal) of issues and shortcomings, documented in RFC 4690 and including this section:
5.1.1. Elimination of All Non-Language Characters
Unicode characters that are not needed to write words or numbers in any of the world's languages should be eliminated from the list of characters that are appropriate in DNS labels. In addition to such characters as those used for box-drawing and sentence punctuation, this should exclude punctuation for word structure and other delimiters. While DNS labels may conveniently be used to express words in many circumstances, the goal is not to express words (or sentences or phrases), but to permit the creation of unambiguous labels with good mnemonic value.
I guess that Emojis lack good mnemonic value. RIP. 🪦
The result of this analysis was the replacement of IDNA2003 with IDNA2008 in... you guessed it... 2010! To be fair, the IDNA2008 suite was "largely completed in 2008", and got submitted to the IETF in October 2008.
The IDNA2008 RFC collection obsoleted the previous RFCs, and thus the stricter
domainpart requirements (no Emojis) were automatically turned into law in
2010, without having to change any of the XMPP specifications.
But we can still have Emojis in usernames and nicknames, right? 🥹
2010 Revenge of the PRECIS
The update to IDNA made Stringprep obsolete, and prompted the creation of the Preparation and Comparison of Internationalized Strings Working Group at the IETF.
While the WG was working on the PRECIS specifications, the XMPP core specifications got a major overhaul in 2011. As part of that, the address format was updated and separated into its own document, RFC 6122:
Because all other aspects of revised documentation for XMPP have been incorporated into [XMPP], the XMPP Working Group decided to temporarily split the XMPP address format into a separate document so as not to significantly delay publication of improved documentation for XMPP. It is expected that this document will be obsoleted as soon as work on a new approach to preparation and comparison of internationalized addresses has been completed.
The updated address format still relied on IDNA2003, but developers were encouraged to look at IDNA2008.
RFC 6122 furthermore introduced the localpart, domainpart and
resourcepart names and changed IPv6 literals to use the bracketed
IP-literal syntax from RFC 3986.
2015 The Next Generation
As announced in the intro of RFC 6122, it was soon replaced by RFC 7622, which we might vaguely remember from the beginning of this post. It was published in 2015, based on the still fresh RFC 7613 PRECIS specification.
The PRECIS suite and the updated XMPP address format introduced case folding,
replaced the Stringprep profiles with the PRECIS classes, profiles and
categories explained above, and effectively disallowed Emojis in the
localpart and domainpart of XMPP addresses (following the IDNA2008
insights).
As mentioned before, RFC 7613 was obsoleted by RFC 8265, which corrected a few things and went from case folding to lowercase again. 🤷
This happened in 2017 and, together with RFC 8266 (Nicknames) is the end of the evolution of the RFCs needed to understand XMPP addresses.
So you just told me that Emojis in nicknames are still allowed, yes?
XMPP address validation in the wild
The IETF is about "rough consensus and running code". We've seen the consensus and how it changed over two decades, but in the end it's the running code that will say "no" when you try to butt dial an XMPP address.
Consensus in distributed systems
Something that you enter might go through up to five different hops (and different XMPP implementations; I'm omitting protocol bridges, but the point should be clear):
- Your own client, which is responsible for sanitizing (or refusing) your input, through its user interface or config file.
- Your server, receiving your input through a client-to-server connection.
- Optionally, a XEP-0045 Multi-User Chat (MUC) room where you are an occupant, through a server-to-server connection.
- The recipient's server, through a server-to-server connection shared with other users.
- The recipient's client, through its own connection to its server.
Your client is the easiest part, as it can simply reject forwarding something it disagrees with. If the "Add contact" button is greyed out, you've arrived at a dead-end. ⛔
The following hops on the path can't grey out the button if they consider your XMPP address, coming through an XML stream, as invalid. According to RFC 6120, they have to treat it as a (recoverable) stanza-related error, and reject the respective XML stanza (and not terminate the XML stream):
8.3.3.8.
jid-malformedThe sending entity has provided (e.g., during resource binding) or communicated (e.g., in the 'to' address of a stanza) an XMPP address or aspect thereof that violates the rules defined in [XMPP‑ADDR]; the associated error type SHOULD be "modify".
So if a recipient disagrees about the PRECIS / IDNA version with your client or your server, it will reject the respective stanza before it can be processed.
Robot Face vs. the MUC Occupants
When joining a MUC, you send a presence stanza to your occupant address,
constructed by appending your nickname as the resourcepart to the room
address. If you choose an evil Emoji nickname and the room rejects it, it will
send an error response, and you won't be able to join the room.
Now if the room does accept your nickname, it will forward the presence, sending it from your occupant address, to all other occupants.
I first ran into this issue, not knowing much about IDNA, PRECIS or Stringprep, back in 2017:
11:28:50 ---> 🤖 joined the room
11:28:50 <--- T....s has left the room (Kicked: jid malformed: The source address
is invalid: prosody@conference.prosody.im/🤖)
11:28:51 <--- N..........s has left the room (Kicked: jid malformed)
11:28:51 <--- d......n has left the room (Kicked: jid malformed: The source address
is invalid: prosody@conference.prosody.im/🤖)
11:28:51 <--- d.......o has left the room (Kicked: jid malformed: The source address
is invalid: prosody@conference.prosody.im/🤖)
11:28:51 <--- a..v has left the room (Disconnected: not-well-formed)
11:29:08 ---> a..v joined the room
11:32:18 ---> T....s joined the room
11:32:18 <--- T....s has left the room (Kicked: jid malformed: The source address
is invalid: prosody@conference.prosody.im/🤖)
Any downstream server or client that does not accept this occupant presence will send a stanza error back to the MUC. The MUC will treat that error as a non-recoverable session error and remove the respective occupants.
As long as you stay in the room, the other clients will repeatedly reconnect, receive your presence, and get kicked out. If you send a message to the room, it will get pushed to joining clients as part of the room history even after you leave.
Today, the situation is only slightly different:
00:14:39 ---> 🤖 joined the room
00:14:39 <--- H....r (....) has left the room due to an error
(Kicked: bad request)
00:14:39 <--- c...........s (Monocles) has left the room due to an error
(Kicked: bad request)
00:14:39 <--- p......d (Conversations) has left the room due to an error
(Kicked: bad request)
00:14:39 <--- E..a (Cheogram) has left the room due to an error
(Kicked: bad request)
00:14:39 <--- m.....x (Conversations) has left the room due to an error
(Kicked: bad request)
00:14:39 <--- y..h (Conversations) has left the room due to an error
(Kicked: bad request)
00:14:39 <--- b.....a (.../Conversations....) has left the room due to an error
(Kicked: bad request)
Most of the affected users seem to be running Conversations or its forks Cheogram and Monocles, and the clients(?) responded to the presence with a "bad-request" error.
In addition to that issue, ejabberd sends the error response over the wrong half of the server-to-server stream, so:
Jul 12 01:19:36 s2sout5e57d6de9d50 debug Received[s2sout]: <presence to='test@chat.yax.im/🤖' type='error' id='noLNT-110871' from='georg@conversations.im/Conversations.jxm9v159lx' xml:lang='de-DE'>
Jul 12 01:19:36 stanzarouter warn Received a stanza claiming to be from conversations.im, over a stream authed for chat.yax.im!
Jul 12 01:19:36 s2sout5e57d6de9d50 debug Disconnecting chat.yax.im->conversations.im[s2sout], <stream:error> is: <stream:error><not-authorized xmlns='urn:ietf:params:xml:ns:xmpp-streams'/></stream:error>
Looking at implementations
So it seems like there is a bit of inertia with implementations to follow a fifteen years old specification update. This warrants a look at the major implementations.
Scroll down for the summary table.
Server implementations
According to the s.j.n stats, the top 5 server implementations on the federated XMPP network are Prosody (60%), ejabberd (24%), Spectrum (6%), biboumi (4%) and "Multi User Chat" (1%). And while biboumi is the only one without its own .IM domain, we can still exclude it and Spectrum from the list, as they are bridges to other networks and need to adhere to the limitations of those networks. "Multi User Chat" is in fact the MUC component of the Tigase server.
A cross-match with the servers connected to yax.im also yields similar results, and adds Openfire as a candidate with 1.5% market share.
prosody
Lua
isn't exactly friends with Unicode,
so prosody went for a manual approach and implemented Stringprep in
encodings.c
using either ICU or
libidn, based on a compile-time switch.
The binary packages built by the prosody team use ICU, so we'll take that for
the comparison.
libICU
ICU (International Components for Unicode) has a very turbulent history - initiated at a spin-off from Apple and IBM, and written in Java, the first version got integrated into the Java SDK in 1997, then developed in parallel and ported from Java to C++ and C. prosody is using the C version. ICU supports IDNA2008, which prosody started using in 2019. However, ICU only supports Stringprep, not PRECIS (probably due to the fact that Stringprep was a required part of IDNA2003).
libidn and libidn2
libidn on the other hand started out as libstringprep, and supports IDNA2003 and Stringprep. libidn2 was created to support IDNA2008, but it removed Stringprep support, so can't be used as a drop-in replacement.
You have the choice between IDNA2003 with Stringprep and IDNA2008 without PRECIS.
ejabberd
ejabberd is written in Erlang, a language that's as powerful as it is obscure. The ejabberd developers have implemented their own stringprep library and use erlang-idna which supports both IDNA2003 and IDNA2008, but haven't tackled PRECIS.
Tigase
Tigase is written in Java and seems to have forked and heavily reformatted the December 2004 libidn 0.5.12 release. There is no mention of IDNA2008, nor of PRECIS in the source, so I would assume IDNA2003 and Stringprep.
Openfire
Openfire uses Tinder for the XMPP stanzas, and that makes use of libidn 1.35. As this is not libidn2, Openfire is at IDNA2003 and Stringprep. But there is an abandoned half-finished PR to implement PRECIS!
Client implementations
According to the JabberFR client stats, the top 5 client implementations are:
- Conversations
- Cheogram (a Conversations fork)
- Monocles (a Conversations fork)
- Gajim
- Monal
- Pidgin
- Blabber.im (an abandoned Conversations fork)
- Dino
Conversations
Conversations is a modern Android client that's making use of jxmpp-stringprep-libidn, which is using libidn 1.15 which gives us IDNA2003 and Stringprep.
Gajim
Gajim was in fact the client that told me that I'm holding it wrong and that made me write this blog post.
Gajim is using nbxmpp and nbxmpp is using precis-i18n, which implements the trifecta of 8264, 8265, and 8266! In addition, idna is used for full IDNA2008 support.
So far, Gajim is the only client that will allow Unicode >3.2 emoji in nicknames (and nowhere else)!
Monal
The authentication code is doing manual stringprep, but other than that there is no support for Stringprep or PRECIS. IDNA2008 is handled by the underlying iOS core library.
Pidgin
Pidgin. My nemesis. The formerly most-widely used XMPP client that made a generation of users believe that XMPP is stuck in 2004. Pidgin is using libpurple, which was famously called "a flock a zero days flying in formation" a decade ago.
A 2009 patch implemented IDNA2003 and Stringprep support based on libidn, and it seems to have survived in the 2.14 "stable" branch, which was last released in January 2025.
The 3.0 development branch does not contain any traces of IDNA, Stringprep or PRECIS.
Dino
Dino, a modern client written in Vala, uses a binding to
libICU, but without the UIDNA_USE_STD3_RULES flag that would enable
IDNA2008.
Implementation overview
The analysis of the client and server implementations shows that most implementations lag behind by a decade. There are two notable exceptions: Gajim implements the current state-of-the-art, and Monal allows everything and lets the server sort things out.
| Implementation | username | hostname | nicknames |
|---|---|---|---|
| Servers | |||
| prosody | 👆️ Stringprep | ❌ IDNA2008 | 👆️ Stringprep |
| ejabberd | 👆️ Stringprep | ❌ IDNA2008 | 👆️ Stringprep |
| Tigase | 👆️ Stringprep | 👆️ IDNA2003 | 👆️ Stringprep |
| Openfire | 👆️ Stringprep | 👆️ IDNA2003 | 👆️ Stringprep |
| Clients | |||
| Conversations | 👆️ Stringprep | 👆️ IDNA2003 | 👆️ Stringprep |
| Gajim | ❌ PRECIS | ❌ IDNA2008 | 🤖 PRECIS |
| Monal | 🤖 anything goes | ❌ IDNA2008 | 🤖 anything goes |
| Pidgin | 👆️ Stringprep | 👆️ IDNA2003 | 👆️ Stringprep |
| Dino | 👆️ Stringprep | 👆️ IDNA2003 | 👆️ Stringprep |
❌ = not allowed | 👆️ = legacy Unicode 3.2 | 🤖 = modern Unicode
Summary / TL;DR
The original XMPP specification (2004-2010; IDNA2003 + Stringprep) didn't
forbid Emojis in any parts of an XMPP address, but was limited to Unicode 3.2,
which only had around 150 Emojis. xmpp:👆️@♻️.❤️/⁉️
When IDNA2003 was replaced by IDNA2008 in 2010, hostnames were restricted to
characters from actual human languages. The two most widely deployed server
implementations enforce this limit, but might support pre-existing legacy
hostnames. xmpp:☹️@𓀐.𓂸/☢️
When the XMPP specification implemented PRECIS in 2017, usernames were also
limited to human languages, but the resource / nickname part was left
permissive, and opened up to all existing and future Unicode specifications.
xmpp:𓀬@ツ.ۃ/🤖
So after going through 22 years of development, 19 RFCs and 17 Unicode standards, I have to say: the internet was right and I was wrong. 👆️.op-co.de is not a valid JID, but it was unti 2010.
The goal of this post is to make an easily accessible (anonymous) webchat for any chatrooms hosted on a prosody XMPP server, using the web client converse.js.
Motivation and prerequisites
There are two use cases:
Have an easily accessible default support room for users having trouble with the server or their accounts.
Have a working "Join using browser" button on search.jabber.network
This setup will require:
A running prosody 0.12+ instance with a
muccomponent (chat.yax.imin our example)The willingness to operate an anomyous login and to handle abuse coming from it (
anon.yax.im)A web-server to host the static HTML and JavaScript for the webchat (
https://yaxim.org/)
There are other places that describe how to set up a prosody server and a web server, so our focus is on configuring anonymous access and the webchat.
Prosody: BOSH / websockets
The web client needs to access the prosody instance over HTTPS. This can
be accomplished either by using
Bidirectional-streams Over Synchronous HTTP (BOSH)
or the more modern WebSocket.
We enable both mechanisms in prosody.cfg by adding the following two
lines to the gloabl modules_enabled list, they can also be used by
regular clients:
modules_enabled = {
...
-- add HTTP modules:
"bosh"; -- Enable BOSH access, aka "Jabber over HTTP"
"websocket"; -- Modern XMPP over HTTP stream support
...
}
You can check if the BOSH endpoint works by visiting the /http-bind/
endpoint on your prosody's HTTPS port (5281 by default).
The yax.im server is using
mod_net_multiplex
to allow both XMPP with Direct TLS and HTTPS on port 443, so the
resulting URL is
https://xmpp.yaxim.org/http-bind/.
Prosody: allowing anonymous logins
We need to add a new anonymous virtual host to the server configuration. By default, anonymous domains are only allowed to connect to services running on the same prosody instance, so they can join rooms on your server, but not connect out to other servers.
Add the new virtualhost at the end of prosody.cfg.lua:
-- add at the end, after the other VirtualHost sections, add:
VirtualHost "anon.yax.im"
authentication = "anonymous"
-- to allow file uploads for anonymous users, uncomment the following
-- two lines (THIS IS NOT RECOMMENDED!)
-- modules_enabled = { "discoitems"; }
-- disco_items = { {"upload.yax.im"}; }
This is a new domain that needs to be made accessible to clients, so you
also need to create an SRV record and ensure that your TLS certificate
covers the new hostname as well, e.g. by updating the parameter list to
certbot.
_xmpp-client._tcp.anon.yax.im. 3600 IN SRV 5 1 5222 xmpp.yaxim.org.
_xmpps-client._tcp.anon.yax.im. 3600 IN SRV 5 1 443 xmpp.yaxim.org.
Converse.js webchat
Converse.js is a full XMPP client written in JavaScript. The default mode is to embed Converse into a website where you have a small overlay window with the chat, that you can use while navigating the site.
However, we want to have a full-screen chat under the /chat/ URL and
use that to join only one room at a time (either the support room or a
room address that was explicitly passed) instead. For this, Converse has
the fullscreen and singleton modes that we need to enable.
Furthermore, Converse does not (properly) support parsing room addresses
from the URL, so we are using custom JavaScript to identify whether an
address was passed as an anchor, and fall back to the support room
yaxim@chat.yax.im otherwise.
The following is based on release 10.1.6 of Converse.
Download the converse tarball (not converse-headless) and copy the
distfolder into your document root.Create a folder
chat/orwebchat/in the document root, where the static HTML will be placedCreate an
index.htmlwith the following content (minimal example):
<html lang="en">
<head>
<title>yax.im webchat</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="browser-based access to the xmpp/jabber chatrooms on chat.yax.im" />
<link type="text/css" rel="stylesheet" media="screen" href="/dist/converse.min.css" />
<script src="/dist/converse.min.js"></script>
</head>
<body style="width: 100vw; height: 100vh; margin:0">
<div id="conversejs">
</div>
<noscript><h1>This chat only works with JavaScript enabled!</h1></noscript>
<script>
let room = window.location.search || window.location.hash;
room = decodeURIComponent(room.substring(room.indexOf('#') + 1, room.length));
if (!room) {
room = "yaxim@chat.yax.im";
}
converse.initialize({
"allow_muc_invitations" : false,
"authentication" : "anonymous",
"auto_join_on_invite" : true,
"auto_join_rooms" : [
room
],
"auto_login" : true,
"auto_reconnect" : false,
"blacklisted_plugins" : [
"converse-register"
],
"jid" : "anon.yax.im",
"keepalive" : true,
"message_carbons" : true,
"use_emojione" : true,
"view_mode" : "fullscreen",
"singleton": true,
"websocket_url" : "wss://xmpp.yaxim.org:5281/xmpp-websocket"
});
</script>
</div>
</body>
</html>
.IM top-level domain Domain Name System Security Extensions Look-aside Validation DNS-based Authentication of Named Entities Extensible Messaging and Presence Protocol TLSA ("TLSA" does not stand for anything; it is just the name of the RRtype) resource record.
Okay, seriously: this post is about securing an XMPP server running on an .IM domain with DNSSEC, using yax.im as a real-life example. In the world of HTTP there is HPKP, and browsers come with a long list of pre-pinned site certificates for the who's'who of the modern web. For XMPP, DNSSEC is the only viable way to extend the broken Root CA trust model with a slightly-less-broken hierarchical trust model from DNS (there is also TACK, which is impossible to deploy because it modifies the TLS protocol, and also unmaintained).
Because the .IM TLD is not DNSSEC-signed yet, we will need to use
DLV (DNSSEC Look-aside Validation), an additional
DNSSEC trust root operated by the ISC (until the end of 2016). Furthermore, we
will need to set up the correct entries for yax.im (the XMPP service domain),
chat.yax.im (the conference domain) and xmpp.yaxim.org (the actual server
running the service).
This post has been sitting in the drafts folder for a while, but now that DANE-SRV has been promoted to Proposed Standard, it was a good time to finalize the article.
Introduction
Our (real-life) scenario is as follows: the yax.im XMPP service is run on a server named
xmpp.yaxim.org (for historical reasons, the yax.im host is a web server
forwarding to yaxim.org, not the actual XMPP server). The service furthermore
hosts the chat.yax.im conference service, which needs to be accessible from
other XMPP servers as well.
In the following, we will create
SRV DNS records
to advertise the server name, obtain a TLS certificate, configure DNSSEC on
both domains and create (signed)
DANE
records that define which certificate a client can expect when connecting.
Once this is deployed, state-level attackers will not be able to MitM users of the service simply by issuing rogue certificates, they would also have to compromise the DNSSEC chain of trust (in our case one of the following: ICANN/VeriSign, DLV, PIR or the registrar/NS hosting our domains, essentially limiting the number of states able to pull this off to one).
Creating SRV Records for XMPP
The service / server separation is made possible with the
SRV record in DNS, which is a more
generic variant of records like MX (e-mail server) or NS (domain name
server) and defines which server is responsible for a given service on a given
domain.
For XMPP, we create the following three SRV records to allow clients
(_xmpp-client._tcp), servers (_xmpp-server._tcp) and conference
participants (_xmpp-server._tcp on chat.yax.im) to connect to the right
server:
_xmpp-client._tcp.yax.im IN SRV 5 1 5222 xmpp.yaxim.org.
_xmpp-server._tcp.yax.im IN SRV 5 1 5269 xmpp.yaxim.org.
_xmpp-server._tcp.chat.yax.im IN SRV 5 1 5269 xmpp.yaxim.org.
The record syntax is: priority (5), weight (1), port (5222 for clients,
5269 for servers) and host (xmpp.yaxim.org). Priority and weight are used
for load-balancing multiple servers, which we are not using.
Attention: some clients (or their respective DNS resolvers, often hidden
in outdated, cheap, plastic junk routers provided by your "broadband" ISP)
fail to resolve SRV records, and thus fall back to the A record. If you set
up a new XMPP server, you will slightly improve your availability by ensuring
that the A record (yax.im in our case) points to the XMPP server as well.
However, DNSSEC will be even more of a challenge for them, so lets write them
off for now.
Obtaining a TLS Certificate for XMPP
While DANE allows rolling out self-signed certificates, our goal is to stay compatible with clients and servers that do not deploy DNSSEC yet. Therefore, we need a certificate issued by a trustworthy member of the Certificate Extorion ring. Currently, StartSSL and WoSign offer free certificates, and Let's Encrypt is about to launch.
Both StartSSL and WoSign offer a convenient function to generate your keypair. DO NOT USE THAT! Create your own keypair! This "feature" will allow the CA to decrypt your traffic (unless all your clients deploy PFS, which they don't) and only makes sense if the CA is operated by an Intelligence Agency.
What You Ask For...
The certificate we are about to obtain must be somehow tied to our XMPP
service. We have three different names (yax.im, chat.yax.im and
xmpp.yaxim.org) and the obvious question is: which one should be entered into
the certificate request.
Fortunately, this is easy to find out, as it is well-defined in the XMPP Core specification, section 13.7:
In a PKIX certificate to be presented by an XMPP server (i.e., a "server certificate"), the certificate SHOULD include one or more XMPP addresses (i.e., domainparts) associated with XMPP services hosted at the server. The rules and guidelines defined in [TLS‑CERTS] apply to XMPP server certificates, with the following XMPP-specific considerations:
Support for the DNS-ID identifier type [PKIX] is REQUIRED in XMPP client and server software implementations. Certification authorities that issue XMPP-specific certificates MUST support the DNS-ID identifier type. XMPP service providers SHOULD include the DNS-ID identifier type in certificate requests.
Support for the SRV-ID identifier type [PKIX‑SRV] is REQUIRED for XMPP client and server software implementations (for verification purposes XMPP client implementations need to support only the "_xmpp-client" service type, whereas XMPP server implementations need to support both the "_xmpp-client" and "_xmpp-server" service types). Certification authorities that issue XMPP-specific certificates SHOULD support the SRV-ID identifier type. XMPP service providers SHOULD include the SRV-ID identifier type in certificate requests.
[...]
Translated into English, our certificate SHOULD contain yax.im and
chat.yax.im according to [TLS-CERTS], which is "Representation and
Verification of Domain-Based Application Service Identity within Internet
Public Key Infrastructure Using X.509 (PKIX) Certificates in the Context of
Transport Layer Security (TLS)", or for short
RFC 6125.
There, section 2.1
defines that there is the CN-ID (Common Name, which used to be the only entry
identifying a certificate), one or more DNS-IDs (baseline entries usable for
any services) and one or more SRV-IDs (service-specific entries, e.g. for
XMPP). DNS-IDs and SRV-IDs are stored in the certificate as
subject alternative names (SAN).
Following the above XMPP Core quote, a CA must add support adding a DNS-ID and should add an
SRV-ID field to the certificate. Clients and servers must support both field
types. The SRV-ID is constructed according to
RFC 4985, section 2, where it
is called SRVName:
The SRVName, if present, MUST contain a service name and a domain name in the following form:
_Service.Name
For our XMPP scenario, we would need three SRV-IDs (_xmpp-client.yax.im for
clients, _xmpp-server.yax.im for servers, and _xmpp-server.chat.yax.im for
the conference service; all without the _tcp. part we had in the SRV
record). In addition, the two DNS-IDs yax.im and chat.yax.im are required
recommended by the specification, allowing the certificate to be (ab)used for
HTTPS as well.
Update: The quoted specifications allow to create an XMPP-only certificate based on SRV-IDs, that contains no DNS-IDs (and has a non-hostname CN). Such a certificate could be used to delegate XMPP operations to a third party, or to limit the impact of leaked private keys. However, you will have a hard time convincing a public CA to issue one, and once you get it, it will be refused by most clients due to lack of SRV-ID implementation.
And then there is one more thing.
RFC 7673
proposes also checking the certificate for the SRV destination
(xmpp.yaxim.org in our case) if the SRV record was properly validated,
there is no associated TLSA record, and the application user was born under
the Virgo zodiac sign.
Summarizing the different possible entries in our certificate, we get the following picture:
| Name(s) | Field Type | Meaning |
|---|---|---|
yax.im or chat.yax.im |
Common Name (CN) | Legacy name for really old clients and servers. |
yax.im chat.yax.im |
DNS-IDs (SAN) | Required entry telling us that the host serves anything on the two domain names. |
_xmpp-client.yax.im _xmpp-server.yax.im |
SRV-IDs (SAN) | Optional entry telling us that the host serves XMPP to clients and servers. |
_xmpp-server.chat.yax.im |
SRV-ID (SAN) | Optional entry telling us that the host serves XMPP to servers for chat.yax.im. |
xmpp.yaxim.org |
DNS-ID or CN | Optional entry if you can configure a DNSSEC-signed SRV record but not a TLSA record. |
...and What You Actually Get
Most CAs have no way to define special field types. You provide a list of
service/host names, the first one is set as the CN, and all of them are stored
as DNS-ID SANs. However, StartSSL offers "XMPP Certificates", which look like
they might do what we want above. Let's request one from them for yax.im and
chat.yax.im and see what we got:
openssl x509 -noout -text -in yaxim.crt
[...]
Subject: description=mjp74P5w0cpIUITY, C=DE, CN=chat.yax.im/emailAddress=hostmaster@yax.im
X509v3 Subject Alternative Name:
DNS:chat.yax.im, DNS:yax.im, othername:<unsupported>,
othername:<unsupported>, othername:<unsupported>, othername:<unsupported>
So it's othername:<unsupported>, then? Thank you OpenSSL, for your
openness! From RFC 4985 we know that "othername" is the basic type of the
SRV-ID SAN, so it looks like we got something more or less correct. Using
this script
(highlighted source, thanks Zash), we
can further analyze what we've got:
Extensions:
X509v3 Subject Alternative Name:
sRVName: chat.yax.im, yax.im
xmppAddr: chat.yax.im, yax.im
dNSName: chat.yax.im, yax.im
Alright, the two service names we submitted turned out under three different field types:
- SRV-ID (it's mising the
_xmpp-client./_xmpp-server.part and is thus invalid) - xmppAddr (this was the correct entry type in the deprecated RFC 3920 XMPP specification, but is now only allowed in client certificates)
- DNS-ID (wow, these ones happen to be correct!)
While this is not quite what we wanted, it is sufficient to allow a correctly implemented client to connect to our server, without raising certificate errors.
Configuring DNSSEC for Your Domain(s)
In the next step, the domain (in our case both yax.im and yaxim.org, but
the following examples will only list yax.im) needs to be signed with DNSSEC.
Because I'm a lazy guy, I'm using BIND 9.9, which does inline-signing (all I
need to do is create some keys and enable the feature).
Key Creation with BIND 9.9
For each domain, a zone signing key (ZSK) is needed to sign the individual records. Furthermore, a key signing key (KSK) should be created to sign the ZSK. This allows you to rotate the ZSK as often as you wish.
# create key directory
mkdir /etc/bind/keys
cd /etc/bind/keys
# create key signing key
dnssec-keygen -f KSK -3 -a RSASHA256 -b 2048 yax.im
# create zone signing key
dnssec-keygen -3 -a RSASHA256 -b 2048 yax.im
# make all keys readable by BIND
chown -R bind.bind .
To enable it, you need to configure the key directory, inline signing and automatic re-signing:
zone "yax.im" {
...
key-directory "/etc/bind/keys";
inline-signing yes;
auto-dnssec maintain;
};
After reloading the config, the keys need to be enabled in BIND:
# load keys and check if they are enabled
$ rndc loadkeys yax.im
$ rndc signing -list yax.im
Done signing with key 17389/RSASHA256
Done signing with key 24870/RSASHA256
The above steps need to be performed for yaxim.org as well.
NSEC3 Against Zone Walking
Finally, we also want to enable NSEC3 to prevent curious people from "walking the zone", i.e. retrieving a full list of all host names under our domains. To accomplish that, we need to specify some parameters for hashing names. These parameters will be published in an NSEC3PARAMS record, which resolvers can use to apply the same hashing mechanism as we do.
First, the hash function to be used. RFC 5155, section 4.1 tells us that...
"The acceptable values are the same as the corresponding field in the NSEC3 RR."
NSEC3 is also defined in RFC 5155, albeit in section 3.1.1. There, we learn that...
"The values for this field are defined in the NSEC3 hash algorithm registry defined in Section 11."
It's right there... at the end of the section:
Finally, this document creates a new IANA registry for NSEC3 hash algorithms. This registry is named "DNSSEC NSEC3 Hash Algorithms". The initial contents of this registry are:
0 is Reserved.
1 is SHA-1.
2-255 Available for assignment.
Let's pick 1 from this plethora of choices, then.
The second parameter is "Flags", which is also defined in Section 11, and must
be 0 for now (other values have to be defined yet).
The third parameter is the number of iterations for the hash function. For a
2048 bit key, it MUST NOT exceed 500.
Bind defaults to 10,
Strotman
references 330 from RFC 4641bis, but it seems that number was removed since
then. We take this number anyway.
The last parameter is a salt for the hash function (a random hexadecimal string, we use 8 bytes). You should not copy the value from another domain to prevent rainbow attacks, but there is no need to make this very secret.
$ rndc signing -nsec3param 1 0 330 $(head -c 8 /dev/random|hexdump -e '"%02x"') yaxim.org
$ rndc signing -nsec3param 1 0 330 $(head -c 8 /dev/random|hexdump -e '"%02x"') yax.im
Whenever you update the NSEC3PARAM value, your zone will be re-signed and
re-published. That means you can change the iteration count and salt value
later on, if the need should arise.
Configuring the DS (Delegation Signer) Record for yaxim.org
If your domain is on an
already-signed TLD
(like yaxim.org on .org), you need to establish a trust link from the
.org zone to your domain's signature keys (the KSK, to be precise). For
this purpose, the
delegation signer (DS) record type has
been created.
A DS record is a signed record in the parent domain (.org) that identifies
a valid key for a given sub-domain (yaxim.org). Multiple DS records can
coexist to allow key rollover. If you are running an important service, you
should create a second KSK, store it in a safe place, and add its DS in
addition to the currently used one. Should your primary name server go up in
flames, you can recover without waiting for the domain registrar to update
your records.
Exporting the DS Record
To obtain the DS record, BIND comes with the dnssec-dsfromkey tool. Just
pipe all your keys into it, and it will output DS records for the KSKs. We
do not want SHA-1 records any more, so we pass -2 as well to get the SHA-256
record:
$ dig @127.0.0.1 DNSKEY yaxim.org | dnssec-dsfromkey -f - -2 yaxim.org
yaxim.org. IN DS 42199 8 2 35E4E171FC21C6637A39EBAF0B2E6C0A3FE92E3D2C983281649D9F4AE3A42533
This line is what you need to submit to your domain registrar (using their web interface or by means of a support ticket). The information contained is:
- key tag:
42199(this is just a numeric ID for the key, useful for key rollovers) - signature algorithm:
8(RSA / SHA-256) - DS digest type:
2(SHA-256) - hash value:
35E4E171...E3A42533
However, some registrars insist on creating the DS record themselves, and
require you to send in your DNSKEY. We only need to give them the KSK (type
257), so we filter the output accordingly:
$ dig @127.0.0.1 DNSKEY yaxim.org | grep 257
yaxim.org. 86400 IN DNSKEY 257 3 8
AwEAAcDCzsLhZT849AaG6gbFzFidUyudYyq6NHHbScMl+PPfudz5pCBt
G2AnDoqaW88TiI9c92x5f+u9Yx0fCiHYveN8XE2ed/IQB3nBW9VHiGQC
CliM0yDxCPyuffSN6uJNVHPEtpbI4Kk+DTcweTI/+mtTD+sC+w/CST/V
NFc5hV805bJiZy26iJtchuA9Bx9GzB2gkrdWFKxbjwKLF+er2Yr5wHhS
Ttmvntyokio+cVgD1UaNKcewnaLS1jDouJ9Gy2OJFAHJoKvOl6zaIJuX
mthCvmohlsR46Sp371oS79zrXF3LWc2zN67T0fc65uaMPkeIsoYhbsfS
/aijJhguS/s=
Validation of the Trust Chain
As soon as the record is updated, you can check the trustworthiness of your
domain. Unfortunately, all of the available command-line tools suck. One of
the least-sucking ones is drill from
ldns. It still needs a root.key
file that contains the officially trusted DNSSEC key for the . (root)
domain. In Debian, the
dns-root-data package
places it under /usr/share/dns/root.key. Let's drill our domain name with
DNSSEC (-D), tracing from the root zone (-T), quietly (-Q):
$ drill -DTQ -k /usr/share/dns/root.key yaxim.org
;; Number of trusted keys: 1
;; Domain: .
[T] . 172800 IN DNSKEY 256 3 8 ;{id = 48613 (zsk), size = 1024b}
. 172800 IN DNSKEY 257 3 8 ;{id = 19036 (ksk), size = 2048b}
[T] org. 86400 IN DS 21366 7 1 e6c1716cfb6bdc84e84ce1ab5510dac69173b5b2
org. 86400 IN DS 21366 7 2 96eeb2ffd9b00cd4694e78278b5efdab0a80446567b69f634da078f0d90f01ba
;; Domain: org.
[T] org. 900 IN DNSKEY 257 3 7 ;{id = 9795 (ksk), size = 2048b}
org. 900 IN DNSKEY 256 3 7 ;{id = 56198 (zsk), size = 1024b}
org. 900 IN DNSKEY 256 3 7 ;{id = 34023 (zsk), size = 1024b}
org. 900 IN DNSKEY 257 3 7 ;{id = 21366 (ksk), size = 2048b}
[T] yaxim.org. 86400 IN DS 42199 8 2 35e4e171fc21c6637a39ebaf0b2e6c0a3fe92e3d2c983281649d9f4ae3a42533
;; Domain: yaxim.org.
[T] yaxim.org. 86400 IN DNSKEY 257 3 8 ;{id = 42199 (ksk), size = 2048b}
yaxim.org. 86400 IN DNSKEY 256 3 8 ;{id = 6384 (zsk), size = 2048b}
[T] yaxim.org. 3600 IN A 83.223.75.29
;;[S] self sig OK; [B] bogus; [T] trusted
The above output traces from the initially trusted . key to org, then to
yaxim.org and determines that yaxim.org is properly DNSSEC-signed and
therefore trusted ([T]). This is already a big step, but the tool lacks some
color, and it does not allow to explicitly query the domain's name servers
(unless they are open resolvers), so you can't test your config prior to going
live.
To get a better view of our DNSSEC situation, we can query some online services:
- DNS Viz by Sandia Labs: yaxim.org results
- DNSSEC Debugger by Verisign: yaxim.org results
- Livetest by Lutz Donnerhacke: yaxim.org results (German, and you need to click through the HTTPS warnings)
Ironically, neither DNSViz nor Verisign support encrypted connections via HTTPS, and Lutz' livetest is using an untrusted root.
Enabling DNSSEC Look-aside Validation for yax.im
Unfortunately, we can not do the same with our short and shiny yax.im domain.
If we try to drill it, we get the following:
$ drill -DTQ -k /usr/share/dns/root.key yax.im
;; Number of trusted keys: 1
;; Domain: .
[T] . 172800 IN DNSKEY 256 3 8 ;{id = 48613 (zsk), size = 1024b}
. 172800 IN DNSKEY 257 3 8 ;{id = 19036 (ksk), size = 2048b}
[T] Existence denied: im. DS
;; Domain: im.
;; No DNSKEY record found for im.
;; No DS for yax.im.;; Domain: yax.im.
[S] yax.im. 86400 IN DNSKEY 257 3 8 ;{id = 17389 (ksk), size = 2048b}
yax.im. 86400 IN DNSKEY 256 3 8 ;{id = 24870 (zsk), size = 2048b}
[S] yax.im. 3600 IN A 83.223.75.29
;;[S] self sig OK; [B] bogus; [T] trusted
There are two pieces of relevant information here:
[T] Existence denied: im. DS- the top-level zone assures that .IM is not DNSSEC-signed (it has noDSrecord).[S] yax.im. 3600 IN A 83.223.75.29- yax.im is self-signed, providing no way to check its authenticity.
The .IM top-level domain for Isle of Man is operated by Domicilium. A friendly support request reveals the following:
Unfortunately there is no ETA for DNSSEC support at this time.
That means there is no way to create a chain of trust from the root zone to yax.im.
Fortunately, the desingers of DNSSEC anticipated this problem. To accelerate adoption of DNSSEC by second-level domains, the concept of look-aside validation was introduced in 2006. It allows to use an alternative trust root if the hierarchical chaining is not possible. The ISC is even operating such an alternative trust root. All we need to do is to register our domain with them, and add them to our resolvers (because they aren't added by default).
After registering with DLV, we are asked to add our domain with its respective
KSK domain key entry. To prove domain and key ownership, we must further
create a signed TXT record under dlv.yax.im with a specific value:
dlv.yax.im. IN TXT "DLV:1:fcvnnskwirut"
Afterwards, we request DLV to check our domain. It queries all of the domains' DNS servers for the relevant information and compares the results. Unfortunately, our domain fails the check:
FAILURE 69.36.225.255 has extra: yax.im. 86400 IN DNSKEY 256 3 RSASHA256 ( AwEAAepYQ66j42jjNHN50gUldFSZEfShF...
FAILURE 69.36.225.255 has extra: yax.im. 86400 IN DNSKEY 257 3 RSASHA256 ( AwEAAcB7Fx3T/byAWrKVzmivuH1bpP5Jx...
FAILURE 69.36.225.255 missing: YAX.IM. 86400 IN DNSKEY 256 3 RSASHA256 ( AwEAAepYQ66j42jjNHN50gUldFSZEfShF...
FAILURE 69.36.225.255 missing: YAX.IM. 86400 IN DNSKEY 257 3 RSASHA256 ( AwEAAcB7Fx3T/byAWrKVzmivuH1bpP5Jx...
FAILURE This means your DNS servers are out of sync. Either wait until the DNSKEY data is the same, or fix your server's contents.
This looks like a combination of two different issues:
- A part of our name servers is returning
YAX.IMwhen asked foryax.im. - The DLV script is case-sensitive when it comes to domains.
Problem #1 is officially not a problem. DNS is case-insensitive, and therefore all clients that fail to accept YAX.IM answers to yax.im requests are broken. In practice, this hits not only the DLV resolver (problem #2), but also the resolver code in Erlang, which is used in the widely-deployed ejabberd XMPP server.
While we can't fix all the broken servers out there, #2 has been reported and fixed, and hopefully the fix has been rolled out to production already. Still, issue #1 needs to be solved.
It turns out that it is caused by
case insensitive response compression.
You can't make this stuff up!
Fortunately,
BIND 9.9.6
added the no-case-compress ACL, so "all you need to do" is to upgrade BIND and
enable that shiny new feature.
After checking and re-checking the dlv.yax.im TXT record with DLV, there is finally progress:
SUCCESS DNSKEY signatures validated.
...
SUCCESS COOKIE: Good signature on TXT response from <NS IP>
SUCCESS <NS IP> has authentication cookie DLV:1:fcvnnskwirut
...
FINAL_SUCCESS Success.
After your domain got validated, it will receive its look-aside validation records under dlv.isc.org:
$ dig +noall +answer DLV yax.im.dlv.isc.org
yax.im.dlv.isc.org. 3451 IN DLV 17389 8 2 C41AFEB57D71C5DB157BBA5CB7212807AB2CEE562356E9F4EF4EACC2 C4E69578
yax.im.dlv.isc.org. 3451 IN DLV 17389 8 1 8BA3751D202EF8EE9CE2005FAF159031C5CAB68A
This looks like a real success. Except that nobody is using DLV in their resolvers by default, and DLV will stop operations in 2017.
Until then, you can enable look-aside validation in your BIND and Unbound resolvers.
Lutz' livetest service supports checking DLV-backed domains as well, so let's verify our configuration:
- yax.im results (again, just click through the HTTPS warnings!)
Creating TLSA Records for HTTP and SRV
Now that we have created keys, signed our zones and established trust into them from the root (more or less), we can put more sensitive information into DNS, and our users can verify that it was actually added by us (or one of at most two or three governments: the US, the TLD holder, and where your nameservers are hosted).
This allows us to add a second, independent, trust root to the TLS
certificates we use for our web server (yaxim.org) as well as for our XMPP
server, by means of TLSA records.
These record types are defined in RFC 6698 and consist of the following pieces of information:
- domain name (i.e.
www.yaxim.org) - certificate usage (is it a CA or a server certificate, is it signed by a "trusted" Root CA?)
- selector + matching type + certificate association data (the actual certificate reference, encoded in one of multiple possible forms)
Domain Name
The domain name is the hostname in the case of HTTPS, but it's slightly
more complicated for the XMPP SRV record, because there we have the service
domain (yax.im), the conference domain (chat.yax.im) and the actual server
domain name (xmpp.yaxim.org).
The behavior for SRV TLSA handling is defined in
RFC 7673,
published as Proposed Standard in October 2015.
First, the
client must validate that the SRV response for the service domain is properly
DNSSEC-signed. Only then the client can trust that the server named in the
SRV record is actually responsible for the service.
In the next step,
the client must ensure that the address response (A for IPv4 and AAAA for
IPv6) is DNSSEC-signed as well, or fall back to the next SRV record.
If both the SRV and the A/AAAA records are properly signed, the client
must
do a TLSA lookup
for the SRV target (which is _5222._tcp.xmpp.yaxim.org for our client
users, or _5269._tcp.xmpp.yaxim.org for other XMPP servers connecting to us).
Certificate Usage
The certificate usage field can take one of four possible values. Translated into English, the possibilities are:
- "trusted" CA - the provided cert is a CA cert that is trusted by the client, and the server certificate must be signed by this CA. We could use this to indicate that our server only will use StartSSL-issued certificates.
- "trusted" server certificate - the provided cert corresponds to the certificate returned over TLS and must be signed by a trusted Root CA. We will use this to deliver our server certificate.
- "untrusted" CA - the provided CA certificate must be the one used to sign the server's certificate. We could roll out a private CA and use this type, but it would cause issues with non-DNSSEC clients.
- "untrusted" server certificate - the provided certificate must be the same as returned by the server, and no Root CA trust checks shall be performed.
The Actual Certificate Association
Now that we know the server name for which the certificate is valid and the type of certificate and trust checks to perform, we need to store the actual certificate reference. Three fields are used to encode the certificate reference.
The selector defines
whether the full certificate (0) or only the
SubjectPublicKeyInfo field (1)
is referenced. The latter allows to get the server key re-signed by a
different CA without changing the TLSA records. The former could be
theoretically used to put the full certificate into DNS (a rather bad idea for
TLS, but might be interesting for S/MIME certs).
The matching type field defines how the "selected" data (certificate or SubjectPublicKeyInfo) is stored:
- exact match of the whole "selected" data
- SHA-256 hash of the "selected" data
- SHA-512 hash of the "selected" data
Finally, the certificate association data is the certificate/SubjectPublicKeyInfo data or hash, as described by the previous fields.
Putting it all Together
A good configuration for our service is a record based on a CA-issued server
certificate (certificate usage 1), with the full certificate (selector 0)
hashed via SHA-256 (matching type 1). We can obtain the required association
data using OpenSSL command line tools:
openssl x509 -in yaxim.org-2014.crt -outform DER | openssl sha256
(stdin)= bbcc3ca09abfc28beb4288c41f4703a74a8f375a6621b55712600335257b09a9
Taken together, this results in the following entries for HTTPS on yaxim.org and www.yaxim.org:
_443._tcp.yaxim.org IN TLSA 1 0 1 bbcc3ca09abfc28beb4288c41f4703a74a8f375a6621b55712600335257b09a9
_443._tcp.www.yaxim.org IN TLSA 1 0 1 bbcc3ca09abfc28beb4288c41f4703a74a8f375a6621b55712600335257b09a9
This is also the SHA-256 fingerprint you can see in your web browser.
For the XMPP part, we need to add TLSA records for the SRV targets
(_5222._tcp.xmpp.yaxim.org for clients and _5269._tcp.xmpp.yaxim.org for
servers). There should be no need to make TLSA records for the service domain
(_5222._tcp.yax.im), because a modern client will always try to resolve SRV
records, and no DNSSEC validation will be possible if that fails.
Here, we take the SHA-256 sum of the yax.im certificate we obtained from
StartSSL, and create two records with the same type and format as above:
_5222._tcp.xmpp.yaxim.org IN TLSA 1 0 1 cef7f6418b7d6c8e71a2413f302f92fc97e57ec18b36f97a4493964564c84836
_5269._tcp.xmpp.yaxim.org IN TLSA 1 0 1 cef7f6418b7d6c8e71a2413f302f92fc97e57ec18b36f97a4493964564c84836
These fields will be used by DNSSEC-enabled clients to verify the TLS certificate presented by our XMPP service.
Replacing the Server Certificate
Now that the TLSA records are in place, it is not as easy to replace your server certificate as it was before, because the old one is now anchored in DNS.
You need to perform the following steps in order to ensure that all clients will be able to connect at any time:
- Obtain the new certificate
- Create a second set of TLSA records, for the new certificate (keep the old one in place)
- Wait for the configured DNS time-to-live to ensure that all users have received both sets of TLSA records
- Replace the old certificate on the server with the new one
- Remove the old TLSA records
If you fail to add the TLSA records and wait the DNS TTL, some clients will have cached a copy of only the old TLSA records, so they will reject your new server certificate.
Conclusion
DANE for XMPP is a chicken-and-egg problem. As long as there are no servers, it will not be implemented in the clients, and vice versa. However, the (currently unavailable) xmpp.net XMPP security analyzer is checking the DANE validation status, and GSoC 2015 brought us DNSSEC support in minidns, which soon will be leveraged in Smack-based XMPP clients.
With this (rather long) post covering all the steps of a successful DNSSEC implementation, including the special challenges of .IM domains, I hope to pave the way for more XMPP server operators to follow.
Enabling DNSSEC and DANE provides an improvement over the rather broken Root CA trust model, however it is not without controversy. tptacek makes strong arguments against DNSSEC, because it is using outdated crypto and because it doesn't completely solve the government-level MitM problem. Unfortunately, his proposal to "do nothing" will not improve the situation, and the only positive contribution ("use TACK!") has expired in 2013.
Finally, one last technical issue not covered by this post is periodic key rollover; this will be covered by a separate post eventually.
In a post from 2009 I described why XEP-0198: Stream Management is very important for mobile XMPP clients and which client and server applications support it. I have updated the post over the years with links to bug tracker issues and release notes to keep track of the (still rather sad) state of affairs. Short summary:
Servers supporting XEP-0198 with stream resumption: Prosody IM.
Clients supporting XEP-0198 with stream resumption: Gajim, yaxim.
Today, with the release of yaxim 0.8.7, the first mobile client actually supporting the specification is available! With yax.im there is also a public XMPP server (based on Prosody) specifically configured to easily integrate with yaxim.
Now is a good moment to recapitulate what we can get from this combination, and where the (mobile) XMPP community should be heading next.
So I have XEP-0198, am I good now?
Unfortunately, it is still not that easy. With XEP-0198, you can resume the previous session within some minutes after losing the TCP connection. While you are gone, the server will continue to display you as "online" to your contacts, because the session resumption is transparent to all parties.
However, if you have been gone for too long, it is better to inform your contacts about your absence by showing you as "offline". This is accomplished by destroying your session, making a later resumption impossible. It is a matter of server configuration how much time passes until that happens, and it is an important configuration trade-off. The longer you appear as "online" while actually being gone, the more frustration might accumulate in your buddy about your lack of reaction – on the other hand, if the session is terminated too early and your client reconnects right after that, all the state is gone!
Now what exactly happens to messages sent to you when the server destroys the session? In prosody, all messages pending since you disconnected are destroyed and error responses are sent back. This is perfectly legal as of XEP-0198, but a better solution would be to store them offline for later transmission.
However, offline storage is only useful if you are not connected with a different client at the same time. If you are, should the server redirect the messages to the other client? What if it already got them by means of carbon copies? How is your now-offline mobile client going to see that it missed something?
Even though XEP-0198 is a great step towards mobile messaging reliability, additional mechanisms need to be implemented to make XMPP really ready for mass-market usage (and users).
Entering Coverage Gaps
With XEP-0280: Message Carbons, all messages you send and receive on your desktop are automatically also copied to your mobile client, if it is online at that time. If you have a client like yaxim, that tries to stay online all the time and uses XEP-0198 to resume as fast as possible (on a typical 3G/WiFi change, this takes less than five seconds), you can have a completely synchronized message log on desktop and mobile.
However, if your smartphone is out of coverage for more than some minutes, the XEP-0198 session is destroyed, no message carbons are sent, and further messages are redirected to your desktop instead. When the mobile client finally reconnects, all it receives is suspicious silence.
XMPP was not designed for modern-day smartphone-based instant messaging. However, it is the best tool we have to counter the proprietary silo-based IM contenders like WhatsApp, Facebook Chat or Google Hangouts.
Therefore, we need to seek ways to provide the same (or a better) level of usability, without sacrificing the benefits of federation and open standards.
Message Synchronization
With XEP-0136: Message Archiving there is an arcane, properly over-engineered draft standard to allow a client to fetch collections of chat messages using a kind of version control system.
An easier, more modern approach is presented in
XEP-0313: Message Archive Management
(MAM). With MAM, it is much easier to synchronize the message log between a
client and a server, as the server extends all messages sent to the client
with an <archived> tag and an ID. Later it is easily possible to obtain all
messages that arrived since then by issuing a query with the last known
archive ID.
Now it is up to the client implementors to add support for MAM! So far, it has been implemented in the web-based OTalk client, more are to come probably.
End-to-End Encryption
In the light of last year's revelations, it should be clear to everybody that end-to-end encryption is an essential part of any modern IM suite. Unfortunately, XMPP is not there yet. The XMPP Ubiquitous Encryption Manifesto is a step into the right direction, enforcing encryption of client-to-server connections as well as server-to-server connections. However, more needs to be done to protect against malicious server operators and sniffing of direct client-to-client transmissions.
There is Off-The Record Messaging (OTR), which provides strong encryption for chat conversations, and at the same time ensures (cryptographic) deniability. Unfortunately, cryptographic deniability provably does not save your ass. The only conclusion from that debacle can be: do not save any logs. This imposes a strong conflict of interest on Android, where the doctrine is: save everything to SQLite in case the OOM killer comes after you.
The other issue with OTR over XMPP (which some claim is solved in protocol version 3) is managing multiple (parallel) logins. OTR needs to keep the state of a conversation (encryption keys, initialization vectors and the like). If your chat buddy suddenly changes from a mobile device to the PC, the OTR state machine is confused, because that PC does not know the latest state. The result is, your conversation degrades into a bidirectional flood of "Can't decrypt this" error messages. This can be solved by storing the OTR state per resource (a resource is the unique identifier for each client you use with your account). This fix must be incorporated into all clients, and such things tend to take time. Ask me about adding OTR to yaxim next year.
Oh, by the way. OTR also does not mix well with archiving or carbons!
There is of course also PGP, which also provides end-to-end encryption, but requires you to store your key on a mobile device (or have a separate key for it). PGP can be combined with all kinds of archiving/copying mechanisms, and you could even store the encrypted messages on your device, requiring an unlock whenever you open the conversation. But PGP is rather heavy-weight, and there is no easy key exchange mechanism (OTR excels here with the Socialist Millionaire approach).
Encrypted Media
And then there are lolcats1. The Internet was made for them. But the XMPP community kind-of missed the trend. There is XEP-0096: File Transfer and XEP-0166: Jingle to negotiate a data transmission between two clients. Both protocols allow to negotiate in-band or proxy-based data transfers without encryption. "In-band" means that your multimedia file is split into handy chunks of at most 64 kilobytes each, base64-encoded, and sent via your server (and your buddy's server), causing some significant processing overhead and possibly triggering rate limiting on the server. However, if you trust your server administrator(s), this is the most secure way to transmit a file in a standards-compliant way.
You could use PGP to manually encrypt the file, send it using one of the mentioned mechanisms, and let your buddy manually decrypt the file. Besides the usability implications (nobody will use this!), it is a great and secure approach.
But usability is a killer, and so of course there are some proposals for encrypted end-to-end communication.
WebRTC
The browser developers did it right with WebRTC. You can have an end-to-end encrypted video conference between two friggin' browsers! This must have rang some bells, and JSON is cool, so there was a proposal to stack JSON ontop of XMPP for end-to-end encryption. Obviously because security is not complicated enough on its own.
XMPP Extensions Graveyard
Then there are ESessions, a deferred XEP from 2007, and Jingle-XTLS, which didn't even make it into an XEP, but looks promising otherwise. Maybe somebody should implement it, just to see if it works.
Custom Extensions
In the OTR specification v3, there is an additional mechanism to exchange a key for symmetric data encryption. This can be used to encrypt a file transmission or stream, in a non-standard way.
This is leveraged by CryptoCat, which is known for its security. CryptoCat is splitting the file into chunks of 64511 bytes (I am sure this is completely reasonable for an algorithm working on 16-byte blocks, so it needs to be applied 4031.9375 times), with the intention to fit them into 64KB transmission units for in-band transmission. AES256 is used in CTR mode and the transmissions are secured by HMAC-SHA512.
In ChatSecure, the OTR key exchange is leveraged even further, stacking HTTP on top of OTR on top of XMPP messages (on top of TLS on top of TCP). This might allow for fast results and a high degree of (library) code reuse, but it makes the protocol hard to understand, and in-the-wild debugging even harder.
A completely different path is taken by Jitsi, where Jingle VoIP sessions are protected using the Zimmerman RTP encryption scheme. Unfortunately, this mechanism does not transfer to file exchange.
And then iOS...
All the above only works on devices where you can keep a permanent connection to an XMPP server. Unfortunately, there is a huge walled garden full of devices that fail this simple task2. On Apple iOS, background connections are killed after a short time, the app developer is "encouraged" to use Apple's Push Service instead to notify the user of incoming chat messages.
This feature is so bizarre, you can not even count on the OS to launch your app if a "ping" message is received, you need to send all the content you want displayed in the user notification as part of the push payload. That means that as an iOS IM app author you have the choice between sacrificing privacy (clear-text chat messages sent to Apple's "cloud") or usability (display the user an opaque message in the kind of "Somebody sent you a message with some content, tap here to open the chat app to learn more").
And to add insult to injury, this mechanism is inherently incompatible with XMPP. If you write an XMPP client, your users should have the free choice of servers. However, as a client developer you need to centrally register your app and your own server(s) for Apple's push service to work.
Therefore, the iOS XMPP clients divide into two groups. In the first group there are apps that do not use Apple Push, that maintain your privacy but silently close the connection if the phone screen is turned off or another app is opened.
In the second group, there are apps that use their own custom proxy server, to which they forward your XMPP credentials (yes, your user name and password! They better have good privacy ToS). That proxy server then connects to your XMPP server and forwards all incoming and outgoing messages between your server and the app. If the app is killed by the OS, the proxy sends notifications via Apple Push, ensuring transparent functionality. Unfortunately, your privacy falls by the wayside, leaving a trail of data both with the proxy operators and Apple.
So currently, iOS users wishing for XMPP have the choice between broken security and broken usability – well done, Apple! Fortunately, there is light at the end of the tunnel. The oncoming train is an XEP proposal for Push Notifications (slides with explanation). It aims at separating the XMPP client, server, and push service tasks. The goal is to allow an XMPP client developer to provide their own push service, which the client app can register with any XMPP server. After the client app is killed, the XMPP server will inform the push service about a new message, which in turn informs Apple's (or any other OS vendor's) cloud, which in turn sends a push message to the device, which the user then can use to re-start the app.
This chain reaction is not perfect, and it does not solve the message-content privacy issue inherent to cloud notifications, but it would be a significant step forward. Let us hope it will be specified and implemented soon!
Summary
So we have solved connection stability (except on iOS). We know how to tackle synchronization of the message backlogs between mobile and desktop clients. Client connections are encrypted using TLS in almost all cases, server-to-server connections will follow soon (GMail, I am looking at you!).
End-to-end encryption of individual messages is well-handled by OTR, once all clients switch to storing the encryption state per resource. Group chats are out of luck currently.
The next big thing is to create an XMPP standard extension for end-to-end encryption of streaming data (files and real-time), to properly evaluate its security properties, and to implement it into one, two and all the other clients. Ideally, this should also cover group chats and group file sharing (e.g. on top of XEP-0214: File Repository and Sharing plus XEP-0329: File Information Sharing).
If we can manage that, we can also convince all the users of WhatsApp, Facebook and Google Hangouts to switch to an open protocol that is ready for the challenges of 2014.
transmitted from one place to another. For the sake of this discussion, streaming content is considered as "multimedia" as much as the transmission of image, video or other files.
because it prevents evil apps from eating the device battery in the background. I am sure it is a feature indeed – one intended to route all your IM traffic through an infinite loop).
XEP-0198: Stream Management is an XMPP extension adding stanza acknowledgements and stream resumption.
What does that mean in English? If you are using Jabber on your cell phone, you are not going to lose messages anymore whenever you get out of service coverage.
So, why do we need it?
Even though Jabber is using TCP, which is called a reliable protocol, messages can be lost when a user loses connectivity: the server still has an open TCP connection to your client, sends the message, and the message vanishes because you are gone. In the good case, the connection is immediately closed by your network provider, causing further messages to be stored on the server. In the bad case however, there is no reply (because your provider employs paranoid admins), and the server takes the usual TCP timeout (several minutes to hours) before taking you offline.
How can XEP-0198 save our lives?
The new proposal can not magically keep the connection while you are offline. However, it introduces two important elements to provide you with the messages you missed when you come back:
stanza acknowledgement allows both the server and the client to request a reply (acknowledgement) on every sent message or a group of messages. Once this reply is received, the messages are considered as delivered.
stream resumption is used when a connection is re-established. By including sequence numbers in the stanza acknowledgements, both client and server can tell each other which messages they received before the interruption, requesting to re-send later messages.
By combining these two enhancements, it is possible to provide a delivery guarantee for every message between you and your server. The same mechanism can also be used on S2S (server to server) links, however these are generally far more reliable.
XEP-0198 also allows to implement throttling of connections on busy servers,
however this does not have such a big effect on the subjective reliability of
Jabber.
Update, 2012-03-31: Protocol Versions
Over the years, the XEP-0198 specification developed, changing the XML stream format on the way. Fortunately, they introduced a protocol version to check for compatibility. Unfortunately, the authors who bothered enough to implement the proposal, only added one single version of the spec, providing no backward compatibility:
| Version | Namespace | Annotations | Implementations |
|---|---|---|---|
| 0.5 (2008-09-29) | urn:xmpp:sm:0 | First version with an official namespace and protocol versioning. | ejabberd |
| 0.6 (2009-03-19) | urn:xmpp:sm:1 | Renamed attributes, added (later deprecated) stanzas attribute. | |
| 0.8 (2009-04-09) | urn:xmpp:sm:2 | Simplified protocol. Added (later out-specced) throttling element <t/>. | Gajim, M-Link, Prosody, Psi, Swift(en)/Stroke |
| 1.2 (2011-03-02) | urn:xmpp:sm:3 | Further simplified protocol. Removed throttling, per-number acks (nobody used these anyway). | |
| 1.3 (2011-06-29) | Clarifications in the spec. |
Where can we get it?
Unfortunately, after XEP-0198 has been proposed to Draft Standard on 2009-06-17, not much has happened. It took some more years for implementations to appear in the wild. The following applications already support the extension, or parts of it (Update: changed from XEP version to application version):
- Gajim 0.15+ (Gnome-oriented XMPP client)
- jabberd2 (deprecated version of the XEP, incompatible to existing clients)
- M-Link ACKs, no stream resumption, as of R15+ (commercial XMPP server by isode)
- Prosody (a lightweight Jabber server written in Lua): 0.8+ with mod_smacks
- Psi (multi-platform XMPP client): forks of Psi and Iris, no official support
- Swift 1.0+ ("your friendly chat client", multi-platform) implements ACKs, but no stream resumption, by means of Swiften, its embedded XMPP library
- Stroke (a Java port of Swiften)
- yaxim 0.8.7+ (yet another Android client, maintained by yours truly)
If your favorite server/client is not in that list, there is something you can do:
- provide patches for your favourite Jabber application.
- kindly request the authors to implement it.
- support existing requests.
Here you can find related requests for different Jabber software:
- SMACK (Updated May 2011, original Forum thread) (a Java client library usable in mobile phones)
- BEEM (Android XMPP client based on SMACK)
- ejabberd (Updated Apr 2014: Up-to-date patchset) (scalable Jabber server written in Erlang)
- ChatSecure (Updated Jul 2012: Security-focused Android client)
- Pidgin (multi-platform multi-protocol IM client)
- xabber (Updated Jul 2012: Android client)
(Thanks to Anonymous commenter for additional bug tracker links)
Notes
There are several related extensions, like XEP-0184: Message Receipts or XEP-0199: XMPP Ping. However, these do not provide the reliability and efficiency of XEP-0198.
If you know of a client or server that supports XEP-0198, please feel free to leave a comment or drop an email to <georg@op-co.de>.