Posts related to the Android platform and Android software development.

Internet security is hard. TLS is almost impossible. Implementing TLS correctly in Java is Nightmare! While the higher-level HttpsURLConnection and Apache's DefaultHttpClient do it (mostly) right, direct users of Java SSL sockets (SSLSocket/ SSLEngine, SSLSocketFactory) are left exposed to Man-in-the-Middle attacks, unless the application manually checks the hostname against the certificate or employs certificate pinning.

The SSLSocket documentation claims that the socket provides "Integrity Protection", "Authentication", and "Confidentiality", even against active wiretappers. That impression is underscored by rigorous certificate checking performed when connecting, making it ridiculously hard to run development/test installations. However, these checks turn out to be completely worthless against active MitM attackers, because SSLSocket will happily accept any valid certificate (like for a domain owned by the attacker). Due to this, many applications using SSLSocket can be attacked with little effort.

This problem has been written about, but CVE-2014-5075 shows that it can not be stressed enough.

Affected Applications

This problem affects applications that make use of SSL/TLS, but not HTTPS. The best candidates to look for it are therefore clients for application-level protocols like e-mail (POP3/IMAP), instant messaging (XMPP), file transfer (FTP). CVE-2014-5075 is the respective vulnerability of the Smack XMPP client library, so this is a good starting point.

XMPP Clients

XMPP clients based on Smack (which was fixed on 2014-07-22):

Other XMPP clients:

Not Vulnerable Applications

The following applications have been checked as well, and contained code to compensate for SSLSockets shortcomings:

  • Jitsi (OSS conferencing client)
  • K9-Mail (Android e-Mail client)
  • Xabber (Based on Smack, but using its own hostname verification)

Background: Security APIs in Java

The amount of vulnerable applications can be easily explained after a deep dive into the security APIs provided by Java (and its offsprings). Therefore, this section will handle the dirty details of trust (mis)management in the most important implementations: old Java, new Java, Android and in Apache's HttpClient.

Java SE up to and including 1.6

When network security was added into Java 1.4 with the JSSE (and we all know how well security-as-an-afterthought works), two distinct APIs have been created for certificate verification and for hostname verification. The rationale for that decision was probably that the TLS/SSL handshake happens at the socket layer, whereas the hostname verification depends on the application-level protocol (HTTPS at that time). Therefore, the X509TrustManager class for certificate trust checks was integrated into the low-level SSLSocket and SSLEngine classes, whereas the HostnameVerifier API was only incorporated into the HttpsURLConnection.

The API design was not very future-proof either: X509TrustManager's checkClientTrusted() and checkServerTrusted() methods are only passed the certificate and authentication type parameters. There is no reference to the actual SSL connection or its peer name. The only workaround to allow hostname verification through this API is by creating a custom TrustManager for each connection, and storing the peer's hostname in it. This is neither elegant nor does it scale well with multiple connections.

The HostnameVerifier on the other hand has access to both the hostname and the session, making a full verification possible. However, only HttpsURLConnection is making use of a HostnameVerifier (and is only asking it if it determines a mismatch between the peer and its certificate, so the default HostnameVerifier always fails).

Besides of the default HostnameVerifier being unusable due to always failing, the API has another subtle surprise: while the TrustManager methods fail by throwing a CertificateException, HostnameVerifier.verify() simply returns false if verification fails.

As the API designers realized that users of the raw SSLSocket might fall into a certificate verification trap set up by their API, they added a well-buried warning into the JSSE reference guide for Java 5, which I am sure you have read multiple times (or at least printed it and put it under your pillow):

IMPORTANT NOTE: When using raw SSLSockets/SSLEngines you should always check the peer's credentials before sending any data. The SSLSocket/SSLEngine classes do not automatically verify, for example, that the hostname in a URL matches the hostname in the peer's credentials. An application could be exploited with URL spoofing if the hostname is not verified.

Of course, URLs are only a thing in HTTPS, but you get the point... provided that you actually have read the reference guide... up to this place. If you only read the SSLSocket marketing reference article, and thought that you are safe because it does not mention any of the pitfalls: shame on you!

And even if you did read the warning, there is no hint about how to implement the peer credentials checks. There is no API class that would perform this tedious and error-prone task, and implementing it yourself requires a Ph.D. degree in rocket surgery, as well as deep knowledge of some related Internet standardsx.

x Side note: even if you do not believe SSL conspiracy theories, or theories confirmed facts about the deliberate manipulation of Internet standards by NSA and GCHQ, there is one prominent example of how the implementation of security mechanisms can be aggravated by adding complexity - the title of RFC 6125: "Representation and Verification of Domain-Based Application Service Identity within Internet Public Key Infrastructure Using X.509 (PKIX) Certificates in the Context of Transport Layer Security (TLS)".

Apache HttpClient

The Apache HttpClient library is a full-featured HTTP client written in pure Java, adding flexibility and functionality in comparison to the default HTTP implementation.

The Apache library developers came up with their own API interface for hostname verification, X509HostnameVerifier, that also happens to incorporate Java's HostnameVerifier interface. The new methods added by Apache are expected to throw SSLException when verification fails, while the old method still returns true or false, of course. It is hard to tell if this interface mixing is adding confusion, or reducing it. One way or the other, it results in the appropriate glue code:

public final boolean verify(String host, SSLSession session) {
    try {
        Certificate[] certs = session.getPeerCertificates();
        X509Certificate x509 = (X509Certificate) certs[0];
        verify(host, x509);
        return true;
    }
    catch(SSLException e) {
        return false;
    }
}

Based on that interface, AllowAllHostnameVerifier, BrowserCompatHostnameVerifier, and StrictHostnameVerifier were created, which can actually be plugged into anything expecting a plain HostnameVerifier. The latter two also actually perform hostname verification, as opposed to the default verifier in Java, so they can be used wherever appropriate. Their difference is:

The only difference between BROWSER_COMPATIBLE and STRICT is that a wildcard (such as "*.foo.com") with BROWSER_COMPATIBLE matches all subdomains, including "a.b.foo.com".

If you can make use of Apache's HttpClient library, just plug in one of these verifiers and have a happy life:

sslSocket = ...;
sslSocket.startHandshake();
HostnameVerifier verifier = new StrictHostnameVerifier();
if (!verifier.verify(serviceName, sslSocket.getSession())) {
    throw new CertificateException("Server failed to authenticate as " + serviceName);
}
// NOW you can send and receive data!

Android

Android's designers must have been well aware of the shortcomings of the Java implementation, and the problems that an application developer might encounter when testing and debugging. They created the SSLCertificateSocketFactory class, which makes a developer's life really easy:

  1. It is available on all Android devices, starting with API level 1.

  2. It comes with appropriate warnings about its security parameters and limitations:

    Most SSLSocketFactory implementations do not verify the server's identity, allowing man-in-the-middle attacks. This implementation does check the server's certificate hostname, but only for createSocket variants that specify a hostname. When using methods that use InetAddress or which return an unconnected socket, you MUST verify the server's identity yourself to ensure a secure connection.

  3. It provides developers with two easy ways to disable all security checks for testing purposes: a) a static getInsecure() method (as of API level 8), and b)

    On development devices, "setprop socket.relaxsslcheck yes" bypasses all SSL certificate and hostname checks for testing purposes. This setting requires root access.

  4. Uses of the insecure instance are logged via adb:

    Bypassing SSL security checks at caller's request
    

    Or, when the system property is set:

    *** BYPASSING SSL SECURITY CHECKS (socket.relaxsslcheck=yes) ***
    

Some time in 2013, a training article about Security with HTTPS and SSL was added, which also features its own section for "Warnings About Using SSLSocket Directly", once again explicitly warning the developer:

Caution: SSLSocket does not perform hostname verification. It is up the your app to do its own hostname verification, preferably by calling getDefaultHostnameVerifier() with the expected hostname. Further beware that HostnameVerifier.verify() doesn't throw an exception on error but instead returns a boolean result that you must explicitly check.

Typos aside, this is very true advice. The article also covers other common SSL/TLS related problems like certificate chaining, self-signed certs and SNI. A must read! The fact that it does not mention the SSLCertificateSocketFactory is only a little snag.

Java 1.7+

As of Java 1.7, there is a new abstract class X509ExtendedTrustManager that finally unifies the two sides of certificate verification:

Extensions to the X509TrustManager interface to support SSL/TLS connection sensitive trust management.

To prevent man-in-the-middle attacks, hostname checks can be done to verify that the hostname in an end-entity certificate matches the targeted hostname. TLS does not require such checks, but some protocols over TLS (such as HTTPS) do. In earlier versions of the JDK, the certificate chain checks were done at the SSL/TLS layer, and the hostname verification checks were done at the layer over TLS. This class allows for the checking to be done during a single call to this class.

This class extends the checkServerTrusted and checkClientTrusted methods with an additional parameter for the socket reference, allowing the TrustManager to obtain the hostname that was used for the connection, thus making it possible to actually verify that hostname.

To retrofit this into the old X509TrustManager interface, all instances of X509TrustManager are internally wrapped into an AbstractTrustManagerWrapper that performs hostname verification according to the socket's SSLParameters. All this happens transparently, all you need to do is to initialize your socket with the hostname and then set the right params:

SSLParameters p = sslSocket.getSSLParameters();
p.setEndpointIdentificationAlgorithm("HTTPS");
sslSocket.setSSLParameters(p);

If you do not set the endpoint identification algorithm, the socket will behave in the same way as in earlier versions of Java, accepting any valid certificate.

However, if you do run the above code, the certificate will be checked against the IP address or hostname that you are connecting to. If the service you are using employs DNS SRV, the hostname (the actual machine you are connecting to, e.g. "xmpp-042.example.com") might differ from the service name (what the user entered, like "example.com"). However, the certificate will be issued for the service name, so the verification will fail. As such protocols are most often combined with STARTTLS, you will need to wrap your SSLSocket around your plain Socket, for which you can use the following code:

sslSocket = sslContext.getSocketFactory().createSocket(
        plainSocket,
        serviceName, /**< set your service name here */
        plainSocket.getPort(),
        true);
// set the socket parameters here!

API Confusion Conclusion

To summarize the different "platforms":

  • If you are on Java 1.6 or earlier, you are screwed!
  • If you have Android, use SSLCertificateSocketFactory and be happy.
  • If you have Apache HttpClient, add a StrictHostnameVerifier.verify() call right after you connect your socket, and check its return value!
  • If you are on Java 1.7 or newer, do not forget to set the right SSLParameters, or you might still be screwed.

Java SSL In the Literature

There is a large amount of good and bad advice out there, you just need to be a farmer security expert to separate the wheat from the chaff.

Negative Examples

The most expensive advice is free advice. And the Internet is full of it. First, there is code to let Java trust all certificates, because self-signed certificates are a subset of all certificates, obviously. Then, we have a software engineer deliberately disable certificate validation, because all these security exceptions only get into our way. Even after the Snowden revelations, recipes for disabling SSL certificate validation are still written. The suggestions are all very similar, and all pretty bad.

Admittedly, an encrypted but unvalidated connection is still a little bit better than a plaintext connection. However, with the advent of free WiFi networks and SSL MitM software, everybody with a little energy can invade your "secure" connections, which you use to transmit really sensitive information. The effect of this can reach from funny over embarassing and up to life-threatening, if you are a journalist in a crisis zone.

The personal favorite of the author is this SO question about avoiding the certificate warning message in yaxim, which is caused by MemorizingTrustManager. It is especially amusing how the server's domain name is left intact in the screenshot, whereas the certificate checksums and the self-signed indication are blackened.

Fortunately, the situation on StackOverflow has been improving over the years. Some time ago, you were overwhelmed with DO_NOT_VERIFY HostnameVerifiers and all-accepting DefaultTrustManagers, where the authors conveniently forgot to mention that their code turns the big red "security" switch to OFF.

The better answers on StackOverflow at least come with a warning or even suggest certificate pinning.

Positive Examples

In 2012, Kevin Locke has created a proper HostnameVerifier using the internal sun.security.util.HostnameChecker class which seems to exist in Java SE 6 and 7. This HostnameVerifier is used with AsyncHttpClient, but is suitable for other use-cases as well.

Fahl et al. have analyzed the sad state of SSL in Android apps in 2012. Their focus was on HTTPS, where they did find a massive amount of applications deliberately misconfigured to accept invalid or mismatching certificates (probably added during app development). In a 2013 followup, they have developed a mechanism to enable certificate checking and pinning according to special flags in the application manifest.

Will Sargent from Terse Systems has an excellent series of articles on everything TLS, with videos, examples and plentiful background information. ABSOLUTELY MUST SEE!

There is even an excellent StackOverflow answer by Bruno, outlining the proper hostname validation options with Java 7, Android and "other" Java platforms, in a very concise way.

Mitigation Possibilities

So you are an app developer, and you get this pesky CertificateException you could not care less about. What can you do to get rid of it, in a secure way? That depends on your situation.

Cloud-Connected App: Certificate Pinning

If your app is always connecting to known-in-advance servers under you control (like only your company's "cloud"), employ Certificate Pinning.

If you want a cheap and secure solution, create your own Certificate Authority (CA) (and guard its keys!), deploy its certificate as the only trusted CA in the app, and sign all your server keys with it. This approach provides you with the ultimate control over the whole security infrastructure, you do not need to pay certificate extortion fees to greedy CAs, and a compromised CA can not issue certificates that would allow to MitM your app. The only drawback is that you might not be as good as a commercial CA at guarding your CA keys, and these are the keys to your kingdom.

To implement the client side, you need to store the CA cert in a key file, which you can use to create an X509TrustManager that will only accept server certificates signed by your CA:

KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
ks.load(new FileInputStream(keyStoreFile), "keyStorePassword".toCharArray());
TrustManagerFactory tmf = TrustManagerFactory.getInstance("X509");
tmf.init(ks);
SSLContext sc = SSLContext.getInstance("TLS");
sc.init(null, tmf.getTrustManagers(), new java.security.SecureRandom());
// use 'sc' for your HttpsURLConnection / SSLSocketFactory / ...

If you rather prefer to trust the establishment (or if your servers are to be used by web browsers as well), you need to get all your server keys signed by an "official" Root CA. However, you can still store that single CA into your key file and use the above code. You just won't be able to switch to a different CA later on if they try to extort more money from you.

User-configurable Servers (a.k.a. "Private Cloud"): TOFU/POP

In the context of TLS, TOFU/POP is neither vegetarian music nor frozen food, but stands for "Trust on First Use / Persistence of Pseudonymity".

The idea behind TOFU/POP is that when you connect to a server for the first time, your client stores its certificate, and checks it on each subsequent connection. This is the same mechanism as used in SSH. If you had no evildoers between you and the server the first time, later MitM attempts will be discovered. OpenSSH displays the following on a key change:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.

In case you fell victim to a MitM attack the first time you connected, you will see the nasty warning as soon as the attacker goes away, and can start investigating. Your information will be compromised, but at least you will know it.

The problem with the TOFU approach is that it does not mix well with the PKI infrastructure model used in the TLS world: with TOFU, you create one key when the server is configured for the first time, and that key remains bound to the server forever (there is no concept of key revocation).

With PKI, you create a key and request a certificate, which is typically valid for one or two years. Before that certificate expires, you must request a new certificate (optionally using a new private key), and replace the expiring certificate on the server with the new one.

If you let an application "pin" the TLS certificate on first use, you are in for a surprise within the next year or two. If you "pin" the server public key, you must be aware that you will have to stick to that key (and renew certificates for it) forever. Of course you can create your own, self-signed, certificate with a ridiculously long expiration time, but this practice is frowned upon (for self-signing and long expiration times).

Currently, some ideas exist about how to combine PKI with TOFU, but the only sensible thing that an app can do is to give a shrug and ask the user.

Because asking the user is non-trivial from a background networking thread, the author has developed MemorizingTrustManager (MTM) for Android. MTM is a library that can be plugged into your apps' TLS connections, that leverages the system's ability for certificate and hostname verification, and asks the user if the system does not consider a given certificate/hostname combination as legitimate. Internally, MTM is using a key store where it collects all the certificates that the user has permanently accepted.

Browser

If you are developing a browser that is meant to support HTTPS, please stop here, get a security expert into your team, and only go on with her. This article has shown that using TLS is horribly hard even if you can leverage existing components to perform the actual verification of certificates and hostnames. Writing such checks in a browser-compliant way is far beyond the scope of this piece.

Outlook

DNS + TLS = DANE

Besides of TOFU/POP, which is not yet ready for TLS primetime, there is an alternative approach to link the server name (in DNS) with the server identity (as represented by its TLS certificate): DNS-based Authentication of Named Entities (DANE).

With this approach, information about the server's TLS certificate can be added to the DNS database, in the form of different certificate constraint records:

  • (0) a CA constraint can require that the presented server certificate MUST be signed by the referenced CA public key, and that this CA must be a known Root CA.
  • (1) a service certificate constraint can define that the server MUST present the referenced certificate, and that certificate must be signed by a known Root CA.
  • (2) a trust anchor assertion is like a CA constraint, except it does not need to be a Root CA known to the client. This allows a server administrator to run their own CA.
  • (3) a domain issued certificate is analogous to a service certificate constraint, but like in (2), there is no need to involve a Root CA.

Multiple constraints can be specified to tighten the checks, encoded in TLSA records (for TLS association). TLSA records are always specific to a given server name and port. For example, to make a secure XMPP connection with "zombofant.net", first the XMPP SRV record (_xmpp-client._tcp) needs to be obtained:

$ host -t SRV _xmpp-client._tcp.zombofant.net
_xmpp-client._tcp.zombofant.net has SRV record 0 0 5222 xmpp.zombofant.net.

Then, the TLSA record(s) for xmpp.zombofant.net:5222 must be obtained:

$ host -t TLSA _5222._tcp.xmpp.zombofant.net     
_5222._tcp.xmpp.zombofant.net has TLSA record 3 0 1 75E6A12CFE74A2230F3578D5E98C6F251AE2043EDEBA09F9D952A4C1 C317D81D

This record reads as: the server is using a domain issued certificate (3) with the full certificate (0) represented via its SHA-256 hash (1): 75:E6:A1:2C:FE:74:A2:23:0F:35:78:D5:E9:8C:6F:25:1A:E2:04:3E:DE:BA:09:F9:D9:52:A4:C1:C3:17:D8:1D.

And indeed, if we check the server certificate using openssl s_client, the SHA-256 hash does match:

Subject: CN=zombofant.net
Issuer: O=Root CA, OU=http://www.cacert.org, CN=CA Cert Signing Authority/emailAddress=support@cacert.org
Validity
    Not Before: Apr  8 07:25:35 2014 GMT
    Not After : Oct  5 07:25:35 2014 GMT
SHA256 Fingerprint=75:E6:A1:2C:FE:74:A2:23:0F:35:78:D5:E9:8C:6F:25:1A:E2:04:3E:DE:BA:09:F9:D9:52:A4:C1:C3:17:D8:1D

Of course, this information can only be relied upon if the DNS records are secured by DNSSEC. And DNSSEC can be abused by the same entities that already can manipulate Root CAs and perform large-scale Man-in-the-Middle attacks. However, this kind of attack is made significantly harder: while a typical Root CA list contains hundreds of entries, with an unknown number of intermediate CAs each, and it is sufficient to compromise any one of them to screw you, with DNSSEC, the attacker needs to obtain the keys to your domain (zombofant.net), to your top-level domain (.net) or the master root keys (.). In addition to that improvement, another benefit of DANE is that server operators can replace (paid) Root CA services with (cheaper/free) DNS records.

However, there is a long way until DANE can be used in Java. Java's own DNS code is very limited (no SRV support, TLSA - what are you dreaming of?) The dnsjava library claims to provide partial DNSSEC verification, there is the unmaintained DNSSEC4j and the GSoC work-in-progress dnssecjava. All that remains is for somebody to step up and implement a DANETrustManager based on one of these components.

Conclusion

Internet security is hard. Let's go bake some cookies!

Comments on HN

Posted 2014-08-05 19:52 Tags: android

tl;dr

Android is using the combination of horribly broken RC4 and MD5 as the first default cipher on all SSL connections. This impacts all apps that did not care enough to change the list of enabled ciphers (i.e. almost all existing apps). This post investigates why RC4-MD5 is the default cipher, and why it replaced better ciphers which were in use prior to the Android 2.3 release in December 2010.

Preface

Some time ago, I was adding secure authentication to my APRSdroid app for Amateur Radio geolocation. While debugging its TLS handshake, I noticed that RC4-MD5 is leading the client's list of supported ciphers and thus wins the negotiation. As the task at hand was about authentication, not about secrecy, I did not care.

However, following speculations about what the NSA can decrypt, xnyhps' excellent post about XMPP clients (make sure to read the whole series) brought it into my focus again and I seriously asked myself what reasons led to it.

Status Quo Analysis

First, I fired up Wireshark, started yaxim on my Android 4.2.2 phone (CyanogenMod 10.1.3 on a Galaxy Nexus) and checked the Client Hello packet sent. Indeed, RC4-MD5 was first, followed by RC4-SHA1:

wireshark-xmpp-4.2.2.png

To quote from RFC 2246: "The CipherSuite list, passed from the client to the server in the client hello message, contains the combinations of cryptographic algorithms supported by the client in order of the client's preference (favorite choice first)." Thus, the server is encouraged to actually use RC4-MD5 if it is not explicitly forbidden by its configuration.

I crammed out my legacy devices and cross-checked Android 2.2.1 (CyanogenMod 6.1.0 on HTC Dream), 2.3.4 (Samsung original ROM on Galaxy SII) and 2.3.7 (CyanogenMod 7 on a Galaxy 5):

Android 2.2.1Android 2.3.4, 2.3.7Android 4.2.2, 4.3
DHE-RSA-AES256-SHARC4-MD5RC4-MD5
DHE-DSS-AES256-SHARC4-SHARC4-SHA
AES256-SHAAES128-SHAAES128-SHA
EDH-RSA-DES-CBC3-SHADHE-RSA-AES128-SHAAES256-SHA
EDH-DSS-DES-CBC3-SHADHE-DSS-AES128-SHAECDH-ECDSA-RC4-SHA
DES-CBC3-SHADES-CBC3-SHAECDH-ECDSA-AES128-SHA
DES-CBC3-MD5EDH-RSA-DES-CBC3-SHAECDH-ECDSA-AES256-SHA
DHE-RSA-AES128-SHAEDH-DSS-DES-CBC3-SHAECDH-RSA-RC4-SHA
DHE-DSS-AES128-SHADES-CBC-SHAECDH-RSA-AES128-SHA
AES128-SHAEDH-RSA-DES-CBC-SHAECDH-RSA-AES256-SHA
RC2-CBC-MD5EDH-DSS-DES-CBC-SHAECDHE-ECDSA-RC4-SHA
RC4-SHAEXP-RC4-MD5ECDHE-ECDSA-AES128-SHA
RC4-MD5EXP-DES-CBC-SHAECDHE-ECDSA-AES256-SHA
RC4-MD5EXP-EDH-RSA-DES-CBC-SHAECDHE-RSA-RC4-SHA
EDH-RSA-DES-CBC-SHAEXP-EDH-DSS-DES-CBC-SHAECDHE-RSA-AES128-SHA
EDH-DSS-DES-CBC-SHAECDHE-RSA-AES256-SHA
DES-CBC-SHADHE-RSA-AES128-SHA
DES-CBC-MD5DHE-RSA-AES256-SHA
EXP-EDH-RSA-DES-CBC-SHADHE-DSS-AES128-SHA
EXP-EDH-DSS-DES-CBC-SHADHE-DSS-AES256-SHA
EXP-DES-CBC-SHADES-CBC3-SHA
EXP-RC2-CBC-MD5ECDH-ECDSA-DES-CBC3-SHA
EXP-RC2-CBC-MD5ECDH-RSA-DES-CBC3-SHA
EXP-RC4-MD5ECDHE-ECDSA-DES-CBC3-SHA
EXP-RC4-MD5ECDHE-RSA-DES-CBC3-SHA
EDH-RSA-DES-CBC3-SHA
EDH-DSS-DES-CBC3-SHA
DES-CBC-SHA
EDH-RSA-DES-CBC-SHA
EDH-DSS-DES-CBC-SHA
EXP-RC4-MD5
EXP-DES-CBC-SHA
EXP-EDH-RSA-DES-CBC-SHA
EXP-EDH-DSS-DES-CBC-SHA

As can be seen, Android 2.2.1 came with a set of AES256-SHA1 ciphers first, followed by 3DES and AES128. Android 2.3 significantly reduced the security by removing AES256 and putting the broken RC4-MD5 on the prominent first place, followed by the not-so-much-better RC4-SHA1.

Wait... What?

Yes, Android versions before 2.3 were using AES256 > 3DES > AES128 > RC4, and starting with 2.3 it was now: RC4 > AES128 > 3DES. Also, the recently broken MD5 suddenly became the favorite MAC (Update: MD5 in TLS is OK, as it is combining two different variants).

As Android 2.3 was released in late 2010, speculations about the NSA pouring money on Android developers to sabotage all of us poor users arose immediately. I needed to do something, so I wrote a minimal test program (APK, source) and single-stepped it to find the origin of the default cipher list.

It turned out to be in Android's libcore package, NativeCrypto.getDefaultCipherSuites() which returns a hardcoded String array starting with "SSL_RSA_WITH_RC4_128_MD5".

Diving Into the Android Source

Going back on that file's change history revealed interesting things, like the addition of TLS v1.1 and v1.2 and its almost immediate removal with a suspicious commit message (taking place between Android 4.0 and 4.1, possible reasoning), added support for Elliptic Curves and AES256 in Android 3.x, and finally the addition of our hardcoded string list sometime before Android 2.3:

 public static String[] getDefaultCipherSuites() {
-       int ssl_ctx = SSL_CTX_new();
-       String[] supportedCiphers = SSL_CTX_get_ciphers(ssl_ctx);
-       SSL_CTX_free(ssl_ctx);
-       return supportedCiphers;
+        return new String[] {
+            "SSL_RSA_WITH_RC4_128_MD5",
+            "SSL_RSA_WITH_RC4_128_SHA",
+            "TLS_RSA_WITH_AES_128_CBC_SHA",
...
+            "SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",
+            "SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA"
+        };
 }

The commit message tells us: We now have a default cipher suite list that is chose to match RI behavior and priority, not based on OpenSSLs default and priorities. Translated into English: before, we just used the list from OpenSSL (which was really good), now we make our own list... with blackjack! ...and hookers! with RC4! ...and MD5!

The test suite comes with another hint:

// Note these are added in priority order as defined by RI 6 documentation.

That RI 6 for sure has nothing to do with MI 6, but stands for Reference Implementation, the Sun (now Oracle) Java SDK version 6.

So what the fine Google engineers did to reduce our security was merely to copy what was there, defined by the inventors of Java!

Cipher Order in the Java Runtime

In the Java reference implementation, the code responsible for creating the cipher list is split into two files. First, a priority-ordered set of ciphers is constructed in the CipherSuite class:

// Definition of the CipherSuites that are enabled by default.
// They are listed in preference order, most preferred first.
int p = DEFAULT_SUITES_PRIORITY * 2;

add("SSL_RSA_WITH_RC4_128_MD5", 0x0004, --p, K_RSA, B_RC4_128, N);
add("SSL_RSA_WITH_RC4_128_SHA", 0x0005, --p, K_RSA, B_RC4_128, N);
...

Then, all enabled ciphers with sufficient priority are added to the list for CipherSuiteList.getDefault(). The cipher list has not experienced relevant changes since the initial import of Java 6 into Hg, when the OpenJDK was brought to life.

Going back in time reveals that even in the 1.4.0 JDK, the first one incorporating the JSEE extension for SSL/TLS, the list was more or less the same:

Java 1.4.0 (2002)Java 1.4.2_19, 1.5.0 (2004)Java 1.6 (2006)
SSL_RSA_WITH_RC4_128_SHASSL_RSA_WITH_RC4_128_MD5SSL_RSA_WITH_RC4_128_MD5
SSL_RSA_WITH_RC4_128_MD5SSL_RSA_WITH_RC4_128_SHASSL_RSA_WITH_RC4_128_SHA
SSL_RSA_WITH_DES_CBC_SHATLS_RSA_WITH_AES_128_CBC_SHATLS_RSA_WITH_AES_128_CBC_SHA
SSL_RSA_WITH_3DES_EDE_CBC_SHATLS_DHE_RSA_WITH_AES_128_CBC_SHATLS_DHE_RSA_WITH_AES_128_CBC_SHA
SSL_DHE_DSS_WITH_DES_CBC_SHATLS_DHE_DSS_WITH_AES_128_CBC_SHATLS_DHE_DSS_WITH_AES_128_CBC_SHA
SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHASSL_RSA_WITH_3DES_EDE_CBC_SHASSL_RSA_WITH_3DES_EDE_CBC_SHA
SSL_RSA_EXPORT_WITH_RC4_40_MD5SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHASSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHASSL_DHE_DSS_WITH_3DES_EDE_CBC_SHASSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
SSL_RSA_WITH_NULL_MD5SSL_RSA_WITH_DES_CBC_SHASSL_RSA_WITH_DES_CBC_SHA
SSL_RSA_WITH_NULL_SHASSL_DHE_RSA_WITH_DES_CBC_SHASSL_DHE_RSA_WITH_DES_CBC_SHA
SSL_DH_anon_WITH_RC4_128_MD5SSL_DHE_DSS_WITH_DES_CBC_SHASSL_DHE_DSS_WITH_DES_CBC_SHA
SSL_DH_anon_WITH_DES_CBC_SHASSL_RSA_EXPORT_WITH_RC4_40_MD5SSL_RSA_EXPORT_WITH_RC4_40_MD5
SSL_DH_anon_WITH_3DES_EDE_CBC_SHASSL_RSA_EXPORT_WITH_DES40_CBC_SHASSL_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_DH_anon_EXPORT_WITH_RC4_40_MD5SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHASSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHASSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHASSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
TLS_EMPTY_RENEGOTIATION_INFO_SCSV

The original list resembles the CipherSpec definition in RFC 2246 from 1999, sorted numerically with the NULL and 40-bit ciphers moved down. Somewhere between the first release and 1.4.2, DES was deprecated, TLS was added to the mix (bringing in AES) and MD5 was pushed in front of SHA1 (which makes one wonder why). After that, the only chage was the addition of TLS_EMPTY_RENEGOTIATION_INFO_SCSV, which is not a cipher but just an information token for the server.

Java 7 added Elliptic Curves and significantly improved the cipher list in 2011, but Android is based on JDK 6, making the effective default cipher list over 10 years old now.

Conclusion

The cipher order on the vast majority of Android devices was defined by Sun in 2002 and taken over into the Android project in 2010 as an attempt to improve compatibility. RC4 is considered problematic since 2001 (remember WEP?), MD5 was broken in 2009.

The change from the strong OpenSSL cipher list to a hardcoded one starting with weak ciphers is either a sign of horrible ignorance, security incompetence or a clever disguise for an NSA-influenced manipulation - you decide! (This was before BEAST made the other ciphers in TLS less secure in 2011 and RC4 gained momentum again)

All that notwithstanding, now is the time to get rid of RC4-MD5, in your applications as well as in the Android core! Call your representative on the Google board and let them know!

Appendix A: Making your app more secure

If your app is only ever making contact to your own server, feel free to choose the best cipher that fits into your CPU budget! Otherwise, it is hard to give generic advice for an app to support a wide variety of different servers without producing obscure connection errors.

Update: Server-Side Changes

The cipher priority order is defined by the client, but the server has the option to override it with its own. Server operators should read the excellent best practices document by SSLLabs.

Further resources for server admins:

Changing the client cipher list

For client developers, I am recycling the well-motivated browser cipher suite proposal written by Brian Smith at Mozilla, even though I share Bruce Schneier's scepticism on EC cryptography. The following is a subset of Brian's ciphers which are supported on Android 4.2.2, and the last three ciphers are named SSL_ instead of TLS_ (Warning: BEAST ahead!).

// put this in a place where it can be reused
static final String ENABLED_CIPHERS[] = {
        "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
        "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA",
        "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA",
        "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA",
        "TLS_DHE_RSA_WITH_AES_128_CBC_SHA",
        "TLS_DHE_RSA_WITH_AES_256_CBC_SHA",
        "TLS_DHE_DSS_WITH_AES_128_CBC_SHA",
        "TLS_ECDHE_RSA_WITH_RC4_128_SHA",
        "TLS_ECDHE_ECDSA_WITH_RC4_128_SHA",
        "TLS_RSA_WITH_AES_128_CBC_SHA",
        "TLS_RSA_WITH_AES_256_CBC_SHA",
        "SSL_RSA_WITH_3DES_EDE_CBC_SHA",
        "SSL_RSA_WITH_RC4_128_SHA",
        "SSL_RSA_WITH_RC4_128_MD5",
    };

// get a new socket from the factory
SSLSocket s = (SSLSocket)sslcontext.getSocketFactory().createSocket(host, port);
// IMPORTANT: set the cipher list before calling getSession(),
// startHandshake() or reading/writing on the socket!
s.setEnabledCipherSuites(ENABLED_CIPHERS);
...

Use TLS v1.2!

By default, TLS version 1.0 is used, and the more recent protocol versions are disabled. Some servers used to be broken when contacted using v1.2, so this approach seemed a good conservative choice over a year ago.

At least for XMPP, an attempt to enforce TLS v1.2 is being made. You can follow with your own app easily:

// put this in a place where it can be reused
static final String ENABLED_PROTOCOLS[] = {
        "TLSv1.2", "TLSv1.1", "TLSv1"
    };

// put this right before setEnabledCipherSuites()!
s.setEnabledProtocols(ENABLED_PROTOCOLS);

Use NetCipher!

NetCipher is an Android library made by the Guardian Project to improve network security for mobile apps. It comes with a StrongTrustManager to do more thorough certificate checks, an independent Root CA store, and code to easily route your traffic through the Tor network using Orbot.

Use AndroidPinning!

AndroidPinning is another Android library, written by Moxie Marlinspike to allow pinning of server certificates, improving security against government-scale MitM attacks. Use this if your app is made to communicate with a specific server!

Use MemorizingTrustManager!

MemorizingTrustManager by yours truly is yet another Android library. It allows your app to ask the user if they want to trust a given self-signed/untrusted certificate, improving support for regular connections to private services. If you are writing an XMPP client or a private cloud sync app, use this!

Appendix B: Apps that do care

Android Browser

Checks of the default Android Browser revealed that at least until Android 2.3.7 the Browser was using the default cipher list of the OS, participating in the RC4 regression.

As of 4.2.2, the Browser comes with a longer, better, stronger cipher list:

ECDHE-RSA-AES256-SHA ECDHE-ECDSA-AES256-SHA SRP-DSS-AES-256-CBC-SHA SRP-RSA-AES-256-CBC-SHA DHE-RSA-AES256-SHA DHE-DSS-AES256-SHA ECDH-RSA-AES256-SHA ECDH-ECDSA-AES256-SHA AES256-SHA ECDHE-RSA-DES-CBC3-SHA ECDHE-ECDSA-DES-CBC3-SHA SRP-DSS-3DES-EDE-CBC-SHA SRP-RSA-3DES-EDE-CBC-SHA EDH-RSA-DES-CBC3-SHA EDH-DSS-DES-CBC3-SHA ECDH-RSA-DES-CBC3-SHA ECDH-ECDSA-DES-CBC3-SHA DES-CBC3-SHA ECDHE-RSA-AES128-SHA ECDHE-ECDSA-AES128-SHA SRP-DSS-AES-128-CBC-SHA SRP-RSA-AES-128-CBC-SHA DHE-RSA-AES128-SHA DHE-DSS-AES128-SHA ECDH-RSA-AES128-SHA ECDH-ECDSA-AES128-SHA AES128-SHA ECDHE-RSA-RC4-SHA ECDHE-ECDSA-RC4-SHA ECDH-RSA-RC4-SHA ECDH-ECDSA-RC4-SHA RC4-SHA RC4-MD5

Update: Surprisingly, the Android WebView class (tested on Android 4.0.4) is also using the better ciphers.

Update: Google Chrome

The Google Chrome browser (version 30.0.1599.82, 2013-10-11) serves the following list:

ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA ECDHE-ECDSA-AES256-SHA DHE-DSS-AES256-GCM-SHA384 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-SHA256 DHE-DSS-AES256-SHA256 DHE-RSA-AES256-SHA DHE-DSS-AES256-SHA AES256-GCM-SHA384 AES256-SHA256 AES256-SHA ECDHE-RSA-DES-CBC3-SHA ECDHE-ECDSA-DES-CBC3-SHA EDH-RSA-DES-CBC3-SHA EDH-DSS-DES-CBC3-SHA DES-CBC3-SHA ECDHE-RSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-ECDSA-AES128-SHA256 ECDHE-RSA-AES128-SHA ECDHE-ECDSA-AES128-SHA DHE-DSS-AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-SHA256 DHE-DSS-AES128-SHA256 DHE-RSA-AES128-SHA DHE-DSS-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA ECDHE-RSA-RC4-SHA ECDHE-ECDSA-RC4-SHA RC4-SHA RC4-MD5

This one comes with AES256-GCM and SHA384! Good work, Google! Now please go and make these the default for the Android runtime!

Update: Firefox

Firefox Browser for Android (version 24.0 from F-Droid) comes with its own cipher suite as well. However, contrary to Chrome, it is missing the GCM ciphers to mitigate the BEAST attack.

ECDHE-ECDSA-AES256-SHA ECDHE-RSA-AES256-SHA DHE-RSA-CAMELLIA256-SHA DHE-DSS-CAMELLIA256-SHA DHE-RSA-AES256-SHA DHE-DSS-AES256-SHA ECDH-RSA-AES256-SHA ECDH-ECDSA-AES256-SHA CAMELLIA256-SHA AES256-SHA ECDHE-ECDSA-RC4-SHA ECDHE-ECDSA-AES128-SHA ECDHE-RSA-RC4-SHA ECDHE-RSA-AES128-SHA DHE-RSA-CAMELLIA128-SHA DHE-DSS-CAMELLIA128-SHA DHE-RSA-AES128-SHA DHE-DSS-AES128-SHA ECDH-RSA-RC4-SHA ECDH-RSA-AES128-SHA ECDH-ECDSA-RC4-SHA ECDH-ECDSA-AES128-SHA SEED-SHA CAMELLIA128-SHA RC4-SHA RC4-MD5 AES128-SHA ECDHE-ECDSA-DES-CBC3-SHA ECDHE-RSA-DES-CBC3-SHA EDH-RSA-DES-CBC3-SHA EDH-DSS-DES-CBC3-SHA ECDH-RSA-DES-CBC3-SHA ECDH-ECDSA-DES-CBC3-SHA FIPS-3DES-EDE-CBC-SHA DES-CBC3-SHA

My favorite pick from that list: SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA.

Enabling TLSv1.2 does not change the cipher list. BEAST is mitigated in TLSv1.2, but the Lucky13 attack might still bite you.

Send In Your App!

If you have an Android app with a significant user base that has a better cipher list, let me know and I will add it to the list.

Further Reading

Posted 2013-10-14 19:06 Tags: android

tl;dr

APRSdroid is an Amateur Radio geo-location (APRS) app for Android licensed under the GPL. It started as a Scala learning experience two years ago, and has become a nice auxiliary income since, despite being Open Source and offering free downloads from the homepage. However, using Scala was not always the easiest path to go.

Project history

In mid-December of 2009 a HAM radio friend asked me: "it can't be too hard to make an Android APRS app, can it?" and because there was none yet, I started pondering. On December 31rd, instead of having fun and alcohol, I made the first commit. January 1st, at 03:37 local time, the first placeholder release 0.2 was created.

Over the course of the next weeks I discovered step-by-step what I had encumbered myself with. APRS is a protocol with a long history of organic growth, firmware limitation workarounds, many different ways to say the same thing (at least four just for position reports), countless protocol amendments and, to add insult to injury, base-91 ASCII encoding.

There was no Java code available to abstract away the protocol and to allow me to keep my sanity. So I read the spec, implemented position encoding, re-read the spec, implemented HTTP and UDP sending code, read the amendments, re-re-re-read the spec, etc.

The first usable release became 0.4 from the end of January. Because APRSdroid always was a leisure time project, phases of activity alternated with idle phases, and the app slowly grew features through 2010.

In early 2011, one year into it, I decided it was high time to make the project pay for itself. Real APRS gear (radio transceivers with GPS and packet radio support) was expensive (on the order of US$500), and the app was not only easier to use but also grew more and more features (except for direct access to the amateur radio spectrum, which does not work well on cell phones).

For some time, I went underground (by omitting git push to github and only providing nightly builds to some friends) and worked on the code behind closed doors (there were no other people contributing source anyway).

On April 1st, I decided to fool the community a little, but was not taken too seriously. In the meantime I was polishing a 1.0 release for Android Market.

Income Report

On April 18th, 2011, APRSdroid 1.0 was "commercially" launched to Android Market. It was important for me to keep up the OSS spirit, so I kept providing source code and APK files from the home page. By buying the app instead of just downloading it, the users got automatic updates and a good feeling of supporting what they liked. Also, I did not make it too obvious in the Market description that the app can be downloaded for free as well ;-)

So far, this scheme has paid off very well. Since the beginning, more than 60% of all app users actually bought it (it is possible to monitor the global user activity on the APRS network), with an average of 350 sales per month, at 2.99€ / 4.49US$ (minus the Google "tax" and subject to local income taxes).

Most users I had contact with were ready to pay for the app even though they knew they could download it for free. Only one person so far demanded the free version to be made available on Android Market (using CAPS and three consecutive Twitter messages, though, so I did not feel too pressed).

So far, I invested the income into real APRS hardware, a Desire Z (or G2 or HTC Vision) and am eagerly awaiting the availability of ICS tablets, aiming at finally adding Fragments support to the app.

Scala + Android = Pitfalls

I decided to use Scala because I do my coding in vim and Java is so crammed up with boilerplate code that you can not sensibly use it with anything but a bloated refactoring IDE. Another reason was that I do not like to repeat myself, and Java provides even less usable abstractions than the good old C language with its #define.

Scala was the language of the day, and I liked what I had read about it so far. It sounded good enough for an experiment anyways. Fortunately, people had already figured out how to make it work on Android without carrying the bloat of the full Scala runtime, so all I had to do were some refinements of build.xml.

The first warning sign was that I had to override def $tag() to work around an issue in the 2.7 beta compiler (IIRC). I complied by cargo-cult-copying the code from some place and moved on.

Another major issue was Android's AsyncTask. The API requires the developer to override protected SomeType doInBackground(OtherType... params). Unfortunately, Scala has trouble with overriding abstract varargs methods from Java, and thus your app crashes with the opaque java.lang.AbstractMethodError: abstract method not implemented exception. After triangulating the source of the problem (who would have suspected a compiler bug?), a wrapper class in Java was written. Another bunch of days well spent.

One of my biggest hopes in Scala was to be able to reduce the boilerplate for Android's numerous single-abstract-method function) parameter workarounds. Unfortunately, this problem is not yet solved in Scala, requiring to write explicit implicit conversion functions for each SAM type.

However, not everything was bad in Scala-land. Scala's traits allowed to reuse the same code in descendants of Android's Activity, ListActivity and MapActivity. Working string comparisons, type based match and a huge amount of syntactic sugar, added on top of a proper ctags config, actually made life good.

Further, the base-91 decoding was elegantly implemented as a map/reduce operation on the ASCII string. Other interesting solutions were: an UrlOpener for buttons and regex based packet matching (Warning: please do not try to understand the regexes!).

What remains in the end is build time (compilation + proguard), which is subjectively higher for the Scala app than for a Java-only project of comparable size. However, that might be due to a bug in my build.xml and so far I was not impatient enough to investigate.

Conclusion

After two years, I am really glad to have gone this way. Learning Scala was a very pleasant experience, and it improved my ability to see problems from different points of view. However, it also significantly restricted the number of people able to contribute. Of over 500 commits to APRSdroid, only three were by another developer. The APRS parsing code has been replaced by javAPRSlib, a Java library with major contributions from several other people.

APRSdroid remains my only Scala project. My other Android projects are written in Java, either because I did not want to restrict contributors, or because I did not expect the Java code to become complex enough.

Would I start a new Scala project on Android? Probably no, as it is already hard enough to find people who would like to contribute to your pet project if it is written in Java. Scala makes that almost impossible.

Would I contribute to an existing Scala project? Yes!

P.S: Starting around March 2012, I will be looking for Android/IT-Sec related freelance jobs. Check github and Android Market for my other projects.

Posted 2011-12-29 19:10 Tags: android

My love hate aversion to SyncML

Some years ago, I accidentally managed to synchronize my Nokia E65 phone to Evolution using Bluetooth, OpenSync packages from a custom repository, a huge amount of patience and a blood sacrifice to the gods of bloated binary XML protocols.

Unfortunately, soon after that my file system crashed, I reinstalled Debian and the magic setup was forever gone. Whatever I tried, all I got were opaque error messages. After many months of moot efforts, I finally gave up the transfer of events onto my phone and of phone numbers onto my PC. Sigh.

It was only last autumn that I dared challenging my luck again. After setting up a new colo box (it is serving this blog article right now) and having upgraded my Android toy-phone to an Android 2.x firmware, it was time to get my data from the good old Nokia phone to the Android device. Somehow.

The Quest of SyncML, part 1: eGroupWare

I began my quest by simply installing the current version of eGroupWare from the Debian Backports repository. Unfortunately, this version (1.6.002) is flawed with regard to SyncML. It worked partially with my cell phone, and failed miserably with Evolution.

After several days of fruitless efforts, I found a set of SyncML patches for eGroupWare written by Jörg Lehrke, which are already integrated into 1.6.003. Fortunately, eGroupWare.org is offering Debian 5.0 packages as well. I just added the following line to my /etc/apt/sources.list and installed the new version:

deb http://download.opensuse.org/repositories/server:/eGroupWare/Debian_5.0/ ./

Do not forget to import the repository key as well:

wget -O - http://download.opensuse.org/repositories/server:/eGroupWare/Debian_5.0/Release.key | apt-key add -

With the shiny new eGroupWare, I only needed to wipe my previous synchronization efforts and to enable the SyncML application for the Default user group. Et voila, I could access my new RPC server at https://<servername>/egroupware/rpc.php

Part 2: Evolution

This step does work more or less properly, an official HOWTO is available. The only thing I have not automated yet is the fact of synchronization. It still requires manually running

syncevolution <servername>

Update, 2011-05-15: If you are running debian, do not use it's default packages. After my last dist-upgrade (sid), syncevolution thought it was a good idea to parse its plaintext config files, generate an XML-based config and throw it up on me due to strange parser errors.

Uninstalling syncevolution* and using the syncevolution-evolution package from

deb http://downloads.syncevolution.org/apt unstable main

solved my troubles however.

Part 3: Nokia E65

Fortunately, Nokia already includes a SyncML client with their smartphones. It is almost trivial to set up following the official howto. However, with eGroupWare 1.6.003, I could set the SyncML version to 1.2 to obtain the full contacts information.

Fortunately, it was also very easy to add the CAcert root certificate to the Nokia device, allowing end-to-end encryption of my sensitive personal data.

Part 4: Android

Now, the real fun began. Android comes preinstalled with a well-working synchronization service which is pushing all your data to Google servers. Not that I would mind Google having the data, I just wanted to be able to snapshot my contacts and calendar whenever I need to.

There are as well clients for other synchronization protocols. ActiveSync is supported out-of-the-box (and there is the GPL'ed Z-Push ActiveSync server); Funambol and Synthesis implement SyncML on Android.

Because I already had SyncML running and Funambol is Open-Source and looked generally promising, I started my work with it. However, the Android client is "optimized" for interacting with the Funambol server (read: it interoperates with other implementations only by chance).

Besides the hell imposed on the unlucky ones trying to compile android-client for themselves instead of using the Market version, there were various compatibility issues. In addition to that, SSL verification is only possible using the certificates already stored in the system. Neither self-signed nor community-signed SSL connections are possible.

If you have root permissions, there is a workaround to add CAcert:

# adb pull /system/etc/security/cacerts.bks .
# keytool -keystore cacerts.bks -storetype BKS -provider org.bouncycastle.jce.provider.BouncyCastleProvider -storepass changeit -import -v -trustcacerts -alias cacert_org_class_3 -file cacert_class3.crt 
Certificate was added to keystore
[Storing cacerts.bks]
# adb remount
remount succeeded
# adb push cacerts.bks /system/etc/security/cacerts.bks

Nevertheless, the experience was so frustrating that I started my own project to improve SSL certificate management on Android.

After many fruitless attempts at getting reproducible synchronization with Funambol's Android client, I decided to test Synthesis. It installed, allowed me to bypass SSL certificate checking (which is not quite perfect, but at least better than no SSL at all) and synced all my contacts at first attempt. Wow! Considering the time I have put into Funambol, paying 18€ for a Synthesis license really looks inexpensive in hindsight.

However, not everything is as shiny as it looks at first. It seems, Synthesis is not providing its own calendar backend. Instead, it is using whatever is available on the device. My device however seems to be lacking any calendar providers, unless I install the Funambol client. So all in all, I am using Synthesis to synchronize events to the Funambol calendar because Funambol fails at it. Funny, isn't it?

Update: After upgrading eGroupWare to 1.8.001, I can actually synchronize my events to my Android using Funambol. Because they change much more often than my contacts, I might actually stick to this software for some more time without buying Synthesis...

Update, 2011-05-15: I finally found the "bug" responsible for my lack of contacts synchronization. I happened to have a contact with an "&" sign, which was transmitted verbatim by eGroupWare, freaking out the Funambol parser. After renaming the contact, life suddenly became great passable!

Conclusion

SyncML is a friggin' huge pile of shi bloat. Just sync your devices to Google and your experience will be great.

Posted 2011-01-09 22:57 Tags: android
Georg Lukas Cheating the Android

Sometimes it is plain frustrating to see how people try to be smarter than you and hard-code functionality which is (almost) impossible to override.

The rant (skip this section or don't complain!)

The one single most-frustrating feature of my HTC Dream/G1 phone looks like an attempt to be smart and save batteries: Whenever the phone connects to a WLAN, the 3G/mobile data connection is terminated.

"What's the problem?" you might retort. Now, most Internet applications are using TCP as their protocol of choice, and TCP maintains a connection bound to an IP address. Whenever you change your Internet access, you switch IP addresses and all existing TCP connections vanish. Your downloads are aborted, your SSH connections are closed, your IM session is terminated (or, even worse, it looks like online but is not).

The smart G* engineers of course have provided a way to detect the change of connectivity, using a NetworkConnectivityListener. They also probably implemented some really smart synchronization protocol into their binary-only applications, to improve the user experience.

However, they did not provide a way to prevent the deactivation of 3G data service. They added in some complicated code to keep a 3G data connection open to the MMS service, but the "normal" data session is just terminated whenever a WLAN is found. This would not be as bad as it sounds if such a state change would only happen twice a day (WLAN → 3G when you leave home; 3G → WLAN when you come back). However, WLAN is eating your batteries really really fast. Thus, the smart G* engineers made the phone automatically switch WLAN off one minute after the display backlight is disabled. That means: you look at the phone clock, it finds a WLAN, terminates all your connections, goes to sleep, turns off WLAN, terminates all your connections, ... GOTO 10

To add insult to injury, they added ConnectivityManager.requestRouteToHost(int networkType, int hostAddress)), which looks like it would set up a route to your destination using the specified network interface. Ha-ha! Fail! That function only works if the requested interface is already up!

For application developers, this basically means that they have to catch the CONNECTIVITY_ACTION events, terminate the stale connection and open a new connection, synchronizing all of the state between the client and server. This of course implies that the application protocol must support re-synchronization. HTTP for example provides the Range: x- header to continue a partial download. For Jabber, there is XEP-0198 (which is still missing in most implementations). Other protocols, like SSH, are basically screwed.

For developers working at a mobile carrier, this is also bad news - there is no way to access the 3G data network when the user is surfing via WLAN.

Compared to this, the Symbian way of presenting the user a list of available networks when an application opens a socket is just a heavenly dream. Sorry, smart G* developers, you f'ed up this one.

The hack

After following the PdpConnectionConnectivityManagerConnectivityService twine of bloat, I saw some really fascinating code in ConnectivityService:

if (!mTestMode && deadnet != null) {
    if (DBG) Log.v(TAG, "Policy requires " +
      deadnet.getNetworkInfo().getTypeName() + " teardown");
    toredown = teardown(deadnet);
    if (DBG && !toredown) {
    Log.d(TAG, "Network declined teardown request");
    }
}

Now, what it means is basically that if mTestMode is enabled, the old connection is not terminated when a new one is established. mTestMode is set as:

mTestMode = SystemProperties.get("cm.test.mode").equals("true")
    && SystemProperties.get("ro.build.type").equals("eng");

On a rooted phone, all we need to get it is to change /system/build.props, reboot, and call requestRouteToHost().

Fortunately, the smart G* engineers fixed this evil exploit for the 2.0 release! GOTO 10 again!

Posted 2009-12-23 01:58 Tags: android