One Line Fix: Wordpress and GitHub's SSL Cipher

It was at some point in the afternoon of Tue 2013-Dec-31 (as I remember it), when one of my site health checks started firing off. Wordpress was responding with an HTTP 500 and partial document body, ending with:

SSL connect error

I use the embed-github-gist plugin to display my code-snippet gists inline in my blog posts. It fetches them on the server side – thank goddess I cache my blog posts – and it appeared that something on Github’s end had blowed up.

TL;DR

I had to add a One Line Fix to Wordpress’ WP_Http component so that it would use the proper SSL Cipher. Scroll down towards the bottom of this post to see the code-snippet.

Whereas the rest of this post recounts the twists & turns I took in coming to that final solution, the process of which may or may not be of use to others. It certainly was a welcomed catharsis to write it all down.

What’s up with WordPress?

My /etc/php.ini has log_errors = On, and they end up in my nginx error log. I was seeing

2014/01/04 20:12:00 [error] 935#0: *95058410 FastCGI sent in stderr: "PHP Fatal error:  Cannot use object of type WP_Error as array in ~/wordpress/wp-content/plugins/embed-github-gist/embed-github-gist.php on line 86" while reading upstream

Cool, that’s useful. So, I read the source code and mock up a URL to try out in curl. It works fine here on my laptop, but on my AWS instance, I get:

% curl -vvv https://api.github.com/gists/2166671?sslverify=false

* About to connect() to api.github.com port 443 (#0)
*   Trying 192.30.252.138... connected
* Connected to api.github.com (192.30.252.138) port 443 (#0)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -12286

Well, that sslverify=false isn’t helping. Nor does using curl with -k/--insecure. So, what’s up with ‘NSS error -12286’? This post states that it’s SSL_ERROR_NO_CYPHER_OVERLAP.

I figured that need to get some better tooling to identify my core issue with GitHub and TLS. Via this post I track down a tool called gnutls-cli:

% yum install gnutls gnutls-utils
% gnutls-cli -p 443 api.github.com

- Certificate[0] info:
# The hostname in the certificate matches 'api.github.com'.
# valid since: Sun Apr 29 20:00:00 EDT 2012
# expires at: Wed Jul  9 08:00:00 EDT 2014
# fingerprint: 55:D8:B2:AC:FA:96:DF:AF:85:32:1C:0F:B2:5A:96:1D
# Subject's DN: C=US,ST=California,L=San Francisco,O=GitHub\, Inc.,CN=*.github.com
# Issuer's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance CA-3

- Certificate[1] info:
# valid since: Wed Apr  2 08:00:00 EDT 2008
# expires at: Sat Apr  2 20:00:00 EDT 2022
# fingerprint: C6:8B:99:30:C8:57:8D:41:6F:8C:09:4E:6A:DB:0C:90
# Subject's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance CA-3
# Issuer's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV Root CA

- Peer's certificate issuer is unknown
- Peer's certificate is NOT trusted

- Version: TLS 1.1
- Key Exchange: RSA
- Cipher: AES 128 CBC
- MAC: SHA

Now, it turns out that the key word here is “cypher”. But since I saw “certificate is NOT trusted”, I went down my first Rabbit Hole instead.

Rabbit Hole: New Certificate Authority

Obviously I needed to update my CA Bundle within OpenSSL, right? The bundle is installed at /etc/pki/tls/certs/ca-bundle.crt per the curl response above. I’m guessing it’s probably quite old, since I’m still running Fedora 8 (as justified later). I’ll probably need to find an alternate source.

Looking at the directory hierarchy, /etc/pki/tls/openssl.cnf gives me a clue that it’s an OpenSSL thing. Moreover, this post assures me that “The ca-bundle.crt is in the openssl rpm”. Sweet.

However, I already have the latest (0.9.8b-17) and this RPM listing doesn’t give me anything more to go on. I suspect I could build a 1.x version of OpenSSL, but that may cause interoperability issues. Crap.

I find a Fedora 19 version of the OpenSSL RPM and pull it down. This post informs me that I need to use rpm2cpio to unpack an RPM as if it were a normal archive. Ultimately though, the resulting ca-bundle.crt does not help. Crap.

So, I guess it’s time to re-build the CA Bundle from scratch, right? Well, let’s see. Curl’s Details on Server SSL Certificates suggests “Get a better/different/newer CA cert bundle!”. Rather than do it through a source download of Firefox, I went with the CA Extract approach, and downloaded a new cacert.pem. That too didn’t help. Crap.

I check with the OpenSSL FAQ: “How can I set up a bundle of commercial root CA certificates?” I can generate them from Mozilla’s current master list using this convenient little script. That’ll provide me with the latest full CA Certificate List, so that’ll totally work! Pull down the file, modify the script a bit, and voila, right?

% wget http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1
% ./mkcabundle.pl > ca-bundle.crt
% chown root:root ca-bundle.crt

No, of course it won’t. Because this is a Rabbit Hole. I even pulled down the offending certificate from DigiCert and tried to use it in isolation. gnutls-cli offers a way for me to extract the public key into a PEM, but that didn’t work, so I viewed the GitHub SSL certificate info in Google Chrome, then did some manual typing (since URL cut-and-paste was prevented for some absurd reason):

% wget --no-check-certificate https://www.digicert.com/CACerts/DigiCertHighAssuranceCA-3.crt
% openssl x509 -inform DER -outform PEM -in DigiCertHighAssuranceCA-3.crt -out DigiCertHighAssuranceCA-3.pem

Somehow, no matter what I do with the CA Bundle, it’s not helping. Also, something seems fishy, because the configuration in /etc/pki/tls/openssl.cnf doesn’t seem to match reality.

NSS

So it turns out that this is not an OpenSSL issue – it’s an NSS issue.

Perhaps I should have paid more attention to the prefixing of the curl error response:

* NSS error -12286

Also, when looking around for clues about CA Bundles, root certificates and the like, I came across this Mozilla wiki which states: “Fedora: nss-tools”. More hints. Also, waaay down on the curl man page, it suggests “If curl is built against the NSS SSL library …“ So there’s that.

A quick review of Fedora 8’s curl package reveals that – okay, fine – yes, it’s an NSS problem.

% yum deplist curl

Finding dependencies:
package: curl.i386 7.16.4-8.fc8
 ...
 dependency: libssl3.so
  provider: nss.i386 3.12.2.0-1.1.fc8
  provider: nss.i386 3.11.7-10.fc8

% yum update nss

No Packages marked for Update

Now, at this point I’m still chasing the wild CA Bundles goose, and this Fedora wiki describes the ‘Shared System Certificates’ Feature which promises to “Make NSS, GnuTLS, OpenSSL and Java share a default source for retrieving system certificate anchors and black list information.” However, it is unimplemented.

Fortunately, by now it’s dawning on me that this is probably not a CA Bundle issue, but rather an SSL Cipher issue. Because, you know, “SSL_ERROR_NO_CYPHER_OVERLAP”.

Rabbit Hole: NSS-Tools

So, obviously what I need here is nss-tools, right? They’re already installed as a standard package, so let’s see what they offer me.

certutil? Nope; again, this is not a CA Bundle issue.
modutil? Nope; the database is fine.

Well, I can’t find any command-line tool to help me manage SSL Ciphers. It looks like there’s just this pre-baked-in set available to me. Under “Available ciphers are:”, I see “TLS_RSA_WITH_AES_128_CBC_SHA”, otherwise known as rsa_aes_128_sha. And gnutls-cli told me:

- Version: TLS 1.1
- Key Exchange: RSA
- Cipher: AES 128 CBC
- MAC: SHA

That’s probably want I want to use! Why curl isn’t using it by default, I’m not sure. Let’s try to force the issue:

% curl -vvv --cipher 'rsa_aes_128_sha' https://api.github.com/gists/2166671

...
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*   subject: CN=*.github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US

Excellent! Finally I’ve got it licked. Why, this shouldn’t take long at all!

Rabbit Hole: .curlrc

I create myself a little ~/.curlrc which contains

And it works great from the shell. And diving a little deeper into the embed-github-gist tells me that it uses Wordpress’ preferred HTTP interaction component, WP_Http.

Yet even after I create ~nginx/.curlrc for my nginx daemon user, I’m still having issues. This post on libcurl bursts my balloon when it states “the entire .curlrc parser and all associated logic is only present in the command line tool code.” Crap.

Still. I’m clearly closing in on a solution, wouldn’t you think?

Patching WP_Http

Perhaps WP_Http has a built-in configuration mechanism to provide curl-related options? A scan through the source code in wordpress/wp-includes/class-http.php doesn’t show any generalized uses of curl_setopt, so it appears not. Since I can’t patch embed-github-gist, it seems I’ll have to patch Wordpress itself.

A look through PHP’s libcurl integration reveals that I want the “CURLOPT_SSL_CIPHER_LIST” flag. So here it, is the One Line Fix, with 3x as many comment lines as code:

--- a/wordpress/wp-includes/class-http.php +++ b/wordpress/wp-includes/class-http.php @@ -1160,6 +1160,12 @@ class WP_Http_Curl { curl_setopt( $handle, CURLOPT_RETURNTRANSFER, true ); curl_setopt( $handle, CURLOPT_SSL_VERIFYHOST, ( $ssl_verify === curl_setopt( $handle, CURLOPT_SSL_VERIFYPEER, $ssl_verify ); + + // GitHub fail + // curl -vvv https://api.github.com/gists/2166671 + // http://blog.cantremember.com/one-line-fix-wordpress-and-git + curl_setopt( $handle, CURLOPT_SSL_CIPHER_LIST, 'rsa_aes_128_sha' + curl_setopt( $handle, CURLOPT_CAINFO, $r['sslcertificates'] ); curl_setopt( $handle, CURLOPT_USERAGENT, $r['user-agent'] ); // The option doesn't work with safe mode or when open_basedir i

No generalization, no resulting pull request, no nothin’. Just the minimum I need to fix the problem, isolated as close the broken component itself as possible. Yay!

In Summary

Ultimately, I deployed the fix on the night of Sat 2014-Jan-04, after throwing back all of the red herrings that I netted along the way.

Hell no, I am not proud of how much time & effort I sank into this so-called trivial change. Granted, I didn’t start the process until after New Year’s Day, and I only had time to spelunk at night after I got home from ‘real’ work. But still … yow!
I’ll freely admit that my deduction techniques were flawed. The Rabbit Hole you’ve been in is something that you only see after coming up from missing that left turn at Albuquerque. This blog post documents the adversity, not the victory.
The OS on my AWS instance is kind of ancient: Fedora 8. Back in 2007 that was reasonable, but it has long since been end-of-lifed. Even so, my preference to stay with pre-ordained packages and not completely upgrade my server at this point in time, which left me working with ancient code. This was a choice that perhaps I should have re-considered, but it seemed a prudent choice in-the-moment.
Ultimately, this ordeal seemed to up following the typical One Line Fix pattern that Developers run into when shit breaks – a lot of research, trial & error, leading to a tiny little patch in just the right place. A ton of thanks goes out to the search-indexed of others’ legacy posts and wikis which lit the way even in my darkest hours.

Of course, in retrospect, the One Line Fix seems like the easiest little thing, doesn’t it? But as we all know, getting there is 95% of the battle. And a battle it was. A struggle was had, adversity overcome, and a victory wrested away from the jaws of defeat!

NOTE: If your screen reader is reading this, please contact me at admin@cantremember.com ... because it shouldn't. FIXME: build this dynamically based upon the maximum content in any sub-Element of this Element. I will call this my "Safari Reader Counterweight". In some of my Posts, I have huge code excerpts, etc. Safari Reader, at least in iOS, will identify the 'main Element', the one it features, based upon its content length. Sometimes those code excerpts get identified as the 'main Element', and the Post is borked in Safari Reader mode. This is a counter-weight; it gives the <main> Element additional content so that it gets featured, algorithmically. Yes, it increases the payload of every page (@see FIXME above). But not by that much. Then again, this is a guess as to how much content any given Element could contain. If it's not enough, BOOM, Safari Reader looks like crap. So, here's a great article on how to enable Safari Reader on your site. It's mostly guesswork, but those guesses helped me debug this obtuse goddamn problem. Oh, and look, you can enter and exit Reader programmatically. JavaScript can fix anything. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload.