At some point in the afternoon of Tue 2013-Dec-31 (as I remember it), one of my site health checks started firing off. WordPress was responding with an HTTP 500 and partial document body, ending with:
SSL connect error
I use the embed-github-gist plugin to display my code-snippet gists inline in my blog posts. It fetches them on the server side — thank goddess I cache my blog posts :) — and apparently at some point on Tuesday, GitHub had switched something over in its SSL termination. Until I fix this, my blog is kinda toast.
I had to add a One Line Fix to WordPress’ WP_Http component so that it would use the proper SSL Cipher. Scroll down towards the bottom of this post to see the code-snippet.
Whereas the rest of this post recounts the twists & turns I took in coming to that final solution, the process of which may or may not be of use to others. It certainly was a welcomed catharsis to write it all down.
What’s up with WordPress?
log_errors = On, and they end up in my nginx error log. I was seeing
2014/01/04 20:12:00 [error] 935#0: *95058410 FastCGI sent in stderr: "PHP Fatal error: Cannot use object of type WP_Error as array in ~/wordpress/wp-content/plugins/embed-github-gist/embed-github-gist.php on line 86" while reading upstream
Cool, that’s useful. So, I read the source code and mock up a URL to try out in curl. It works fine here on my laptop, but on my AWS instance, i get:
% curl -vvv https://api.github.com/gists/2166671?sslverify=false * About to connect() to api.github.com port 443 (#0) * Trying 220.127.116.11... connected * Connected to api.github.com (18.104.22.168) port 443 (#0) * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * NSS error -12286
sslverify=false isn’t helping. Nor does using curl with
-k/--insecure. So, what’s up with ‘NSS error -12286′? This post states that it’s SSL_ERROR_NO_CYPHER_OVERLAP.
I figured that need to get some better tooling to identify my core issue with GitHub and TLS. Via this post I track down a tool called
% yum install gnutls gnutls-utils % gnutls-cli -p 443 api.github.com - Certificate info: # The hostname in the certificate matches 'api.github.com'. # valid since: Sun Apr 29 20:00:00 EDT 2012 # expires at: Wed Jul 9 08:00:00 EDT 2014 # fingerprint: 55:D8:B2:AC:FA:96:DF:AF:85:32:1C:0F:B2:5A:96:1D # Subject's DN: C=US,ST=California,L=San Francisco,O=GitHub\, Inc.,CN=*.github.com # Issuer's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance CA-3 - Certificate info: # valid since: Wed Apr 2 08:00:00 EDT 2008 # expires at: Sat Apr 2 20:00:00 EDT 2022 # fingerprint: C6:8B:99:30:C8:57:8D:41:6F:8C:09:4E:6A:DB:0C:90 # Subject's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance CA-3 # Issuer's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV Root CA - Peer's certificate issuer is unknown - Peer's certificate is NOT trusted - Version: TLS 1.1 - Key Exchange: RSA - Cipher: AES 128 CBC - MAC: SHA
Now, it turns out that the key word here is “cypher”. But since I saw “certificate is NOT trusted”, I went down my first Rabbit Hole instead.
Rabbit Hole: New Certificate Authority
Obviously I needed to update my CA Bundle within OpenSSL, right? The bundle is installed at
/etc/pki/tls/certs/ca-bundle.crt per the curl response above. I’m guessing it’s probably quite old, since I’m still running Fedora 8 (as justified later). I’ll probably need to find an alternate source.
Looking at the directory hierarchy,
/etc/pki/tls/openssl.cnf gives me a clue that it’s an OpenSSL thing. Moreover, this post assures me that The ca-bundle.crt is in the openssl rpm. Sweet.
However, I already have the latest (0.9.8b-17) and this RPM listing doesn’t give me anything more to go on. I suspect I could build a 1.x version of OpenSSL, but that may cause interoperability issues. Crap.
I find a Fedora 19 version of the OpenSSL RPM and pull it down. This post informs me that I need to use
rpm2cpio to unpack an RPM as if it were a normal archive. Ultimately though, the resulting
ca-bundle.crt does not help. Crap.
So, I guess it’s time to re-build the CA Bundle from scratch, right? Well, let’s see. Curl’s Details on Server SSL Certificates suggests Get a better/different/newer CA cert bundle!. Rather than do it through a source download of Firefox, I went with the CA Extract approach, and downloaded a new
cacert.pem. That too didn’t help. Crap.
I check with the OpenSSL FAQ: How can I set up a bundle of commercial root CA certificates? I can generate them from Mozilla’s current master list using this convenient little script. That’ll provide me with the latest full CA Certificate List, so that’ll totally work! Pull down the file, modify the script a bit, and voila, right?
% wget http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1 % ./mkcabundle.pl > ca-bundle.crt % chown root:root ca-bundle.crt
No, of course it won’t. Because this is a Rabbit Hole. I even pulled down the offending certificate from DigiCert and tried to use it in isolation.
gnutls-cli offers a way for me to extract the public key into a PEM, but that didn’t work, so I viewed the GitHub SSL certificate info in Google Chrome, then did some manual typing (since URL cut-and-paste was prevented for some absurd reason):
% wget --no-check-certificate https://www.digicert.com/CACerts/DigiCertHighAssuranceCA-3.crt % openssl x509 -inform DER -outform PEM -in DigiCertHighAssuranceCA-3.crt -out DigiCertHighAssuranceCA-3.pem
Somehow, no matter what I do with the CA Bundle, it’s not helping. Also, something seems fishy, because the configuration in
/etc/pki/tls/openssl.cnf doesn’t seem to match reality.
So it turns out that this is not an OpenSSL issue — it’s an NSS issue.
Perhaps I should have paid more attention to the prefixing of the curl error response:
* NSS error -12286
Also, when looking around for clues about CA Bundles, root certificates and the like, I came across this Mozilla wiki which states: Fedora: nss-tools. More hints. Also, waaay down on the curl man page, it suggests If curl is built against the NSS SSL library … So there’s that.
A quick review of Fedora 8′s curl package reveals that — okay, fine — yes, it’s an NSS problem..
% yum deplist curl Finding dependencies: package: curl.i386 7.16.4-8.fc8 ... dependency: libssl3.so provider: nss.i386 22.214.171.124-1.1.fc8 provider: nss.i386 3.11.7-10.fc8 % yum update nss No Packages marked for Update
Now, at this point I’m still chasing the wild CA Bundles goose, and this Fedora wiki describes the ‘Shared System Certificates’ Feature which promises to Make NSS, GnuTLS, OpenSSL and Java share a default source for retrieving system certificate anchors and black list information. However, it is unimplemented.
Fortunately, by now it’s dawning on me that this is probably not a CA Bundle issue, but rather an SSL Cipher issue. Because, you know, “SSL_ERROR_NO_CYPHER_OVERLAP”.
Rabbit Hole: NSS-Tools
So, obviously what I need here is
nss-tools, right? They’re already installed as a standard package, so let’s see what they offer me.
Well, I can’t find any command-line tool to help me manage SSL Ciphers. It looks like there’s just this pre-baked-in set available to me. Under Available ciphers are:, I see “TLS_RSA_WITH_AES_128_CBC_SHA”, otherwise known as
gnutls-cli told me:
- Version: TLS 1.1 - Key Exchange: RSA - Cipher: AES 128 CBC - MAC: SHA
That’s probably want I want to use! Why curl isn’t using it by default, I’m not sure. Let’s try to force the issue:
curl -vvv --cipher 'rsa_aes_128_sha' https://api.github.com/gists/2166671 ... * SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA * Server certificate: * subject: CN=*.github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US
Excellent! Finally I’ve got it licked. Why, this shouldn’t take long at all!
Rabbit Hole: .curlrc
I create myself a little
~/.curlrc which contains
And it works great from the shell. And diving a little deeper into the
embed-github-gist tells me that it uses WordPress’ preferred HTTP interaction component, WP_Http.
Yet even after I create
~nginx/.curlrc for my nginx daemon user, I’m still having issues. This post on libcurl bursts my balloon when it states the entire .curlrc parser and all associated logic is only present in the command line tool code. Crap.
Still. I’m clearly closing in on a solution, wouldn’t you think?
WP_Http has a built-in configuration mechanism to provide curl-related options? A scan through the source code in
wordpress/wp-includes/class-http.php doesn’t show any generalized uses of
curl_setopt, so it appears not. Since I can’t patch
embed-github-gist, it seems I’ll have to patch WordPress itself.
A look through PHP’s libcurl integration reveals that I want the “CURLOPT_SSL_CIPHER_LIST” flag. So here it, is the One Line Fix, with 3x as many comment lines as code:
No generalization, no resulting pull request, no nothin’. Just the minimum I need to fix the problem, isolated as close the broken component itself as possible. Yay!
Ultimately, I deployed the fix on the night of Sat 2014-Jan-04, after throwing back all of the red herrings that I netted along the way.
- Hell no, I am not proud of how much time & effort I sank into this so-called trivial change. Granted, I didn’t start the process until after New Year’s Day, and I only had time to spelunk at night after I got home from ‘real’ work. But still … yow!
- I’ll freely admit that my deduction techniques were flawed. The Rabbit Hole you’ve been in is something that you only see after coming up from missing that left turn at Albuquerque. This blog post documents the adversity, not the victory.
- The OS on my AWS instance is kind of ancient: Fedora 8. Back in 2007 that was reasonable, but it has long since been end-of-lifed. Even so, my preference to stay with pre-ordained packages and not completely upgrade my server at this point in time, which left me working with ancient code. This was a choice that perhaps I should have re-considered, but it seemed a prudent choice in-the-moment.
- Ultimately, this ordeal seemed to up following the typical ‘One Line Fix’ pattern that Developers run into when shit breaks — a lot of research, trial & error, leading to a tiny little patch in just the right place. A ton of thanks goes out to the search-indexed of others’ legacy posts and wikis which lit the way even in my darkest hours.
Of course, in retrospect, the One Line Fix seems like the easiest little thing, doesn’t it? But as we all know, getting there is 95% of the battle. And a battle it was. A struggle was had, adversity overcome, and a victory wrested away from the jaws of defeat!