Archive for the ‘Uncategorized’ Category

one line fix: WordPress and GitHub’s SSL Cipher

Sunday, January 5th, 2014

At some point in the afternoon of Tue 2013-Dec-31 (as I remember it), one of my site health checks started firing off. WordPress was responding with an HTTP 500 and partial document body, ending with:

SSL connect error

I use the embed-github-gist plugin to display my code-snippet gists inline in my blog posts. It fetches them on the server side — thank goddess I cache my blog posts :) — and apparently at some point on Tuesday, GitHub had switched something over in its SSL termination. Until I fix this, my blog is kinda toast.

TL;DR

I had to add a One Line Fix to WordPress’ WP_Http component so that it would use the proper SSL Cipher. Scroll down towards the bottom of this post to see the code-snippet.

Whereas the rest of this post recounts the twists & turns I took in coming to that final solution, the process of which may or may not be of use to others. It certainly was a welcomed catharsis to write it all down.

What’s up with WordPress?

My /etc/php.ini has log_errors = On, and they end up in my nginx error log. I was seeing

2014/01/04 20:12:00 [error] 935#0: *95058410 FastCGI sent in stderr: "PHP Fatal error:  Cannot use object of type WP_Error as array in ~/wordpress/wp-content/plugins/embed-github-gist/embed-github-gist.php on line 86" while reading upstream

Cool, that’s useful. So, I read the source code and mock up a URL to try out in curl. It works fine here on my laptop, but on my AWS instance, i get:

% curl -vvv https://api.github.com/gists/2166671?sslverify=false

* About to connect() to api.github.com port 443 (#0)
*   Trying 192.30.252.138... connected
* Connected to api.github.com (192.30.252.138) port 443 (#0)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -12286

Well, that sslverify=false isn’t helping. Nor does using curl with -k/--insecure. So, what’s up with ‘NSS error -12286′? This post states that it’s SSL_ERROR_NO_CYPHER_OVERLAP.

I figured that need to get some better tooling to identify my core issue with GitHub and TLS. Via this post I track down a tool called gnutls-cli:

% yum install gnutls gnutls-utils
% gnutls-cli -p 443 api.github.com

- Certificate[0] info:
# The hostname in the certificate matches 'api.github.com'.
# valid since: Sun Apr 29 20:00:00 EDT 2012
# expires at: Wed Jul  9 08:00:00 EDT 2014
# fingerprint: 55:D8:B2:AC:FA:96:DF:AF:85:32:1C:0F:B2:5A:96:1D
# Subject's DN: C=US,ST=California,L=San Francisco,O=GitHub\, Inc.,CN=*.github.com
# Issuer's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance CA-3

- Certificate[1] info:
# valid since: Wed Apr  2 08:00:00 EDT 2008
# expires at: Sat Apr  2 20:00:00 EDT 2022
# fingerprint: C6:8B:99:30:C8:57:8D:41:6F:8C:09:4E:6A:DB:0C:90
# Subject's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance CA-3
# Issuer's DN: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV Root CA

- Peer's certificate issuer is unknown
- Peer's certificate is NOT trusted

- Version: TLS 1.1
- Key Exchange: RSA
- Cipher: AES 128 CBC
- MAC: SHA

Now, it turns out that the key word here is “cypher”. But since I saw “certificate is NOT trusted”, I went down my first Rabbit Hole instead.

Rabbit Hole: New Certificate Authority

Obviously I needed to update my CA Bundle within OpenSSL, right? The bundle is installed at /etc/pki/tls/certs/ca-bundle.crt per the curl response above. I’m guessing it’s probably quite old, since I’m still running Fedora 8 (as justified later). I’ll probably need to find an alternate source.

Looking at the directory hierarchy, /etc/pki/tls/openssl.cnf gives me a clue that it’s an OpenSSL thing. Moreover, this post assures me that The ca-bundle.crt is in the openssl rpm. Sweet.

However, I already have the latest (0.9.8b-17) and this RPM listing doesn’t give me anything more to go on. I suspect I could build a 1.x version of OpenSSL, but that may cause interoperability issues. Crap.

I find a Fedora 19 version of the OpenSSL RPM and pull it down. This post informs me that I need to use rpm2cpio to unpack an RPM as if it were a normal archive. Ultimately though, the resulting ca-bundle.crt does not help. Crap.

So, I guess it’s time to re-build the CA Bundle from scratch, right? Well, let’s see. Curl’s Details on Server SSL Certificates suggests Get a better/different/newer CA cert bundle!. Rather than do it through a source download of Firefox, I went with the CA Extract approach, and downloaded a new cacert.pem. That too didn’t help. Crap.

I check with the OpenSSL FAQ: How can I set up a bundle of commercial root CA certificates? I can generate them from Mozilla’s current master list using this convenient little script. That’ll provide me with the latest full CA Certificate List, so that’ll totally work! Pull down the file, modify the script a bit, and voila, right?

% wget http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1
% ./mkcabundle.pl > ca-bundle.crt
% chown root:root ca-bundle.crt

No, of course it won’t. Because this is a Rabbit Hole. I even pulled down the offending certificate from DigiCert and tried to use it in isolation. gnutls-cli offers a way for me to extract the public key into a PEM, but that didn’t work, so I viewed the GitHub SSL certificate info in Google Chrome, then did some manual typing (since URL cut-and-paste was prevented for some absurd reason):

% wget --no-check-certificate https://www.digicert.com/CACerts/DigiCertHighAssuranceCA-3.crt
% openssl x509 -inform DER -outform PEM -in DigiCertHighAssuranceCA-3.crt -out DigiCertHighAssuranceCA-3.pem

Somehow, no matter what I do with the CA Bundle, it’s not helping. Also, something seems fishy, because the configuration in /etc/pki/tls/openssl.cnf doesn’t seem to match reality.

NSS

So it turns out that this is not an OpenSSL issue — it’s an NSS issue.

Perhaps I should have paid more attention to the prefixing of the curl error response:

* NSS error -12286

Also, when looking around for clues about CA Bundles, root certificates and the like, I came across this Mozilla wiki which states: Fedora: nss-tools. More hints. Also, waaay down on the curl man page, it suggests If curl is built against the NSS SSL library … So there’s that.

A quick review of Fedora 8′s curl package reveals that — okay, fine — yes, it’s an NSS problem..

% yum deplist curl

Finding dependencies:
package: curl.i386 7.16.4-8.fc8
 ...
 dependency: libssl3.so
  provider: nss.i386 3.12.2.0-1.1.fc8
  provider: nss.i386 3.11.7-10.fc8

% yum update nss

No Packages marked for Update

Now, at this point I’m still chasing the wild CA Bundles goose, and this Fedora wiki describes the ‘Shared System Certificates’ Feature which promises to Make NSS, GnuTLS, OpenSSL and Java share a default source for retrieving system certificate anchors and black list information. However, it is unimplemented.

Fortunately, by now it’s dawning on me that this is probably not a CA Bundle issue, but rather an SSL Cipher issue. Because, you know, “SSL_ERROR_NO_CYPHER_OVERLAP”.

Rabbit Hole: NSS-Tools

So, obviously what I need here is nss-tools, right? They’re already installed as a standard package, so let’s see what they offer me.

  • certutil? Nope; again, this is not a CA Bundle issue.
  • modutil? Nope; the database is fine.

Well, I can’t find any command-line tool to help me manage SSL Ciphers. It looks like there’s just this pre-baked-in set available to me. Under Available ciphers are:, I see “TLS_RSA_WITH_AES_128_CBC_SHA”, otherwise known as rsa_aes_128_sha. And gnutls-cli told me:

- Version: TLS 1.1
- Key Exchange: RSA
- Cipher: AES 128 CBC
- MAC: SHA

That’s probably want I want to use! Why curl isn’t using it by default, I’m not sure. Let’s try to force the issue:

curl -vvv --cipher 'rsa_aes_128_sha' https://api.github.com/gists/2166671

...
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
* 	subject: CN=*.github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US

Excellent! Finally I’ve got it licked. Why, this shouldn’t take long at all!

Rabbit Hole: .curlrc

I create myself a little ~/.curlrc which contains

# ~/.curlrc
ciphers=rsa_aes_128_sha

And it works great from the shell. And diving a little deeper into the embed-github-gist tells me that it uses WordPress’ preferred HTTP interaction component, WP_Http.

Yet even after I create ~nginx/.curlrc for my nginx daemon user, I’m still having issues. This post on libcurl bursts my balloon when it states the entire .curlrc parser and all associated logic is only present in the command line tool code. Crap.

Still. I’m clearly closing in on a solution, wouldn’t you think?

Patching WP_Http

Perhaps WP_Http has a built-in configuration mechanism to provide curl-related options? A scan through the source code in wordpress/wp-includes/class-http.php doesn’t show any generalized uses of curl_setopt, so it appears not. Since I can’t patch embed-github-gist, it seems I’ll have to patch WordPress itself.

A look through PHP’s libcurl integration reveals that I want the “CURLOPT_SSL_CIPHER_LIST” flag. So here it, is the One Line Fix, with 3x as many comment lines as code:

No generalization, no resulting pull request, no nothin’. Just the minimum I need to fix the problem, isolated as close the broken component itself as possible. Yay!

In Conclusion

Ultimately, I deployed the fix on the night of Sat 2014-Jan-04, after throwing back all of the red herrings that I netted along the way.

  • Hell no, I am not proud of how much time & effort I sank into this so-called trivial change. Granted, I didn’t start the process until after New Year’s Day, and I only had time to spelunk at night after I got home from ‘real’ work. But still … yow!
  • I’ll freely admit that my deduction techniques were flawed. The Rabbit Hole you’ve been in is something that you only see after coming up from missing that left turn at Albuquerque. This blog post documents the adversity, not the victory.
  • The OS on my AWS instance is kind of ancient: Fedora 8. Back in 2007 that was reasonable, but it has long since been end-of-lifed. Even so, my preference to stay with pre-ordained packages and not completely upgrade my server at this point in time, which left me working with ancient code. This was a choice that perhaps I should have re-considered, but it seemed a prudent choice in-the-moment.
  • Ultimately, this ordeal seemed to up following the typical ‘One Line Fix’ pattern that Developers run into when shit breaks — a lot of research, trial & error, leading to a tiny little patch in just the right place. A ton of thanks goes out to the search-indexed of others’ legacy posts and wikis which lit the way even in my darkest hours.

Of course, in retrospect, the One Line Fix seems like the easiest little thing, doesn’t it? But as we all know, getting there is 95% of the battle. And a battle it was. A struggle was had, adversity overcome, and a victory wrested away from the jaws of defeat!

weird sync issue = iPhone Notes will not die

Saturday, November 6th, 2010

hola, amigos. s’up? i know it’s been a long time since I rapped at ya, but I’ve had a lot of stuff goin’ down.

i’ve had an issue with Notes on my iPhone. sure, i could use a content service like Evernote — which i did — but an even simpler tact for us Mac folks is to keep a couple long-lived Notes around and just edit them. lists n’ shit. you know the deal

anyway, so every once in a while Mail.app will have a sync conflict, and every once in a while i’ll click [Sync Later], always by mistake. now what i noticed is that there was a one-to-one (or nearly) relationship between the times i made that mistake, and Notes that would appear on my iPhone … that i couldn’t delete

  • i’d delete them, then they’d reappear after sync
  • there was no trace of them in Mail.app. i explicitly flushed my Trash a couple times, but to no effect
  • you’d figure that the the mobile Notes would have singular authority & ownership, but nope. i’d delete them, they’d come back, and they kept getting older and more annoying
  • the friendly Genius Bar staff had no other advice to give besides doing an explicit iTunes re-sync via Advanced | Replace information on this iPhone | [x] Notes, but to no effect

now, i know that this isn’t a particularly technical post. however since i couldn’t find any reference to anyone else having this problem — at least based on the keywords i was using — i figured i’d drop a few of them and describe the solution:

  • view the Note, select all, and delete
  • the mobile app auto-deletes the empty Note
  • voila!. it’s gone for good

there. now you know what i know :)

Prowl = Growl iPhone push notifications

Wednesday, July 8th, 2009

i just installed Prowl, and it’s a handy little beast. it allows me to route my locally-issued Growl notifications to their Apple Push Notification server, which then routes them to the Prowl iPhone client. it all makes a simple, sweet, and extremely versatile combination

the hierarchy of preferences is somewhat distributed however …

  • the Prowl site will explain all the installation stuff. the first level of preferences is configuring the Prowl plug-in to be your default Growl notification route. it’s good that the Mac plug-in allows you to choose a pass-through notification method so that you will still see the events locally
  • then there’s the iPhone client preferences, under Settings. they explain in the FAQ that you need to change your settings, then start up the Prowl client to commit them to the server. confusing, because your changes don’t take effect until that commit, so it’s just a matter of knowing. also, the Settings only allow you to disable Sound while the app is running (which is a nice feature)
  • it took me a further read to figure out that i needed to disable the background push Sound & Vibrate feature through the general Notification Settings. you can tweak the Prowl app so that it’s seen and not heard

i’m a big fan of TwitterFox because it’s equally simple & straightforward. i chose Scalaris as my Mac Twitter client of choice for the same reason — i don’t needs one of the feature-rich hefty ones

eventually i’ll be updating my AWS health checks to use the prowl Ruby gem. very thoughtful!

thanks to @laughingsquid for the tip on Prowl. i’m looking forward to a further geometrical increase in information overload!