Well, that sure made for an interesting morning!

And I’d sent out the DMs out at 1a PST, which means that anyone who (a) uses Twitter’s SMS capabilities, and (b) has a text message notification sound would have been rudely interrupted in the middle of the night.

Fortunately, I haven’t lost any followers (yet). But this was a perfect case of how mixing business with pleasure can have unintended consequences.

What Have I Learned

Or rather, what have I re-learned

Soft-disable features in Production at Launch Time

My Twitter engines are built with both an :enable_tweet and :enable_greeting config setting. In the git repo, they’re both true. When I did my local testing, I’d disabled them correctly. However, when I launched in Production, I neglected to make those quick-and-dirty changes; after all, everything was working swell! And once they started up, my scripts correctly responded to the no-initial-state condition, and they promptly greeted everybody.

Launch preparation is critical, even for little projects. The start-up mentality is to move fast & stay lean, but there’s such as thing as too fast, and probably there’s a too lean as well. Gradual uptake migration is a wise strategy even for the ‘little things’.

Mock and Integration Testing Only Gets You So Far

I used rspec to mock out the full capabilities of the engine. I found some real-world issues and resolved them. I also wrote some core integration tests, ran them locally. There were immediate failures. I had mocked documented features that didn’t actually exist. I fixed ‘em, re-mocked, re-tested, fixed ‘em again, etc.

It’s another great reminder that you can only mock something you trust, and how can you trust something you haven’t actually run under integration conditions to start with? I re-tested under integration conditions, and everything passed with flying colors. Sure, the features worked great now! And when I launched them, they did exactly what I asked.

So, as if we haven’t heard it enough times, be careful what you ask for!

No Surprises There

All of this is familiar to anyone who has made a mistake in the software industry. It’s not like I haven’t successfully executed dozens of critical launches in the past, and most with virtually no issues at all. But what’s interesting is what happens when these mistakes happen in a public forum, and whom you expose them to – say, your friends :open_mouth:

And who can say when two ounces of caution is more deserving than one … without the benefit of hindsight.

Just ask anyone who has a stringent backup policy how much time & effort they invest to avoid an event that may never actually happen. That stringency usually comes from that one unforgettable experience, and from there is born an extra layer of caution, and an additional time-sink (eg. mock & integration testing).

Heh. Our Principles of Development. There’s KISS. So, what exactly is “simple”? There’s DRY. It’s supposed to be a time-saver, right? I guess it depends on what you’re not trying to repeat. It would seem that these cuddly and liberating acronyms can still be open to interpretation …

Experience taints everything.