i’m working for a new company now, and they’re rocking it for RoR apps on the iPhone. sounds like a good place to be. one of the many reasons why this position works for me is because these guys are all about GTD and getting it out there. lean ‘n’ mean

whereas i’ve become very used to a holistic detail-orented, wisened test-backed process. great for Enterprise, but not so much for the reckless streets of Startup 3.0 . so i’m in a learning process. i’ve turned around some good stuff quickly, and it’s very satisfying

but i’ve screwed the pooch twice since i’ve been there. it’s totally a judgement call thing — i’m shooting too fast from the hip, and don’t feel like i really grasp the balance here …

first project i worked on was related to account management. they wanted a quick turn-around, i gave it a shot, had the whole thing backed with solid testing, and ready for on-time deployment with a smile. and in trying to keep track of all the new system permutations — i’d been there 2 weeks or so — i forgot one basic thing, and forgot to test for another. a nice little Perfect Storm. one emergency 1am database rollback later, we had a load of pissed customers and a helluva lot of explaining to do

so, then this past week, i went in to fix a minor rounding issue bug. those can be touchy. the right way to do it is with BigDecimal. yep, i’ve done that in Java too with BigDecimal. overall, it’s somewhat ponderous, detail-oriented, and can easily be polluted with Floats and the like. so i’d taken a shortcut, realizing that the low-level C impl was doing String conversion without the rounding issue. so i took the low-hanging fruit:

total.to_s.to_i

awesome !!1!. well, that is until you get into the 100-of-trillions area, otherwise shown as 1.0e+14. guess what happens when you parse that into a Fixnum? no database rollback this time, but Da Boss had to spend days sorting out the visceral impact of ridiculous sums of bogus exploit money pouring into our RPG

security, privacy and account management. payment calculations. not the sort of things to take shortcuts on. yet, if you’re embracing a culture that wants it done quickly and with minimum impact, it’s a risk you might be willing to take. it’s not like i didn’t have test scripts … i just forgot to head into scientific notation territory. just like i forgot to check for the implication of null password acceptance ( long story there, special account cases, etc. )

i’m putting these things up here for my fellow developers to laugh at.   “I mean, c’mon. All that’s totally obvious stuff.”   “I’d never miss that, that’s sophmore shit.”   good, get it out of your system, laughing boy

but believe me, when you’re on the other end of it, and had been in the middle of it and all full of all the other things that you needed to keep track of at that time, heh, well, that’s when you’ll really need to keep yerself laughing :)

Fortunately, I haven’t lost any followers (yet). But this was a perfect case of how mixing business with pleasure can have unintended consequences.

What Have I Learned

Or rather, what have I re-learned …

Soft-disable features in Production at Launch Time

 
My Twitter engines are built with both an :enable_tweet and :enable_greeting config setting. In the git repo, they’re both true. When I did my local testing, I’d disabled them correctly. When I launched in Production, I neglected to make the quick-and-dirty changes; after all, everything worked great. And once started, my scripts correctly responded to the no-initial-state condition, and greeted everybody.

Launch preparation is critical, even for little projects. The start-up mentality is to move fast and lean, but there’s such as thing as too fast, and probably as too lean too. Gradual uptake migration is a wise strategy even for the ‘little things’.

Mock and Integration Testing Only Gets You So Far

 
I used rspec to mock out the full capabilities of the engine. Found some real-world issues, resolved them. I also wrote some core integration tests, ran them locally. Immediate failures. I had mocked documented features that didn’t actually exist. Fixed, re-mocked, re-tested, fixed again, etc .

Another great reminder that you can only mock something you trust, and how can you trust something you haven’t actually run under integration conditions to start with! Re-tested integration, and everything passed with flying colors. Sure, the features worked great now! And when I launched them, they did exactly what I asked.

So, as if we haven’t heard it enough times, be careful what you ask for!

All of this is familiar to anyone who has made a mistake in the software industry. It’s not like I haven’t successfully executed dozens of critical launches in the past, and most with virtually no issues at all. But what’s interesting is what happens when these mistakes happen in a public forum, and whom you expose them to — say, your friends :)

And who can say when two ounces of caution is more deserving than one … without the benefit of hindsight.

Just ask anyone who has a stringent backup policy how much time & effort they invest to avoid an event that may never actually happen. That stringency usually comes from that one unforgettable experience, and from there is born an extra layer of caution, and an additional time-sink (eg. mock & integration testing)

Heh. KISS. So, what exactly is simple? DRY. Isn’t that supposed to be a time-saver? Well, it depends on what you’re not trying to repeat. Strange how these cuddly and liberating acronyms can have more than one interpretation.

Experience taints everything.