so, i’ve been doing plenty, i’m just not chatty about it. i built a Ruby migration framework using bundler, pry, spreadsheet, sequel and mp3info to build a JSON document version of my SEB Broadcast database. next up is some node.js to serve it up, then some RequireJS, mustache (?) & jQuery goodness to spiff up the SEB site

but in the meanwhile, i wrote this little gem at work:

// returns a function that will invoke the callback 'delay' ms after it is last called
//   eg. invoke the callback 500ms after the last time a key is pressed
// based on http://stackoverflow.com/questions/2219924/idiomatic-jquery-delayed-event-only-after-a-short-pause-in-typing-e-g-timew
//   but fixes the following:
//   (a) returns a unique timer scope on every call
//   (b) propagates context and arguments from the last call to the returned closure
//   and adds .cancel() and .fire() for additional callback control
// then of course, you could use underscore's _.debounce
//   it doesn't have the .cancel and .fire, but you may not care :)
function delayedCallback(callback, delay) {
    return (function(callback, delay) {                          // (2) receives the values in scope
        var timer = 0;                                           // (3a) a scoped timer
        var context, args;                                       // (3b) scoped copies from the last invocation of the returned closure
        var cb = function() { callback.apply(context, args); }   // (3c) called with proper context + arguments
        var dcb = function() {                                   // (4) this closure is what gets returned from .delayedCallback
            context = this;
            args = arguments;
            window.clearTimeout(timer);
            timer = window.setTimeout(cb, delay);                // (5) only fires after this many ms of not re-invoking
        };
        dcb.cancel = function() { window.clearTimeout(timer); }; // (6a) you can cancel the delayed firing
        dcb.fire  = function() { this.cancel(); cb(); };         // (6b) or force it to fire immediately
        return dcb;
    })(callback, delay);                                         // (1) capture these values in scope
}

yes, i know. so it turns out that i didn’t know about underscore’s _.debounce() when i wrote it. eh. so much for DRY :)

still — i’m glad i thought it through. to me, this implementation captures the most powerful aspects of ECMAScript itself:

• scope-capturing closures
• specifiable function context
• freestyle properties on Object instances
• single-threading (look ma, no synchronize { ... } !)

anyway. bla dee blah. this post also gave me the incentive to start embedding gists in my blog. nice helper widget, dflydev !

peace out

i’ve had an issue with Notes on my iPhone. sure, i could use a content service like Evernote — which i did — but an even simpler tact for us Mac folks is to keep a couple long-lived Notes around and just edit them. lists n’ shit. you know the deal

anyway, so every once in a while Mail.app will have a sync conflict, and every once in a while i’ll click [Sync Later], always by mistake. now what i noticed is that there was a one-to-one (or nearly) relationship between the times i made that mistake, and Notes that would appear on my iPhone … that i couldn’t delete

• i’d delete them, then they’d reappear after sync
• there was no trace of them in Mail.app. i explicitly flushed my Trash a couple times, but to no effect
• you’d figure that the the mobile Notes would have singular authority & ownership, but nope. i’d delete them, they’d come back, and they kept getting older and more annoying
• the friendly Genius Bar staff had no other advice to give besides doing an explicit iTunes re-sync via Advanced | Replace information on this iPhone | [x] Notes, but to no effect

now, i know that this isn’t a particularly technical post. however since i couldn’t find any reference to anyone else having this problem — at least based on the keywords i was using — i figured i’d drop a few of them and describe the solution:

• view the Note, select all, and delete
• the mobile app auto-deletes the empty Note
• voila!. it’s gone for good

there. now you know what i know :)

Here’s all the bumps I ran into, pretty much in reverse order as I tried (unsuccessfully) to do the minimum amount of work possible :) I had to go pretty much this entire process twice — once on my work Macbook Pro, once on my identical home verison — so I figured I might as well document all of this crap down in the hope that it may reduce the shock and awe of future migrators. Of course you may run into a mess of fun issues not described here … And pardon me if the instructions aren’t perfect, because I’ve tried to boil a lot of this down to the it-finally-worked steps extracted from the frenzied back-and-forth of my real-time upgrade experience :)

Backups

Yes, this is just common-sense developer stuff (as is a number of the other obvious things that I call out in this post).

You’ll probably want to do a full mysqldump before upgrading. You can dump your port install and gem list --local listings up front as well, or wait ’til you get to those respective steps below.

MacPorts

If you also chose the MacPorts library system, you’ll need to re-install it from scratch. You’ll need X11 from the Snow Leopard from the OS/X install disks, and download the latest version of Xcode. Follow the migration steps as outlined on their Wiki; it does the trick.

Save off your port install list as a reference. Now, your MacPorts install will be completely toast, so that command won’t work until you re-install. No problem though — all of your packages will still be listed even after you upgrade.

The port clean step in the Wiki will crap out in the http* range, but that’s fine … you can probably skip that step anyway. Re-install your core packages and you’re good to go. I suggest installing readline if you haven’t, because it’s very useful in irb or any Ruby console.

MySQL

It was not necesary for me to build MySQL from source. Instead, I just installed the x86_64 version of MySQL — the latest was mysql-5.1.42-osx10.6-x86_64.dmg af the time of this writing.

If this is a 5.x verison upgrade for you as well, the install will just re-symlink /usr/local/mysql, so your old data will still be in your previous install dir.

I didn’t make mysqldumps before I did the upgrade (handpalm) so I had to copy over my data/ files raw and hope that the new version would make friends with them. Initially I had problems with InnoDB. It wasn’t showing up under show engines;, and when I tried to manually install the plug-in — per this bug report, which explains the whole thing — it would fail on the ‘Plugin initialization function’. Turns out you need to do two things when you bring over raw data:

• Whack your /var/log/mysql/*binary* / binary log files in order to get past mysqld startup errors.
• Whack your ib_logfile* files too. Once you do that, there’s a good chance MySQL will regen them in recovery mode. Me, I had no choice (except rolling back with Time Machine). Miracle of miracles … it works!

Don’t try this at home kids. Make your backups. Note: here’s the correct link to the manual page on InnoDB tablespace fun.

x86_64 ARCHFLAGS

Snow Leopard is a lot more native 64-bit than previous OS/X versions, and when you do your manual builds & makes, you may want to set the following environment variable:

export ARCHFLAGS="-Os -arch x86_64 -fno-common"

You’ll see a set of similar (though mixed) recommendations in the blogs I reference below; this particular flagset worked for me.

Ruby

I built Ruby 1.9 at work, and 1.8.7 on my personal machine. Either path is fine, just pick up the latest source of your choosing. Chris Cruft’s blog post goes into some of the details I’m describing here as well. Basically, the README boils down to:

autoconf
./configure --with-readline-dir=/usr/local
make clean && make
make install

Though there’s no reason in the world that you’d want to — it has been superceded — do not install ruby 1.9.1p243. If you do, you’ll never get the mysql gem to work. Or wait, or was it mongrel? Well, it was one or the other … just trust me, it’s bad.

Gem

I re-built gem from source from scratch as well, just to be sure. Save off your gem env and gem list --local as a reference. And before you start installing gems, you’ll also probably want to make sure you’re fully up-to-date with gem update --system, though that’s probably redundant.

Uninstall and re-install all of your gems; if some won’t uninstall even though they’re listed, it may be an install path issue. Use gem list -d GEMNAME to find where your gem was installed, and then use gem uninstall -i DIRNAME GEMNAME to finish the job.

With the ARCHFLAGS in place, the vast majority of the native source gem builds will go smoothly, but there are some notable exceptions …

The mysql Gem

Uh huh, this is the one gem that gets me every time. And again, you won’t have needed to have built MySQL from source.

For starters, you may way want to glance over this very useful iCoreTech blog post to see if it works for you. But if you run into a lot of issues like I did, you may need to do it in two steps:

Fetch and Install the Gem

At the time of this writing, either mysql gem version 2.7 or 2.8.1 will do the trick.

gem fetch mysql --version VERSION
gem install mysql -- --with-mysql-dir=/usr/local --with-mysql-config=/usr/local/mysql/bin/mysql_config
gem unpack mysql

Sadly, it may fail, either during the build or when you try to test it. I was able to successfuly run the (included?) test.rb at my workplace, but as simple as that sounds, I swear I don’t remember how I did it ! The second time, at home, I only found the problems retroactively when I tried to get my Rails projects to boot. If you do find and run the test.rb, you’ll need to make sure that the standard MySQL test database exists.

Both times, one of the big blockers that I — and many other people — ran into was:

NameError: uninitialized constant MysqlCompat::MysqlRes

If so, try this:

Manually Re-build the Binary

Go into ext/mysql_api, make sure your ARCHFLAGS are exported as described above, and …

ruby extconf.rb --with-mysql-config=/usr/local/mysql/bin/mysql_config
make clean && make
make install

Hopefully your newly built & installed binaries will resolve the issue.

Mongrel

It took me a little effort to build mongrel on Ruby 1.9 with the x86_64 architecture. My memory is a little hazy — since my 1.8.7 build at home worked perfectly through standard gem install — but buried deep in this Is It Ruby contributory blog post are probably all the answers you’ll need.

Under Ruby 1.9, I did had to modify the source, which (to paraphrase) involved some global code replacements:

• RSTRING(foo)->len with RSTRING_LEN(foo)
• RSTRING(foo)->ptr with RSTRING_PTR(foo)
• change the one-line case ... when statements from Java-esque : delimiters to then‘s.

And then the re-build:

ruby extconf.rb install mongrel
make
make install
cd ../..
ruby setup.rb
gem build mongrel.gemspec
gem install mongrel.gem

Conclusion

And that’s as far as I had to go. Whew! I certainly hope that my post has been of some assistance to you (and with a minimum of unintended mis-direction). Of course, I learned everything the I reiterated above by searching the ‘net and plugging away. And there’s plenty of other folks who’ve gone down this insane path as well. Good luck, brave soul!

Base-line Aesthetics

The first step in bringing a Wookie to life was to establish a basic phoenetic dialect. So I came up with a set of candidates ‘words’ along the lines of; auhrn, rghrn, gahrn, hraur, urau, ehruh, and nrauh. Hey, they sounded good. There’s about 60 of them total, comprised of the letters A E G H N R and U. After having watched The Empire Strikes Back, where Chewbacca seems to get most of his good lines, I expanded into a few W words as well (wahr seems to work particularly well). Many folks postulate that he was capable of speaking Os and Ks, yet I myself do not subscribe to that opinion.

I then used MadderLib to construct simple word generators which allowed the phoenemes to stretch appropriately for different word lengths. Long vowels, double Rs and Hs, whatever looked good. And with a little compositing logic, I came up with some sentence patterns that were quite fun to read out loud:

rauhr nruuuhh raghr uhr rghrrnnauhrnaauuuuuurhhh
nrauuh euu gauuhrr ruhr
urhn ehrraaah rhhreuhraahhrrrrnn gaurrh

Those are rather plain-looking though. In order to make the Wookie’s statements seem more like txt argot, some variety was needed. Punctuation was obvious, both terminating and delimiting (commas, semicolons). Plus there’s the proper use of txt idioms, the LOLs and WTFs that are so popular with the youngsters these days (AFAIK). Throw in a nice little collection of emoticons, and behold; the Wookie has charm:

LOLZ! uhrrn euueuuhhaur, eruh hraur nrauururhnehrrraaah auhrn harn aurreruhaaarruuuhh rraaghrrr ^_^
hraur rghneruh nuh waahhr???
rhhhnnghn uhrrrnn ehrah. euu urau ehuuurrr urrn aurheuuuh haarn uhrrrn k?… haauurrr nruharuhuhhrrh hruaaaauuh ghrn rghn nrruuhh

Well, yeah, they still look sorta flat. Real people quote and capitalize portions of what they type, and there are other non-verbal components to the average sentence. So, the Wookie was taught to inject numbers, times, abbreviations, and even Star Wars calendar years into his sentences:

gahrn rghhr ghrrnehur rghhrrn hruauh raghrehurauuuuuuhrn gahrn?? hrraaauu: hrrraaauu rauhr ;) _ehruuh_ ‘Rahr Ehrrraaaah’
raauh hrau Ehrahurrrrh *rauhr* gruh
auhrneuuhr. harnuhrrnuhrrr uhr – raauhneuuu 1:30 gahrn raauuughrrr *nuuh* aahhrruuuunn uhr. wuurh harn rhr?

Once the Wookie was at this point, he could talk for quite some time and produce diverse aesthetic results. Reading them out loud is a hoot! Thus was born the first Personality Engine bot (the Wookie is comprised of four of them). But he still wasn’t really tweeting until he could follow some of the core Twitter memes.

Twitter Memes

Hash tags were the first obvious choice, since they were easy to fake. To this day, the Wookie can simply prepend a # to any word or composite that he speaks. But to make this feature really zing, I added support for Twitter’s trending searches. This allowed him to use real-world releveant tags, injecting them into his sentences, or appending them to his tweet (as is common convention). It turns out that one of the joys of a nonsense grammar is that anything which isn’t nonsense magically becomes the ‘meaning’ of the sentence:

Harn ehureruheuuu & ahrn rrghhhn! AAAHHRRNNAUHRN EHUUR! #itsnotgonnawork
WAHR NUUUH RAUHRR!!! euu ‘aauurr nraur’ haaauurrurau! #fact
GRRUUUH! uhr hraheuuuuhhr aauurh #ChargersSuck rrhnn aur! gauhr aaurh haurerruuuhh! !! #aurh

Of course, no Twitter user can resist posting shortened URLs. They’ve been a cornerstone to the explosive growth of the service, maybe because there’s just so much interesting fast-moving crap out there on teh Internets. The Wookie follows several aggregation services — Digg, Technorati — and a smattering of other popular blogs — TMZ, The Onion Daily, LOLcats — etc. He pulls out links to recent content and shortens them with the bit.ly API. Again, since the Wookie is totally faking it, the results just cannot be accounted for. The best he can hope for is that the emotional texture of his tweet sometimes support the referenced source:

nrauh rrhn hrauuur ehrruhauhrnurrh. nrraauuuhh IMHO. hraur grraauhurhnauhrn rauh http://bit.ly/mjOD7
Ghrn euuhrr? haarneruuuhh urrn aruh rghrnn aaur, uhr hruh urrr :) http://bit.ly/1a9j1K

And no tweeter lives in a vacuum; their posts are replete with the user handles of friends, comrades and mentors.
The Wookie wasn’t about to make up handles, so his likely choices were his followees and followers.
Rather than take the name-dropping approach here — more on that later — the Wookie chooses to occasionally reference his most recent followers:

OMG! _rrrhnnneuh_ euh: urr wuuurrh rghnurrnh urhn hruhn @sleepbotzz rhagn ghrn rrhn waaahhr hruauhehuraghrrrrrnnn rhagn harrnaurh

After these features were implemented, the Wookie’s posts started to look almost real-ish. And whenever he tweets on his own, that is his range of capabilities. But he’s still not a real member of the Twitter community until he could play some other tricks. Thus began a completely separate effort; how to translate English into Wookie.

Mocking

Did I say ‘translate’? What I meant to say was ‘mock‘. After all, what can you really do with a nonsense grammar except make it look like it has meaning.

So, the Wookie was taught to mock existing sentences into his own dialect. He simply matches the initial letter (vowel / not) and preserves the word’s length and non-alpha characters (for contractions and the like). Special mappings were also added to deal with short words (the dialect only generates words 4-letters and above). And within a given tweet, he re-uses the same fake word for each instance of the real one. It’s an obscure feature, but it makes a helluva difference in some specific cases.

The totally awesome part of effort is identifying the words that don’t get mocked. There was no way I wanted to deal with semantic grammar detection, since tweets are often wildly non-grammatical. So as per usual, the Wookie fakes it. It mainly comes down to a weak analysis of quoting and capitalization patterns. He also keeps hash tags, links, handles, many acronyms, and argot — to the best of his ability.

And just for fun, he also recognizes a rather large lexicon of terms from the Star Wars universe. Well, except for the term ‘Star Wars’ that is. He doesn’t know what that means.

It took a lot of experimenting to get it right, and he still makes mistakes, but he’s getting smarter all the time. One of the interesting things I learned during development was how staccato the English language is, as compared to the long smooth yawls of Wookie. Reading back a mocked sentence out loud is a sublime experience.

You may ask, how can this awesome power best be used to serve the Good?

Re-Tweeting

Darn right the Wookie re-tweets! He simply selects a few users that he follows, derives their recent tweets, and mocks one of them up. There are some users — @darthvader for instance — which he will always re-tweet if the user has posted anything fresh. Otherwise it’s a simple random selection, after avoiding repeats (there’s extensive repeat-avoiding code all throughout the Wookie implementation). There is the slight hint of name-dropping here, since he tends to follow a lot of popular accounts, but that’s just the nature of this beast.

RT @warrenellis Rauuh’r @neilhimself ehr #neilfail ar hruun rraaauuuuhrrr? Au. #warrenfail. http://bit.ly/16IiUE
RT @KurzweilAINews: First Close Look At Stimulated Brain: Aghhrrrrnn gaauuuhhrrr ar hrrauur aauuuhrrn u gahrrn … http://bit.ly/17dg34
RT @cnnbrk Hruh. Urrnn nrrauuuhh Ted Kennedy ur “rrrhhrrr ghr rauhn hru rau wahr; wahr au Democratic Party; … http://bit.ly/3FmLyq

It turns out that injecting the re-tweet ‘header’ will often push longer ones past the 140ch barrier. He will attempt to preserve as much of the original tweet as possible, focusing on trailing URLs and hash tags. And if the tweet is short enough, he posts a shortened link to the original post, primarily to show off his mad skills. He is much inclined towards tweets which have a good blend of mockable and preservable words, again, to show off his mad skills.

This became the second Personality Engine bot. Yet still, re-tweeting is a one-way street, and interaction is the real key to user engagement.

Playing Well With Others

The third Personality Engine bot was borne of the need to perpetuate the following cycle of fun. On a regular basis, the Wookie will search for references to relevant words — wookiee, kashyyyk, etc. — and will respond to the user with a generated tweet. This is much less invasive and cruel than auto-following, a botting practice which I find to be quite gauche. I can only imagine the surprise on these user’s faces:

@amynicole21 WTF! aruh nrauuhehruhaaauuurr rraaahhhrr nruuh urrrrn ghrn wurh: euu haarrn nuuh grruhuhrnuhhrrn erruuuuhh :)
@DZ1641 gauuuhrr??
@vfigueroa1 rghrurr waahr! rrhneuuuhr urr nruuh hrauh – *ehrraaah* nrauh ^_^ nraur hruun rrrghhnn

However, before he goes searching, he first looks at his recent mention history, specifically at tweets starting with @cr_wookie. If one is found, he will mock and publicly respond to it, linking back to the original post when length permits. So if you talk to the Wookie, there’s a reasonable chance that he’ll republish you. To minimize abuse of this feature, he doesn’t follow quite the same word preservation rules as he does for follower re-tweeting. But he’ll keep Star Wars words, and that opens up a vast realm of potential amusement.

. @kindadodgy Nurh U hraaauuuu rghn Wookie rauuhrr wau’r hruh nrh ghrn hrun ‘rhngn ruhrn uhr wurh rn nuuh gh, ahrn au nrruuuuhh gauurh.
> @adamlampert Ar. U hru nrh raaghrr gh HR’N! Rghn ruuhr au! http://bit.ly/591qW
.@Lillput Nrh, rhag’n rauuhr nurh ghr haurr au a rghhnn ur a rghn.

Greeting New Followers

The fourth and final Personality Engine bot is the greeter. When you follow him, he’ll DM you. Short and to the point.

Summary

Whew! All in all, the project required about 6 weeks of spare time. My only hope is that much hilarity will ensue from these efforts.

If you want to read a bit more about the Wookie — and who wouldn’t, right? — you can check out his Wiki page.

When Twitter came back online yesterday afternoon after their networking attacks, I got a torrent of @cr_snake_bacon tweets. Wasn’t sure why, but it seemed suspicious. Twitter’s API had flopped around for most of the day, so the logs were full of Exceptions and … oops! … re-connect attempts!

Of course I’d built the bots to re-tweet on an Exception. They’re all configured to wait 60 seconds, then try again. But of course until I fixed the configuration over night, they did exactly what a bot would do … conspicuous …

The service attacks on Twitter continued through today, and I’m sure that the birdy techs are furiously building black ice fortresses in Scala even now. Again, I saw a burst this afternoon from all of my bots. Pokey the Penguin, Conet Project, and Chewbacca all had several things to say, all at once. Obviously I had fucked something else up, so I hurriedly checked the logs. And nope … actually, my change had worked … Twitter had just un-blocked my IP.

*whew*

I’m not exactly sure how many bots are out there … here’s a nice wiki being kept of them. But I can imagine I’m not the only one who made that try-again coding mis-calculation. What’s sweet is that it’s un-done now, and my toys can continue prattling on.

Thanks, guys. Sorry we looked like a vicious autonoma for a while there. Glad to be back.

the hierarchy of preferences is somewhat distributed however …

• the Prowl site will explain all the installation stuff. the first level of preferences is configuring the Prowl plug-in to be your default Growl notification route. it’s good that the Mac plug-in allows you to choose a pass-through notification method so that you will still see the events locally
• then there’s the iPhone client preferences, under Settings. they explain in the FAQ that you need to change your settings, then start up the Prowl client to commit them to the server. confusing, because your changes don’t take effect until that commit, so it’s just a matter of knowing. also, the Settings only allow you to disable Sound while the app is running (which is a nice feature)
• it took me a further read to figure out that i needed to disable the background push Sound & Vibrate feature through the general Notification Settings. you can tweak the Prowl app so that it’s seen and not heard

i’m a big fan of TwitterFox because it’s equally simple & straightforward. i chose Scalaris as my Mac Twitter client of choice for the same reason — i don’t needs one of the feature-rich hefty ones

eventually i’ll be updating my AWS health checks to use the prowl Ruby gem. very thoughtful!

thanks to @laughingsquid for the tip on Prowl. i’m looking forward to a further geometrical increase in information overload!

my Ambience for the Masses app is a Spring / Hibernate / JSP stack, with a couple of other sweet components. i run it in on an AWS instance. it’s been purring along just wonderfully for months now. then, about ten days ago, it just stopped working

normally Java apps don’t die without throwing some sort of Exception. but that’s just what was happening. so i stripped out various components — thank you, Dependency Injection pattern! — and found that it would sometimes die instantly (if i was lucky) but usually it took a couple of hours. i don’t have a lot of free time to track down random intermittent bullshit like this, so it took me about a week to boil it down

it was somewhere in the Current Listener Map — my Shoutcast-listener-tracking geo-positioning statistics-gathering data sculpture back-end engine. i hear that the kids call them things mash-ups. the geo-lookup APIs were the most delicate part, and it seemed to work with them omitted. that red herring aside, it turned out to be the IP address resolution

try {
	if (getLog().isTraceEnabled())
		getLog().trace("lookup : " + hostname);

	// oh, lookup!
	InetAddress ipAddress = InetAddress.getByName(hostname);

	// NOTE: this is where it went bad on the AWS image
	//   *sigh*

    return ipAddress.getHostAddress();
}
catch (UnknownHostException e) { }
catch (Exception e) {
	getLog().warn("failed to lookup " + hostname, e);
}

that block was just a couple of lines until i’d added all the logging and desperate Exception handling. the app launches a lot of threads, so tracking down the issue was annoying … but eventually there it was in the traces. the stack just terminated when the app tried to getHostAddress (not during getByName though … must be a lazy-loading thing)

so i nearly had it all tracked down to that. then Tomcat inexplicably became unable to find basic JARs in /usr/share/java — i was using Fedora 8′s RPM version vs. raw Apache, and it’s organized real funny-like

so i threw up my hands and started up a fresh instance of my webapp AWS image. i’d rebooted the existing one, and that hadn’t helped at all. of course, the issue magically disappeared on the new instance. did anyone see that coming from a distance? ya probably did. cuz it’s ironic. and it’s the title line of the damn blog post

the hostname resoultion is surely a low-level OS thing. both Linux JavaSE 6 and IcedTea 7 just shat the bed when they got to that point, unlikely unless they both leveraged the same lib call. something must have gone wonk in the virtualization, and apparently a key part of the solution was running it inside of a different farm. i wasted a helluva lot of time to find that out

lesson !! if weird inexplicable freaky-ass things start happening to your cloud-hosted app, load it up on a new VM earlier than later. i’d taken a late-stage backup of the failed instance and assumed it would be corrupted with the Mystery Bug (read as: a waste of time to attempt). but oh no, it worked just great :) . next time it’ll be a cinch to just bundle the instance up, image it, and use it to launch a new one

and ultimately … it wasn’t a bug in my code !!!

i’m working for a new company now, and they’re rocking it for RoR apps on the iPhone. sounds like a good place to be. one of the many reasons why this position works for me is because these guys are all about GTD and getting it out there. lean ‘n’ mean

whereas i’ve become very used to a holistic detail-orented, wisened test-backed process. great for Enterprise, but not so much for the reckless streets of Startup 3.0 . so i’m in a learning process. i’ve turned around some good stuff quickly, and it’s very satisfying

but i’ve screwed the pooch twice since i’ve been there. it’s totally a judgement call thing — i’m shooting too fast from the hip, and don’t feel like i really grasp the balance here …

first project i worked on was related to account management. they wanted a quick turn-around, i gave it a shot, had the whole thing backed with solid testing, and ready for on-time deployment with a smile. and in trying to keep track of all the new system permutations — i’d been there 2 weeks or so — i forgot one basic thing, and forgot to test for another. a nice little Perfect Storm. one emergency 1am database rollback later, we had a load of pissed customers and a helluva lot of explaining to do

so, then this past week, i went in to fix a minor rounding issue bug. those can be touchy. the right way to do it is with BigDecimal. yep, i’ve done that in Java too with BigDecimal. overall, it’s somewhat ponderous, detail-oriented, and can easily be polluted with Floats and the like. so i’d taken a shortcut, realizing that the low-level C impl was doing String conversion without the rounding issue. so i took the low-hanging fruit:

total.to_s.to_i

awesome !!1!. well, that is until you get into the 100-of-trillions area, otherwise shown as 1.0e+14. guess what happens when you parse that into a Fixnum? no database rollback this time, but Da Boss had to spend days sorting out the visceral impact of ridiculous sums of bogus exploit money pouring into our RPG

security, privacy and account management. payment calculations. not the sort of things to take shortcuts on. yet, if you’re embracing a culture that wants it done quickly and with minimum impact, it’s a risk you might be willing to take. it’s not like i didn’t have test scripts … i just forgot to head into scientific notation territory. just like i forgot to check for the implication of null password acceptance ( long story there, special account cases, etc. )

i’m putting these things up here for my fellow developers to laugh at.   “I mean, c’mon. All that’s totally obvious stuff.”   “I’d never miss that, that’s sophmore shit.”   good, get it out of your system, laughing boy

but believe me, when you’re on the other end of it, and had been in the middle of it and all full of all the other things that you needed to keep track of at that time, heh, well, that’s when you’ll really need to keep yerself laughing :)

I sought advice from Steve Enzer, and his answer was ‘create a separate FB app id’. Yep, you can do that. I’m sure it works just fine. However, I wanted to come up with a single-app solution, you know, just because.

My app backbone was based upon Facebooker, a great gem for just such purposes. And assuming that you’re using that gem, you’re provided with:

rake facebooker:tunnel:start

Which starts up the traditional server tunnel to your local dev box; Facebook’s proxy won’t know the difference. I’ve also set up my canvas page URLs and the like with a fixed IP — according to FB’s best practices, because they don’t want to deal with all the DNS resolution stuff — and it listens on port 80 with a dedicated context. So, I’ve got my facebooker.yml set up for port 3100:

tunnel:
  public_host: cantremember.com
  public_port: 3100
  local_port: 3100

And here’s my nginx config:

#
#	included into vhosts/localhost.conf (*.by-ip)
#		answers to :80
#

#	registered with FaceBook
location /forgiveness {
	#	no-trailing-/ case (the rewritten URI has a zero length)
	if ($request_filename = '') {
		rewrite  (.*)  $1/;
	}

	#	the only way i could figure to get conditional proxying
	#		proxy_pass via variables = no
	#		if { proxy_pass } = no
	#		so ... two more goofy contexts

	#	magic tunnel cookie (any value)
	if ($http_cookie ~ '_forgiveness_tunnel=1') {
		#	last : stop rewriting, but re-process through nginx (vs. break)
		rewrite  /forgiveness(.*)  /forgiveness-dev$1  last;
	}
	rewrite  /forgiveness(.*)  /forgiveness-pub$1;
}

location /forgiveness-pub {
	#	root contextify
	#	break to terminate re-processing through nginx
	rewrite /forgiveness-pub(.*)  $1  break;

	proxy_pass  http://cantremember_forgiveness_cluster;
}

location /forgiveness-dev {
	rewrite /forgiveness-dev(.*)  $1  break;

	#	same as public upstream
	proxy_pass  http://127.0.0.1:3100;
}

Here I omit the upstream cluster configs, and I’ll let the comments speak for themselves … I always try to keep track of red herrings at least at some level, and there were several I bumped into while trying out this solution. The best approach that I found was to simply set a cookie with a naming convention that nginx could parse (yes, there is a regex performance penalty here). The development context is routed locally, which is where the tunnel is waiting to pick up and run with it.

Setting the cookie, well that’s another trick. Though not much of one. Of course your app is completely masked behind the apps.facebook.com domain, so trying to set a cookie to your internal domain (eg. IP) through your browser isn’t so easy, and it’s repetetive manual work. So, just bake it into a dedicated action in your app:

#	allows for Facebook pub / dev tunneling
#		have to set the Cookie within the scope of the app
#		can't use Firefox cookie editor to hack it
def tunnel
	#	how convenient!
	value = params[:id]

	#	this is how we get pub / dev tunneling
	#		see support/nginx
	if value == 'nil'
		cookies.delete :_forgiveness_tunnel
	else
		cookies[:_forgiveness_tunnel] = value
	end

	###render :text => "OK #{value}", :content_type => 'text/plain'
	redirect_to root_url
end

Then use /:controller/tunnel/1 to engage, and /:controller/tunnel/0 to disengage. A little bit of auth logic, and you’re good to go.

i ponied up cash for the Apple Developer License a while back. and lo & behold, developers can download the DMG for the 3.0 Beta firmware update from the iPhone Dev Center! so i went and installed iPhone OS 3.0 Beta tonight … i wanted to check out the cut’n'paste capabilities, etc. downloaded and extracted the firmware update package, started iTunes, held down [Ctrl+Option] when clicking ‘Check for Update’ to bring up the file selector dialog, and installed the 3.0 Beta IPSW

great! hooray! it’s the usual slow process. however, once the device restarts, it goes into pre-activation mode … and iTunes rejects it as not behing a registered development device. open up the Device Management Portal, and it tells you how about Locating a Unique Device ID … “The 40 hex character string in the Identifier field is your device’s UDID.” i believe they refer to it as an ICCID in other places

alright, where can one find this magical string? well, it’s shown when your device is connected to iTunes … but of course, that’s the problem! it already won’t accept my device = meta-problem. so, then i read further … “Please DO NOT install the iPhone OS before registering device UDIDs, as installation on non-registered devices will render them inoperable.”

so i figured that i’m toast. i can easily find the Serial Number and IMEI, but not the UDID, at least according to Apple’s instructions

making a long story short … look at System Profiler (if you haven’t already) … under USB :: USB High Speed Bus :: iPhone :: Serial Number. register that as a developer device, and you’re good to go as soon as you re-boot the phone & iTunes. there’s a much more elaborate description of the process using Xcode, but it doesn’t seem to be mandatory cause i sure ain’t done that

and yes … cut’n'paste is really nice!

Fortunately, I haven’t lost any followers (yet). But this was a perfect case of how mixing business with pleasure can have unintended consequences.

What Have I Learned

Or rather, what have I re-learned …

Soft-disable features in Production at Launch Time

 
My Twitter engines are built with both an :enable_tweet and :enable_greeting config setting. In the git repo, they’re both true. When I did my local testing, I’d disabled them correctly. When I launched in Production, I neglected to make the quick-and-dirty changes; after all, everything worked great. And once started, my scripts correctly responded to the no-initial-state condition, and greeted everybody.

Launch preparation is critical, even for little projects. The start-up mentality is to move fast and lean, but there’s such as thing as too fast, and probably as too lean too. Gradual uptake migration is a wise strategy even for the ‘little things’.

Mock and Integration Testing Only Gets You So Far

 
I used rspec to mock out the full capabilities of the engine. Found some real-world issues, resolved them. I also wrote some core integration tests, ran them locally. Immediate failures. I had mocked documented features that didn’t actually exist. Fixed, re-mocked, re-tested, fixed again, etc .

Another great reminder that you can only mock something you trust, and how can you trust something you haven’t actually run under integration conditions to start with! Re-tested integration, and everything passed with flying colors. Sure, the features worked great now! And when I launched them, they did exactly what I asked.

So, as if we haven’t heard it enough times, be careful what you ask for!

All of this is familiar to anyone who has made a mistake in the software industry. It’s not like I haven’t successfully executed dozens of critical launches in the past, and most with virtually no issues at all. But what’s interesting is what happens when these mistakes happen in a public forum, and whom you expose them to — say, your friends :)

And who can say when two ounces of caution is more deserving than one … without the benefit of hindsight.

Just ask anyone who has a stringent backup policy how much time & effort they invest to avoid an event that may never actually happen. That stringency usually comes from that one unforgettable experience, and from there is born an extra layer of caution, and an additional time-sink (eg. mock & integration testing)

Heh. KISS. So, what exactly is simple? DRY. Isn’t that supposed to be a time-saver? Well, it depends on what you’re not trying to repeat. Strange how these cuddly and liberating acronyms can have more than one interpretation.

Experience taints everything.

Coming Up To Speed on Rspec

So, learning rspec has been on my list for a while. I finally got around to it. Nice framework. I am familiar with EasyMock, and am aware of JMock. I never had an opportunity to get into Mockito, but I’d give it a glance next time I put on my Java hat.

There is some decent documentation on it out there. I found these links particularly helpful:

• The formal documentation.
• David Chelimsky’s from-the-ground-up description
• rspec.info’s drill-down into Mocks and Stubs
• Luke Redpath’s cross-over into usage under Rails

During my ramping-up, I took the usual meta-approach of creating a test suite — and yaaay, that’s what rspec is meant for! — which I then used to test out its own range of capabilities. The Modules in the rdoc which I’ve found to provide the most value are:

• Spec::Expectations::ObjectExpectations for conditionals (eg. should & should_not)
• Spec::Matchers for expectations (eg. equal(value), be_a(class), respond_to(method_sym), raise_error)
• Spec::Mocks::Methods for mock method definition (eg. should_receive(method_sym))
• Spec::Mocks::MessageExpectation for mock behaviour (eg. with(*args), once, exactly(n).times, any_number_of_times)
• Spec::Mocks::ArgumentConstraints for mock arguments (eg. an_instance_of(class), anything)
• Spec::Mocks::BaseExpectation for mock responses (eg. and_return(value), and_yield(&block))

I won’t got into the deep details, but here are some examples of conditionals and expectations that I paraphrased into my meta-test:

specify "knowledge of nil" do
	''.should_not be_nil
end

specify "numeric calculations" do
	(355.0 / 113).should be_close(Math::PI, 0.1)
end

specify "changes made by a closure" do
	array = []
	lambda {
		array << :a
	}.should change(array, :size)
end

specify "equality" do
	5.should eql 5
	5.should equal 5
	[:a].should_not equal [:a]
end

specify "raising of errors" do
	#	always construct fresh!!!
	#		otherwise it won't be re-wrappable
	def raiser(s=nil)
		lambda { raise(RuntimeError, s) if s }
	end

	raiser('x').should raise_error
	raiser('y').should raise_error(RuntimeError, 'y')
end

specify "closure satisfaction" do
	5.should satisfy {|n| (4..6).include?(n) }
end

specify "what lists do" do
	[].should respond_to(:each, :find, :size)
end

And here’s a little bit of silly mocking:

def expect_raise(type=Spec::Mocks::MockExpectationError, &block)
	abort 'block must be provided' unless block_given?
	block.should raise_error(type)
end

specify "the basics" do
	@mock.should_receive :hello
	@mock.should_not_receive :goodbye

	@mock.hello
	expect_raise { @mock.stay }
	expect_raise { @mock.goodbye }
end

specify "how many times, and with what" do
	@mock.should_receive(:one).once.with(1)
	@mock.one 1

	@mock.should_receive(:string).exactly(1).times.with(an_instance_of(String))
	@mock.string 'a string'

	@mock.should_receive(:anything).exactly(3).times.with(any_args())
	@mock.anything
	@mock.anything :again
	@mock.anything :third, 'time'

	@mock.should_receive(:array_ish).with(duck_type(:each, :find, :size))
	@mock.array_ish [:item]
end

specify "what i return, raise or throw" do
	@mock.should_receive(:get_one).and_return(1)
	@mock.should_receive(:put_one).with(1)
	@mock.put_one @mock.get_one

	@mock.should_receive(:increment).any_number_of_times.with(instance_of(Fixnum)).and_return {|i| i + 1 }
	@mock.increment(1).should equal(2)

	@mock.should_receive(:raises_string).and_raise('something runtime')
	expect_raise(RuntimeError) { @mock.raises_string }

	@mock.should_receive(:pitch).and_throw(:ball)
	lambda { @mock.pitch }.should throw_symbol(:ball)
end

specify "yielding in a complex fashion" do
	@mock.should_receive(:gimmee).exactly(3).times.and_yield(:x)

	holder = []
	@mock.gimmee {|value| holder << value }
	holder.should eql([:x])
	2.times { @mock.gimmee {|value| holder << value } }
	holder.should eql([:x, :x, :x])
end

specify "validation via closure" do
	(@mock.should_receive(:threely) do |value|
		value.to_s.size.should eql(3)
	end).exactly(3).times

	@mock.threely 'x' * 3
	@mock.threely :key
	@mock.threely 333
end

These are just ways I thought of to exercise the width and breadth of the library. Very nice. I hope that these are useful examples for people new to this gem.

I recommend the other references from above for filling in the missing details. Once you have a context / specify or a describe / it specification set up, you’ll be good to go. There’s much more to the library — Stories, for instance — but that’s for another day.

An Informative Walk through Twitter4R

No, I didn’t really want to spend time learning rspec — I mean heck, I’m busy — but I had a personal need to expand the Twitter4R gem. Specifically, I wanted to add on some Twitter search features, and I was very impressed with how this library has been built. Contribution-wise, the final step that Susan Potter recommends is to craft up some rspec tests.

Of course, mock testing is only as good as the framework you’re built upon. The assumption is that Net::HTTP is going to do it’s job, so mock it up and you can even test your Twitter features offline. When I built Bitly4R (given that name, my thinking has clearly been influenced), I did everything as full-on functional tests. It was easy; bit.ly has both shorten and expand commands, so I could reverse-test real values without having any fixed expectations.

However, Twitter is live and user-generated, so who knows what you’ll find. Mocking covers that for you. And of course not having to hit the service itself shortest testing time dramatically.

Here’s one of my tests, again foreshortened:

before(:each) do
	Twitter::Client.send :public, :create_http_get_request

	@client = Twitter::Client.new
	@client_clone = Twitter::Client.new
end

it "produces a querystring with all the crazy stuff Twitter imagined" do
	the_params = nil

	@client.should_receive(:create_http_get_request).and_return {|uri, params|
		the_params = params
		@client_clone.create_http_get_request(uri, params)
	}
	@client.should_receive(:http_connect).and_return Net::HTTPNotFound.mock(:code => '404')

	#	pass through a few we can pick back out
	#	also, side-effect ... nil on invalid response
	@client.search('query_value', :rpp => 3).should be_nil

	the_params.should_not be_nil
	the_params[:q].should eql('query_value')
	the_params[:rpp].should eql(3)
end

Plus a little mocking of Net::HTTPResponse:

class Net::HTTPResponse
	def self.mock(ops={})
		#	defaults
		ops = {
			:http_version => '1.1',
			:code => '200',
			:message => 'OK',
			:body => nil,
		}.merge(ops || {})

		#	construct
		###clazz = CODE_TO_OBJ[ops[:code]]
		clazz = self
		response = clazz.new(ops[:http_version], ops[:code], ops[:message])

		#	inject
		ops.each do |k, v|
			response.instance_variable_set "@#{k}".to_sym, v
		end

		#	mockulate
		response.instance_eval %q{
			def body; @body; end
		}

		response
	end
end

Nothing like having some good inspiration to get you thinking about maintainability. I’m a not a strong adherent to any of the TDD or BDD denominations, but testability itself is close to my heart. Just ask the folks who’ll be looking at the micro-assertion Module that I wrote for my Facebook Puzzle submissions (which work locally under test conditions but fail miserably once thrown over their wall). Then again, I won’t need to write that (and test that) from scratch again.

So, back to talking glowingly about the Twitter4R architecture. Yes, I used that word. Yet regardless of that term’s heavyweightedness, it’s simply the result of carefully thinking out the formalization of a complex design. And once a non-author delves into extending an existing implementation, that impl’s demeanor becomes readily clear :)

Some of the interesting things I saw:

• An inner bless-ish strategy, to inject the Twitter::Client into instanciated result classes. I’ve seen blessing as an OO-Perl-ism, for self-ication, and that metaphor carries over very nicely to this context.
• Generous use of blocks / lambdas / closures in methods, for contextual iteration (eg. iterate through each Twitter::Message in a timeline response). Unnecessary, but an excellent convenience for a language that offers that optional capability in a very disposable fashion (from the caller’s perspective).
• Retroactive sub-object population after construction. Twitter4R relies upon the json gem, which deals primarily in Hashes. Post-injection, the library itereates through Arrays of Hashes and transforms them into suitable Twiltter::User containers, etc. A great place to put such logic in the construction chain, and it doesn’t take long to get really tired of Hash hierarchies.

Good stuff, and learning with in a Ruby-centric mindset was invaluable for me. We all have to start somewhere, eh.

The Acute Long-Term Pain of Staticness

There was one issue that I ran into; static configuration. During my years of using the Spring Framework, my IOC-addled brain started thinking of everything in terms of instances — Factory Beans, POJOs, Observers & Listeners, Chain-of-Responsibility wrappers. Static configuration is a common metaphor, and in this case, there’s a static Twitter::Config instance. Convenient and centralized. Makes perfect sense.

I mean, the fact that it was a configurable library at all was awesome. I was able to easily set up a Twitter::Client to reference search.twitter.com. However, of course as soon as I did that, I whacked the ability for clients to talk to twitter.com in the process. Oops!

On GPs, I refused to modify the original code. And I wanted to make sure that my superficial tweaks to the library would be thread-safe — temporarily swaping out the global Twitter::Config in mid-operation would be an issue. Using Mutex.synchronize seemed like the perfect choice. After finding that the same thread can’t lock a Mutex instance twice — grr!, that’s a great trick if you can work it — I overrode the one method that cared the most about @@config:

@@config_mutex = Mutex.new
def configuration_mutex
	@@config_mutex
end

alias :raw_http_connect :http_connect
def http_connect(*args, &block)
	ops = (Hash === args.last) ? args.pop : {}

	result = nil
	self.configuration_mutex.synchronize do
		ops[:call_before].call if ops[:call_before]
		result = raw_http_connect *args, &block
		ops[:call_after].call if ops[:call_after]
	end

	result
end

#
#	...
#

@@SEARCH_CONNECT_HOOKS = {
	:call_before => lambda { self.configure_apply(:search) },
	:call_after => lambda { self.configure_apply(:default) },
}

#
#	...
#

#	broken out of closure for mocking
req = create_http_get_request(uri, params)
response = http_connect(@@SEARCH_CONNECT_HOOKS) {|conn|	req }
return nil unless Net::HTTPOK === response

model = bless_model(Twitter::Search.unmarshal(response.body))

It works like a charm. Please, everyone just line up to tell me how I could have done it better. And I don’t mean quicker or cheaper, I mean better. Believe me, I would not have sunk the time into this end-around approach, if not for the fact that:

1. I don’t want to maintain a local gem code modification (even though my impl is closely coupled to the gem impl already)
2. I intend to follow that practice, so every opportunity to pull and end-around is a Valuable Learning Experience.

So, now my local Twitter4R has search capability gracefully latched onto it (and implemented much in the flavor of the library itself). I have a mass of rspec examples to work off in the future.

Now, I haven’t spent a great amount of time testing the thread-safeness — no one in their right mind wants to do that — but my sundry Twitter::Client instances play nicely together in between searches and normal status operations.

And I had something useful to blog about!

• Your Account Number
• An Access Key
• A Secret Access Key
• An SSL Certificate file
• The Private Key for your SSL Cert

The various AWS command APIs and tools will require you to provide one more more of these pieces of information. Rather than actually host all of this information on my instances, I have instead chosen to build Ruby scripts using:

• Glenn Rempe’s Amazon EC2 gem (amazon-ec2)
• The net-ssh and net-scp gem suite

I’m small-scale for the moment, so I can keep a centralized list of my instance IDs. Without too much effort, I can look up the instance(s), identify their public DNS (plus everything else), and then open up an ssh connection and push data into a channel’s environment or upload files for bundling purposes.

Here’s a few things that I’ve learned in the process.

Channels

Most of your core work gets done on a Net::SSH::Connection::Channel, and sometimes asynchronously (as in the case of my Health Check script). There are a lot of library shorthands — the prototypical Hello World example is channel-less:

Net::SSH.start("localhost", "user") do |ssh|
	# 'ssh' is an instance of Net::SSH::Connection::Session
	ssh.exec! "echo Hello World"
end

But as with all examples, you soon find that surface-level tools won’t quite do the trick. The main thing you’ll need to have all the time is convenient access to the data come back from the channel, asynchronously or otherwise. So create yourself a simple observer:

def observer_struct
	@observer_struct ||= Struct.new(:channel, :stdout, :stderr, :exit_status, :exit_signal)
end

def observe_channel(ch, ops={})
	observer = observer_struct.new(ch, '', '', 0, nil)
	is_puts = !!ops[:puts]

	#	stdout
	ch.on_data do |ch, data|
		$stdout.print data if is_puts
		observer.stdout << data
	end
	# only handle stderr
	ch.on_extended_data do |ch, type, data|
		next unless [1, :stderr].include? type
		$stderr.print data if is_puts
		observer.stderr << data
	end

	ch.on_request("exit-status") do |ch, data|
		observer.exit_status = data.read_long
	end
	ch.on_request("exit-signal") do |ch, data|
		observer.exit_signal = data.read_long
	end

	observer
end

def open_channel(ssh, ops={}, &block)
	ch = if block_given?
		ssh.open_channel &block
	else
		ssh.open_channel
	end

	#	somebody's watching you
	self.observe_channel ch, ops
end

Then during and after executions, you’ll have access to all the core information you’ll need to make conditional decisions, etc.

Pushing Your Environment

Many of the AWS tools will recognize specific environment variables. The usual suspects are:

• AMAZON_ACCESS_KEY_ID
• AMAZON_SECRET_ACCESS_KEY

So, you can use Net::SSH::Connection::Channel.env to inject key-value pairs into your connection. However, you’ll need to make some config changes first — and major thanks to the Net::SSH documentation for clarifying this in its own RDoc:

“Modify /etc/ssh/sshd_config so that it includes an AcceptEnv for each variable you intend to push over the wire”

#       accept EC2 config
AcceptEnv  AMAZON_ACCESS_KEY_ID AMAZON_SECRET_ACCESS_KEY
...

After you make your changes, make sure to bounce the ssh daemon:

/etc/init.d/sshd restart

Then your selected environment will shine through.

Running sudo Remotely

It’s reasonable to execute root-level commands using sudo, because you can provide a good amount of granular control.

The first thing that we’ll all run into is:

sudo: sorry, you must have a tty to run sudo

Fortunately, there’s Net::SSH::Connection::Channel.request_pty:

ch.request_pty do |ch, success|
	raise "could not start a pseudo-tty" unless success

	#	full EC2 environment
	###ch.env 'key', 'value'
	###...

	ch.exec 'sudo echo Hello 1337' do |ch, success|
		raise "could not exec against a pseudo-tty" unless success
	end
end

I’ve taken the approach of allowing NOPASSWD execution for everything I do remotely, after making darn sure that I had constrained exactly what could be done under those auspices (HINT: pounds of caution, tons of prevention). You can configure all of this by editing /etc/sudoers:

# EC2 tool permissions
username  ALL=(ALL)  NOPASSWD: SETENV: /path/to/scripts/*.rb
...

Also, if you intend to have those scripts consume the environment variables that you’ve injected into your channel, you’ll also need to annotate the line with SETENV, or they won’t export across into your sudo execution.

If you are more security minded, then there’s many other variations you can play with /etc/sudoers. I haven’t yet experimented within pushing a sudo password across the wire, but that may be where Net::SSH::Connection::Channel.send_data comes into play.

Port Forwarding

I wanted to easily do SSH tunneling / port-forwarding against an instance, referenced by my local shorthand. So I wrote my usual EC2 instance ID lookup and started an Net::SSH connection. Here’s how you put yourself in ‘dumb forward’ mode:

def ports, offset = [], 0
# ...

Net::SSH.start(hostname, user, :forward_agent => true, :verbose => :warn) do |ssh|
	ports.each do |port|
		ssh.forward.local port + offset, 'localhost', port
	end

	# loop, with an event loop every 0.1s, until Ctrl-C is pressed
	puts "[Ctrl-C] to terminate..."
	interrupted = false
	trap('INT') { interrupted = true }
	ssh.loop(0.1) {
		not interrupted
	}

	ports.each do |port|
		ssh.forward.cancel_local port + offset
	end
end

And then [Ctrl-C] will break you out.

Inconsistent EC2 Marshalling

I’d gotten all of my EC2 data parsing working great, then I started to expand my capabilities with the Amazon S3 API gem (aws-s3). Suddenly and magically, the results coming back from my EC2 queries had taken on a new form. Moreover, this was not consistent behavior across all operating systems; at least I do not see it on WinXP, under which my health monitoring checks execute.

The Amazon APIs are SOAP-driven, so I guessed that amazon-ec2 (and aws-3) would both leverage soap4r. Well, that is in fact not the case; Glenn has commented below on this known issue. He uses the HTTP Query API, and the cause of the discrepancy are (as of this writing) still undetermined.

Regardless, our job is to get things to work properly. So for the meanwhile, let’s use a quick & dirty work-around. The two core differences are:

• the data coming back may be wrapped in an additional key-value pair
• single value result-sets come back as an Object (vs. an Array with a single Object)

So I crafted a little helper singleton, which which still allows me to detect between ‘valid’ and ‘empty’ requests (nil vs []):

class SOAPHelper
	class << self
		def data_from(data, wrapper)
			data[wrapper] || data
		end

		def items_from(set)
			unless set && set.item
				nil
			else
				items = set.item
				items = [items] unless Array === items
				items
			end
		end
	end
end

data = SOAPHelper.data_from(
	@ec2.describe_instances,
	'DescribeInstancesResponse'
)
reservations = SOAPHelper.items_from(data.reservationSet) || []
...

That will address our immediate needs for the moment, until a benificent coding Samaritan comes along.

After some consideration, I chose to go with msmtp. I like its flexible configuration, and it’s just the right size for the job (with room to grow). The major thing I was looking for was STARTTLS support. I wasn’t so concerned with the trust files and certificates, I just had a need to support GMail’s minimum requirements. Yes, msmtp gives you that and the whole 9 yards, for when I need them all.

msmtp Configuration for GMail

With a combination of their official configuration example plus a few targeted suggestions from Grey Bearded Geek’s take at ssmtp, I came up with the following:

# Set default values for all following accounts.
defaults

host  smtp.gmail.com
port  587
timeout  off
protocol  smtp
domain  localhost

# /usr/bin/msmtp --version
#  plain cram-md5 digest-md5 gssapi external login ntlm
auth  on
user  GMAIL-USER
password  GMAIL-PASSWD

tls  on
tls_starttls  on
# WARNING: When the checks are disabled, TLS/SSL sessions will be vulnerable to man-in-the-middle attacks
tls_certcheck  off

logfile  /var/log/msmtp.log
###syslog  on

# A system wide configuration is optional.
# If it exists, it usually defines a default account.
# This allows msmtp to be used like /usr/sbin/sendmail.
account default

# Construct envelope-from addresses of the form "user@oursite.example".
###auto_from on
from  USER@DOMAIN
maildomain  DOMAIN

You plop in the GMAIL-USER and GMAIL-PASSWD, and you’re good to go.

Custom From: Address

I soon learned that the from and maildomain settings are irrelevant; Google will not arbitrarily change the From: header of your mail. That makes sense. So the mail will appear as if it’s coming from you, personally. Well, it turns out that there’s a few things you can do to get around that.

• Create yourself a dedicated GMail account. Now you have isolated your soon-to-be-wildly-popular start-up’s e-mail account from your personal one.
• Follow the instructions on adding a custom From: address to your account. I had to use the older version of the GMail interface to do so. Google will verify that you own the address — you’d better be able to receive mail at that address — and then you can make it your default.

  GMail will now send your mail as it were coming from that address, but it will do so without providing an alias.

• When sending your outbound mail, you can include the following headers:

  From:  ALIAS
  Reply-To:  ALIAS <USER@DOMAIN>

  Google will respect the ALIAS portion of the From: address, though not the address itself. The Reply-To: is optional, but respected in its entirety (alias and address).

Works like a charm.

That being said, it still needed some adjustments …

WordPress

I’m currently running WordPress 2.7 under nginx 0.7.27. Here’s my end configuration:

server {
	include  extra/proxy.conf;
	include  mime.types;

	listen  80;
	server_name  blog.DOMAIN.NAME;

	access_log  logs/LOGFILE-access.log;
	###error_log  logs/LOGFILE-error.log  debug;
	error_log  logs/LOGFILE-error.log;

	# Settings | General | WordPress address (URL)
	location /WP-CONTEXT {
		# physical location on server
		root  /PATH/TO/WP-DIR;
		index  index.php index.html index.htm;

		# remove virtual path, make it root-relative
		rewrite  ^/WP-CONTEXT(.*)$  $1;

		# this serves static files that exist without running other rewrite tests
		if (-f $request_filename) {
			expires 30d;
			break;
		}

		# addresses '/wp-admin/' and similar vein
		if (-f $request_filename/index.php) {
			# produces '...//index.php', acceptable loss
			rewrite  ^(.+)$  /WP-CONTEXT$1/index.php  last;
		}

		# this sends all non-existing file or directory requests to index.php
		if (!-e $request_filename) {
			rewrite  ^(.+)$  /WP-CONTEXT/index.php?q=$1  last;
		}
	}

	location ~ \.php$ {
		# remove virtual path, make it root-relative
		rewrite  ^/WP-CONTEXT(.*)$  $1;

		fastcgi_pass  fastcgi_cluster;
		fastcgi_index index.php;
		# physical location on server
		fastcgi_param SCRIPT_FILENAME /PATH/TO/WP-DIR$fastcgi_script_name;
		include  fastcgi_params;
	}

	# Settings | General | Blog address (URL)
	location / {
		# WordPress handles this perfectly as-is
		fastcgi_pass  fastcgi_cluster;
		# physical location on server
		fastcgi_param SCRIPT_FILENAME /PATH/TO/WP-DIR/index.php;
		include  fastcgi_params;
	}
}

I’ll provide the content of extra/proxy.conf and fastcgi_params down below — they won’t surprise you — plus the configuration for my upstream fastcgi_cluster.

The purpose of DOMAIN.NAME and LOGFILE is obvious, so let’s skip to the useful stuff.

/PATH/TO/WP-DIR

This is simply the fully-qualified path to the directory where you have installed WordPress. Big shock, I know. I put mine in ‘/var/www/wordpress’.

WP-CONTEXT

Specifically, I’m referencing the ‘WordPress address (URL)’ capture block.

WordPress 2.7 supports a differentiation between the root context of your blog and the root context of the WordPress resources themselves. I’ve taken this approach … the URL of this blog post is root-relative to my virtual hostname, but if you do a View Source you’ll see:

<link rel="stylesheet" href="http://blog.cantremember.localhost/wordpress/wp-content/themes/cantremember/style.css" type="text/css" media="screen" />
...
<link rel="pingback" href="http://blog.cantremember.localhost/wordpress/xmlrpc.php" />

It’s a nice-to-have, and in many ways allows the configuration to be somewhat easier. In WordPress Admin, under Settings | General, I have configured:

• WordPress address (URL) = http://blog.cantremember.localhost/wordpress
• Blog address (URL) = http://blog.cantremember.localhost

So my WP-CONTEXT is ‘wordpress’.

Usage

Here are the core differentiations between my config and the Elastic Dog one:

My core two sections are the ones with WP-CONTEXT. Before doing anything, I make the $request_filename context-less, so that it’s corrected relative to root. Granted, I could have skipped that step because I used ‘wordpress’ for each, but that doesn’t make for as good an example, and regex’s aren’t that expensive (don’t they have dedicated chips for them by now?).

I was having issues when WordPress wanted to take me to the Admin screen. It used the shortcut ‘/WP-CONTEXT/wp-admin‘, which is great if you’re not doing all this fancy re-writing and fastcgi_index can take over. But we are being fancy. That’s why the $request_filename/index.php text exists. It works like a charm, although there may be a more efficient way to do this.

And here is where it became an advantage to differentiate between blog URLs and WordPress resources. I’ve chosen to make my permalinks dateless — /%postname%/ . Call me crazy, but I like the way it looks on Laughing Squid. Given that’s the case, it’s hard to differentiate between ‘/some-permalink/’ and ‘/wp-admin/’. Splitting them off with the ‘wordpress’ context made this possible.

The final context-less ‘Blog address (URL)’ capture block is exactly what you’d expect.

PmWiki

I liked the simplicity and capabilities of PmWiki 2.2.0. It’s an easier decision, since I have no intention of being a grand-scale collective document facility. PmWiki is a powerful and flexible implementation with a lot of great processing directives that you can embed in a page. Yet that also makes security something of a concern (as some reviewers will point out as well). Global multi-tier password auth is available, and user-based auth is available as necessary.

This configuration is a natural extension of the WordPress one above:

server {
	include  extra/proxy.conf;
	include  mime.types;

	listen  80;
	server_name  wiki.DOMAIN.NAME;

	access_log  logs/LOGFILE-access.log;
	###error_log  logs/LOGFILE-error.log  debug;
	error_log  logs/LOGFILE-error.log;



	# PHP execution only
	location ~ \.php$ {
		fastcgi_pass  fastcgi_cluster;
		fastcgi_index pmwiki.php;
		# physical location on server
		fastcgi_param SCRIPT_FILENAME /PATH/TO/PMWIKI-DIR$fastcgi_script_name;
		include  fastcgi_params;
	}

	location / {
		# physical location on server
		root  /PATH/TO/PMWIKI-DIR;
		index  index.php index.html index.htm;

		# this serves static files that exist without running other rewrite tests
		if (-f $request_filename) {
			expires 30d;
			break;
		}

		# this sends all non-existing file or directory requests to index.php
		if (!-e $request_filename) {
			# ? => &, then remove the leading /
			rewrite  ^(.*)\?(.*)$  $1&$2;
			rewrite  ^/(.+)$  /?n=$1  last;
		}
	}
}

Everything here is obvious, including /PATH/TO/PMWIKI-DIR. Mine is ‘/var/www/pmwiki’. Here’s the lowdown:

In the *.php capture block, you’ll see that the default script is pmwiki.php. I had created a symlink to rename it index.php, but after my config re-adjustment, that became obsolete.

The non-existing file test will be triggered by the following requests:

• /Main/HomePage
• /Main/HomePage?action=edit

Those URLs exist because I’m leveraging a feature called $EnablePathInfo. The referenced documentation doesn’t do it justice … this feature allows me to have bare Group/Name URLs, much like I’m doing with my bare blog URLs. I’ll just say that I’m being SEO-minded and leave it at that.

Turning on that feature informs PmWiki to generate the URLs in that format, and it also makes the PHP script capable of parsing the CGI headers to do-the-right-thing. My original configuration required me to perform the following override hack:

include  fastcgi_params;
fastcgi_param SCRIPT_NAME '';

But the revised configuration above simply re-writes the URL into the standard '?n=‘ format and the script never has to deal with CGI headers. The only other rewrite considerations were to transform any querystring ‘?’ into ‘&’ and to remove the leading ‘/’ from the Group/Name combo.

Hooray!

Supporting Configuration

For all means and purposes, I’m using nginx’s default fastcgi_params.

This is extra/proxy.conf, derived from their NginxFullExample, with notes-to-self intact:

# can't set this here!
###proxy_redirect          default;

proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

#client_max_body_size    10m;
#client_body_buffer_size 128k;

proxy_connect_timeout   15;
proxy_send_timeout      90;
proxy_read_timeout      30;

# turn off for Comet!
proxy_buffering         on;
proxy_buffers           32 8k;

proxy_ignore_client_abort  on;

# for caching, via Last-Modified:
#proxy_store          on;
#proxy_store_access   user:rw  group:rw  all:r;
#proxy_temp_path

This is fastcgi_cluster, which is just a simple example of how to do clustering:

upstream fastcgi_cluster {
	# sticky by IP
	###ip_hash;

	# max_fails=3  fail_timeout=15s  weight=2
	# down  backup
	server  127.0.0.1:PORT-1;
	server  127.0.0.1:PORT-2;
	server  127.0.0.1:PORT-3;
}

I tried using some of the additional server setting features — commented out above — but they weren’t working in my build of 0.7.27. I can live without them at the moment, but the upstream block capabilities are quite powerful.

I’m running 3 FastCGI instances, each with 5 worker threads. Again, good enough for government work. I custom-built fcgi on OS X, but for my AWS Fedora Core 8 image I just went with spawn-fcgi that comes along with the lighttpd package.

This cluster config is also a nice starter reference for adding load-balancing capabilities to external AWS instances. Given the volatile nature of VM image mappings, I’ve split the cluster config off into its own file for scripted generation.

In Summary

I’m very pleased with nginx. It has been very stable — the only time I’ve taken it down is when setting it up for infinte HTTP 302 redirects, and even then it took several hours of user activity to knock it over. The configuration syntax is very powerful, and I haven’t once been wistful for my old Apache habits :) .

Assertions

The first thing I did was to follow the great advice of Dierk Koenig, writer of Groovy in Action and general Groovy / Grails advocate. The book itself doesn’t use the typical print-the-result-based code examples; it encourages the reader to learn the language through assertions. And that’s how I learned Groovy; I took the examples from the book, paraphrased them, tried variations on a theme, and then asserted that my results were true. Now when I need to know how to use a language feature, I simply look back at my assertion code to re-edjumicate myself.

I learned the core Ruby language via Test::Unit. I spent three weeks (please don’t laugh) worth of my daily commute writing assertions for the core language, the standard packages, plus ActiveRecord and other common gems. It allowed me to get a handle on the concepts, syntax, semantics and the sheer range of capabilities of he language. I frequently look back at my language_test_unit.rb to figure out the best use of Array.collect, catch..throw, Regexp quirks, and using declared lambdas as blocks (etc). More importantly, I’ve already written code using all of those techniques, so it’s just a referesher.

I cannot recommend this technique enough for coming up to speed on a language!

Podcasts

With that under my belt, plus some command-line scripts and a solid Rails project, I’m not spending time back-consuming posts from the following Podcast blogs:

Rubyology

Site : Feed

I’m actively back-consuming a lot of content from this wealth that Chris Matthieu has provided. There are some great talks on Journeta, using EC2, great tutorials covering basic and RoR, and some scaling recommendations.

sd.rb Podcast

Site : Feed

Straight from the mouth of the San Diego Ruby Users group. A good variety of topics, focusing more on the Ruby language than on the Rails poster-child itself. Nice talks on rspec, MySQL clustering and Arduino, amongst many others.

Railscasts

Site : Feed

With 145 postings and counting, there’s a lot to be consumed here. However, this is the last on my list, because none of them download to my iPhone 3G :( . Lots of cross coverage on Capistrano, Partials, custom Routes, ruby-debug … the list goes on.

Ruby on Rails Podcast

Site : Feed

Geoffrey Grosenbach’s podcasts are seminal. I’ll leave it up to the reader to pore through the years of accumulated wisdom. How can you go wrong when you’re part of the rubyonrails.org domain!

In Summary

A number of these feeds provide screencasts and/or video. A few of the files are old-school QuickTime MOVs which are problematic for the iPhone, which is annoying (definitely not the podcaster’s fault… get your head in gear, Apple). And unfortunately when I break away to write down something in Evernote.app, and there’s any visuals associated with the cast, the iPhone halts playback. Grr. So I’m getting into the archaic habit of creating a Notes page and mailing it to myself :)

I recommend each and all of these podcasts. Be prepared to sink a lot of time into them, so you might as well upload them onto your iPhone and take them to the beach!

This turned out not to be as easy as I’d liked. Even with a bevy of useful resources available — lotsa people have run into these issues — it took a while to get the right combination. Let’s hope I can save you a bit of that pain …

JConsole / JMX Remoting via SSH Tunnels

As I mentioned, I’m hosting this solution in the cloud. So, when things go bad, I need to be able to debug remotely. I don’t want to open up my security group to general traffic, so using SSH tunnels is the best option. JConsole is a great tool for measuring current statistics and performance of your Java app, and relies on JMX remoting. It works great locally, or even within a trusted network, but once you’re behind a firewall with broad port-blocking, there are some significant issues. There are several core Java forum topics related to this discussion:

• http://forums.sun.com/thread.jspa?threadID=5267091
• http://forum.java.sun.com/thread.jspa?forumID=4&threadID=565139
• http://forum.java.sun.com/thread.jspa?forumID=4&threadID=5119655
• http://forum.java.sun.com/thread.jspa?forumID=58&threadID=703567

Daniel Fuchs has written several articles which illustrate these issues and provide good solutions. He explains that JMX remoting needs two ports: one for the RMI registry, and one for the RMI connection objects which are stubs used for remoting all the critical data. If you’re using the default JVM agent, you’ll tend to use the following JVM System.property’s on the server:

-Djava.rmi.server.hostname
-Djava.rmi.server.useLocalHostname
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port
-Dcom.sun.management.jmxremote.ssl
-Dcom.sun.management.jmxremote.ssl.need.client.auth
-Dcom.sun.management.jmxremote.authenticate
-Dcom.sun.management.jmxremote.access.file
-Dcom.sun.management.jmxremote.password.file

I’ll come back to these, but the one that’s important here is jmxremote.port. It allows you to specify the RMI registry port, the one that you’ll use to establish your remote connection via JConsole. However, the port for RMI export, which is used for all the critical data callbacks, is randomly chosen (on a session basis or JVM basis, not sure) and cannot be specified. And you can’t open a port for tunneling if you don’t know what it is.

You can see this issue if you crank up the debugging on JConsole. I was having issues getting the logging output so I took the double-barrel approach, using both the -debug argument and the custom java.util.logging descriptor, the contents of which I stole from here. Invoke it as follows:

jconsole -debug -J"-Djava.util.logging.config.file=FILENAME"

The quotes are optional. Provide the logging descriptor filename. You can call out the JMX Service URL or the hostname:port combination at the end if you like. Now you’ll eventually a debug output much like this:

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] connecting...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] finding stub...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] connecting stub...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] getting connection...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] failed to connect: java.rmi.ConnectException: Connection refused to host: IP-ADDRESS; nested exception is:
java.net.ConnectException: Operation timed out

PORT will be the RMI registry port you’re tunneling into. IP-ADDRESS is special, we’ll get to that, and it’s important to note that it’s a ‘ConnectException‘ occurring against that host.

This debugging information can show up rather late in the whole connection process, undoubtedly because it’s an ‘Operation timed out‘ issue, so don’t be surprised if it takes a while. Fortuntely, you can also see immediate verbose feedback when you set up your ssh tunnel connection (see below).

Addressing the Randomly-Generated RMI Export Port

The first problem I chose to resolve was the one relating to the random RMI export port issue. Daniel has provided a fine example of how to implement a custom ‘pre-main’ Agent which you can use to supplant the standard JVM one. There’s his quick’n'dirty version which doesn’t address security — which is where I started. And then there’s a more full-fledged version, which I modified to be configurable.

Most importantly, it builds its JMXConnectorServer with the following service URL:

service:jmx:rmi://HOSTNAME:RMI-EXPORT-PORT/jndi/rmi://HOSTNAME:RMI-REGISTRY-PORT/jmxrmi

Traditionally, you’ll see this service URL from the client perspective, where HOSTNAME:RMI-EXPORT-PORT is not defined and you just have ‘service:jmx:rmi:///jndi/rmi://...‘. JConsole will build that sort of URL for you if you just provide HOSTNAME:RMI-REGISTRY-PORT (eg. hostname:port) when connecting.

By calling out the RMI-EXPORT-PORT in the agent’s service URL, you can affix it and tunnel to it. You can use the same port as the RMI registry; this only requires you to open one port for tunneling.

On your client / for JConsole, the HOSTNAME will probably be localhost, where you’ve opened your tunnel like so:

ssh -N -v -L9999:REMOTE-HOST:9999 REMOTE-HOST

9999 is just an example port. REMOTE-HOST is the host you’re tunneling to. You can remove the -v argument, but it’s good to have that around so that you can see the network activity. You can also use the -l argument to specify the login username on the remote host. Note that you’re opening the same port locally as you’re hitting on the server, with no offset. You’ll need to open the same port because your agent on the server is going to need to know what port to callback to itself on for RMI export, and that won’t work if you have an offset. So you might as well use the same port for both the RMI registry and RMI export, and just keep that one port available locally.

On the server in your agent, the HOSTNAME part of the service URL can either be InetAddress.getLocalHost().getHostName(), or an IP address (127.0.0.1), or in my case ‘localhost’ just worked fine.

The major reason to create the custom agent is to build the port-qualifying service URL. As usual, the example code takes a lot of shortcuts. So, I built myself a more re-usable agent — influenced by the standard JVM agent’s System.property’s — which allowed me to configure the same sorts of things as mentioned above:

• hostname : to be used as the HOSTNAME value above
• port.registry : to be used for RMI-REGISTRY-PORT
• port.export : to be used for RMI-EXPORT-PORT, defaulting to the same as port.registry if not provided
• ssl : true to enable SSL
• authenticate : true to enable credentials / authentication
• access.file : specifies the location of the user credentials file
• password.file : specifies the location of the user password file

And so I was able to configure my agent service URL for localhost, using the same port for both RMI requirements, and using simple password-based auth. I did not go down the SSL route, though many of the posts from Daniel and others explain this as well. Do that once you’ve tackled the core problem :)

Another great post relating to this issue mentions that Tomcat has a custom Listener for setting up a similar agent. The example was:

<Listener className="org.apache.catalina.mbeans.JMXAdaptorLifecycleListener" namingPort="RMI-REGISTRY-PORT" port="RMI-EXPORT-PORT" host="localhost"/>

I didn’t look any deeper into this to see whether it supports SSL and/or basic authentication. But it seems clear that this *not* a Java agent, because you have to set those up via System.property’s. Here’s what I needed to add to Tomcat startup for my custom agent:

-D___.hostname=localhost
-D___.port.registry=RMI-REGISTRY-AND-EXPORT-PORT
-D___.authenticate=true
-D___.password.file=$CATALINA_HOME/conf/jmxremote.password
-D___.access.file=$CATALINA_HOME/conf/jmxremote.access
-javaagent:PATH/TO/AGENT.jar

I’ve omitted the namespace I used for my agent, and the formal name of the ‘pre-main’-compatible JAR file that I built using the instructions that Daniel provided. Tomcat won’t start up properly until the agent is happy; after that, then you’re golden.

So I got Tomcat running, started up an ssh tunnel, and invoked JConsole. And still no matter what I did, I still got ConnectException: Operation timed out. I tried to connect via JConsole in all the following ways:

HOSTNAME:PORT
service:jmx:rmi:///jndi/rmi://HOSTNAME:PORT/jmxrmi
service:jmx:rmi://HOSTNAME:PORT/jndi/rmi://HOSTNAME:PORT/jmxrmi

All of these are valid URLs for connecting via JConsole. For a while there I wasn’t sure whether you could use the same port for both the RMI registry and export, so I could see that the JConsole log was different when I called out the RMI export info explicitly in the service URL. Still, it didn’t seem to help.

Then I started to realize that there were two separate issues going on, although they tended to blend together a lot in the posts I’d been reading.

Addressing the RMI Export Hostname

The short version is, even if you’ve set up your JMX service URL properly on the server — yes, even if you’ve set its HOSTNAME up to be ‘localhost’ — you’ll still need to tell JMX remoting which hostname the RMI export objects should use for callbacks. This requires you to provide the following System.property’s as well:

-Djava.rmi.server.hostname=localhost
-Djava.rmi.server.useLocalHostname=true

The useLocalHostname may not be relevant, but it doesn’t hurt. All this time I’d thought that because I was configuring that information in the service URL that RMI would build the objects accordingly. But I was wrong … it doesn’t … you need to call that out separately.

What was not apparent to me — until I started to see the same articles pop up when I revised my search criteria — was the IP-ADDRESS in this exception dump:

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] failed to connect: java.rmi.ConnectException: Connection refused to host: IP-ADDRESS; nested exception is:
java.net.ConnectException: Operation timed out

It was the IP address of my instance in the cloud. The callbacks were being made back to my VM, but they needed to be made to ‘localhost’ so that they could go through the tunnel that I’d opened. The ‘Operation timed out‘ was due to the port being closed, which is the whole reason you’re using ssh tunnels in the first place. Once the RMI exported objects know to use ‘localhost’, that addresses the problem. And magically, JConsole will connect and show you all the data in the world about your server.

So you must provide those System.property’s above regardless of what other configuration you’ve provided in your custom JMX agent.

Additional Concerns

There were a number of other red herrings that I followed for a while, but they were called out as being potential issues, so I kept note.

• If your server is running Linux, there are a couple of things you’ll want to check, to make sure that your /etc/hosts is self-referencing correctly, and that you’re not filtering out packets.
• You will have troubles stopping Tomcat when it has been started with your custom JMX agent; you’ll have to kill the process. Apparently agents don’t release their thread-pools very nicely. Daniel provides an example of an agent with a thread-cleaning thread — which still has some limitations, and raises the philosophical question ‘who cleans the thread-cleaner‘? He also provides an agent that can be remotely stopped — which is reasonably complex. I’ll save that one for a rainy day.
• If you want to use SSL in your authentication chain, read up on Daniel’s other postings, and use the following System.property’s on both the server and when starting JConsole:

  -Djavax.net.ssl.keyStore
  -Djavax.net.ssl.keyStorePassword
  -Djavax.net.ssl.trustStore
  -Djavax.net.ssl.trustStorePassword

• I have built some Ruby scripts which allow me to dynamically look up an AWS instance’s public DNS entry and then start up a Net::SSH process with port-forwarding tunnels. This works fine for HTTP and even for remote JVM debugging, but it did not work for JMX remoting. I’m not sure why, so you should stick with using ssh for setting up your tunnels.
• I started out this exercise using Wireshark for packet sniffing. I’m using OS X, and I installed Wireshark — the successor to Ethereal — via MacPorts. It runs under X11, which you’ll need to install from either Apple’s site or your Optional Installs DVD. I couldn’t get any Interfaces (ethernet card, etc.) to show up, until I learned that I should:

  sudo wireshark

  The app will warn you that this is unsafe, but it works. The Nominet team says that you can address this issue by providing:

  sudo chmod go+r /dev/bpf*

  However that is volatile, and has to be done whenever your Mac starts up. More config involved, so I took the easy path.

• If you’re using a script to start and stop Tomcat, you’ll need to somehow separate out the System.property’s that should be used on startup, and omit them when invoking shutdown. If you invoke shutdown with your debug and/or RMI ports specified, the shutdown will fail because those ports are already in use.

  I’m using the newest standard Tomcat RPM available for Fedora Core 8 — tomcat5-5.5.27 — and it’s uniquely nutty in terms of how it is deployed:

  /etc/init.d/tomcat5
  /usr/bin/dtomcat5
  /etc/tomcat5/*.conf

  That’s a very non-standard arrangement. The init.d script awks the *.conf files, and a whole array of other exciting things. I still haven’t gotten it to properly do an init.d restart due to how it blends the JAVA_OPTS handling. So that’s left as a case-specific exercise.

• The whole reason I went down this path was to address a memory leak relating to Hibernate sessions which I blogged about a long time ago. The fix required me to invoke Tomcat with the following System.property’s:

  -Djavax.rmi.CORBA.PortableRemoteObjectClass
  -Djava.naming.factory.initial

  The org.objectweb.carol JAR, which these settings were targeted at, is part of my weapp, so it’s available in its own Classloader. However, once I put the custom JMX agent in place, I got:

  FATAL ERROR in native method: processing of -javaagent failed
  Exception in thread "main" java.lang.reflect.InvocationTargetException
  Caused by: java.lang.ClassNotFoundException: org.objectweb.carol.jndi.spi.MultiOrbInitialContextFactory

  Attempting to create a symlink to the app-specific JAR in either common/lib, common/endorsed or shared/lib did not address the issue. I had to hack the JAR into the --classpath in order to get Tomcat to start. And yes, hack was the operative term (again).

In Summary

Frankly, all that discovery was enough for one day. And yes, it took me that long to find all of the corner-cases i was dealing with. I hope that if you find this article that it will make your path a bit easier. I know I’ll be glad that I blogged about the details the next time I bump into the issue!

Yes, you can cron if you want to. Or maybe, as the Mac children recommend, you can create Launch Daemons. It’s simply a custom launchd.plist, an XML file to define tasks in Apple’s terms. Sweet, I can do that…

The logrotated.plist Daemon

From what I could tell, out of the box, Leopard doesn’t give you logrotate. So, here it is as a Launch Daemon:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Disabled</key>
	<false/>
	<key>Label</key>
	<string>logrotate</string>

	<key>ProgramArguments</key>
	<array>
		<string>/opt/local/sbin/logrotate</string>
		<string>-f</string>
		<string>/etc/logrotate/*.conf</string>
	</array>
	<key>EnableGlobbing</key>
	<true/>

	<key>KeepAlive</key>
	<false/>
	<key>LaunchOnlyOnce</key>
	<false/>
	<key>RunAtLoad</key>
	<false/>
<!--
causes 'Throttling respawn: Will start in 10 seconds' in system.log
	<key>OnDemand</key>
	<false/>
-->

<!--
http://www.nabble.com/launchd-support-ranges-for-StartCalendarInterval - - td14857719.html
-->
	<key>StartCalendarInterval</key>
	<!-- 3:05a, 3:05p -->
	<array>
		<dict>
			<key>Hour</key>
			<integer>3</integer>
			<key>Minute</key>
			<integer>05</integer>
		</dict>
		<dict>
			<key>Hour</key>
			<integer>15</integer>
			<key>Minute</key>
			<integer>05</integer>
		</dict>
	</array>
</dict>
</plist>

Great. So what do I do with it? Name it anything ‘*.plist’ — logrotated.plist perhaps — and put it here:

• /System/Library/LaunchDaemons or
• /Library/LaunchDaemons

I tend to use System for any services I wan to launch on boot (chkconfig-ish), and anything that’s just a scheduled task is more pedestrian.

You register daemons through launchctl. When you change it, just:

$ launchctl unload FILE.plist
$ launchctl load FILE.plist

Sure, not a hard thing. What I did want to point out were some of the things that I learned:

• Provide each space-separated component of the command line as <string /> element in the ProgramArgument <array />. It’s the easiest way to go.
• I tend to keep the KeepAlive, LaunchOnlyOnce and RunAtLoad all in sync for each daemon. It seems to guarantee compatability.
• I still haven’t gotten quite the hand of OnDemand (c’mon, I’m new to this). When you are developing your daemons, keep a watch on /var/log/system.log — if you screw up, you’ll probably see it there in one fashion or another.
• StartCalendarInterval is the equivalent of the cron schedule mask. See the Nabble post i commented above. You can’t do ranges, but wildcards are do-able sorta by omission. If you want mulltiple disparate schedules, as you see above, you need to pass in an <array /> of <dict />s, not just a single <dict /> when you only need one mask.
• Yes, datatypes matter! I couldn’t figure out for the life of me why my daemon wouldn’t start, until I realized that my Hour and Minute values were configured as <string />s — vs <integer />s — because I’d been, well, you know, trying to do ranges and wildcards the cron-style way. launchctl didn’t complain, it just didn’t … work. So don’t do that.
• Any daemon which has a schedule that fires off when the system is down will be automatically executed shortly after the system re-awakens. I don’t know exactly how quickly, but I’ve seen it in action (though I’ve read some dispute about its efficacy in my searches).

The Trials and Tribulations of MacPorts

But of course this Epistle wouldn’t be any fun without a twist, right? Well, lucky us, because it wasn’t that easy.

I’m using MacPorts as my yum-my package manager. It’s great, and easy to set up, but it’s quite as stable as I’ve seen under Linux. Or rather, I’ve seen a disproportionate number of issues in the times I’ve used it. However, I’m eternally grateful because it saves me so much time … the successes far outweigh the problems.

When things do go bad, as they did when I built logrotate 3.7.7, you have to set up a custom Portfile and source repository so that you can effectively drive MacPorts to fetch and build your specific version. logrotate 3.7.1_1 turned out to be much more stable.

Techonova and Joe Homs go into very excellent and welcome detail in their posts on how to pull this off. A summary of what you need to do is:

• watch the build failure — debug with port -d — to identify where the dying source is located. That’ll be PATH/TO/PROJECT
• visit http://trac.macosforge.org/projects/macports/log/trunk/dports/PATH/TO/PROJECT and snag the Portfile. That’s a MacPorts spec file, and you can tweak it to do your bidding.
• create a source directory that you’ll keep around for a while (mine is /Users/Shared/macports).
• register that directory — your local Portfile repository — with /opt/local/etc/macports/sources.conf (using file:// protocol)
• copy the Portfile into a PATH/TO/PROJECT sub-directory structure (eg. sysutils/logrotate). Basically, the same path you’ll have snagged it from.
• pull down a previous source revision and snapshot it locally. That could come from SVN or git, tarball, whatever. Start search from the homepage setting of the Portfile.
• tweak the Portfile to ‘make sense’ for the source you’ve pulled down. If you based upon a close-enough version, it should be as easy as tweaking the version, etc. — no custom build tasks. I’m glossing over when I say that it’s usually a matter of version-based naming conventions and MD5 checksums.
• re-build the repository’s PortIndex (which you’ll need to do every time you make an addition / change)

Please feel free to consult the other two posts to fill in the details that I was so cavalier about.

In Summary

This was another one of those occasions when I was glad I kept track of what I was doing while I was in the moment. Pack your config files with comments, because they’re invaluable. And drop READMEs around where you’re likely to find them … I filled in a lot of the extra details for this post from those breadcrumbs I left for myself, things I could have easily wasted 15-20m on re-discovering by braille :)

Peace.

Building Ruby and RubyGems

Many thanks to Dan Benjamin for his excellent posting on doing a custom build of Ruby & RubyGems on OS X. I was just getting started with using a Mac at the time, and it was a very straight-forward and helpful guide. For most software, I’ll use MacPorts or standard Apple packages, but for Ruby I’ll make an exception.

For starters, I just want to expand a bit on his excellent tutorial. I won’t re-iterate his whole piece, but the outline is:

• start with OS X Leopard, and XCode (from your Optional Installs DVD)
• get /usr/local and its derivatives into your path
• set aside a good long-term home for your custom build code
• pull down the source for Ruby
• run ./configure, then build and install with make
• pull down the source for RubyGems, run its setup.rb

After that you can install all of the gems that your heart desires. I initially pulled down source for the Ruby 1.9 candidate release, but there are significant enough language changes that it destabilized a lot of tried-and-true gems. Instead, I went with the latest generation of 1.8.7.

I did however run into a little issue when installing the native mysql gem. Dan’s instructions weren’t valid for my case. Fortunately, the README.html in the gem suggested building it manually as such:

$ sudo gem install mysql -- --with-mysql-config
  or
$ ruby extconf.rb --with-mysql-config

That worked well, since I’d already installed MySQL and my /etc/mysql.cnf was in its standard location. Hooray!

Configuring RadRails to use the Custom-Built Componentry

Since I switch back and forth between Ruby and Java, I chose to stick with the Eclipse IDE. Aptana has created a nice development suite called RadRails which is highly componentized, and it works rather well for script development as well. Yes, if I don’t use TextMate I’m a heretic, but at some point I’ll hunker down and pony out the bucks for it :)

The install of Eclipse 3.4 Ganymede that I started out with contained several plugins / features from rubypeople.org. As well intended as they were, I ran into several areas of conflict between them and RadRails. I thus chose to remove these plugins from Eclipse, and stability ensued.

However, even after removed, I still have two Eclipse sections for ‘Ruby’. The one for RadRails is the one with fewer sub-sections (also known as ‘the one that looks less like the Java language configuration section’).

Since you’ve custom built your install of Ruby, you’ll want to make the following configuration changes to Eclipse / RadRails:

• Under Ruby | Interpreters, add a new Generic Ruby interpreter. Its executable should be /usr/bin/local/ruby, and the rest will take care of itself (I didn’t specify any arguments).
• Under Rails, set up the ‘rails’ and ‘mongrel_rails’ paths to reference the respective binaries in /usr/local/bin, since that’s where all of your installed gems will have been exposed.
• Under Java | Build Path | Classpath Variables, you’ll want to add GEM_LIB, which should reference /usr/local/lib/ruby/gems/1.8/gems. It seems out of place under ‘Java’, but the IDE was crying out for it loudly.

That should address the majority of the IDE configuration. If anything doesn’t quite work correctly, you may want to examine the steps that I describe below as being ‘optional’, particularly the one where I created a mock Ruby.framework setup.

Optional (?) Configuration

I question whether they’re mandatory or not because this configuration has been stable on my machine for a little while now, and I have a tendency to get very detail-oriented, not all of which turns out to actually be necessary. In the spirit of completeness — yep, there’s detail-oriented for you right there! — I’ll lay out what I have put into play.

I’ve injected ‘/usr/local/bin’ and ‘/usr/local/sbin’ into /etc/paths; that works like a charm.

I consistently install gems as root, or else I end up with account-specific discrepancies driven by each ~/.gems folder. I merely point that out to illustrate that such user-specific directories exists :)

I’ve added the following unnecessary env var:

export GEM_LIB=/usr/local/lib/ruby/gems/1.8/gems

It matches the Classpath Variable you set in Eclipse (above). If nothing else, it’s an easy ‘$’ substitution when I need to go spelunking into gem source code.

There’s been at least once where I have also needed the following env var. It should be configured to point at the expanded source code of your custom Ruby build.

export RUBY_SOURCE_DIR=...

I’ve created the following directory structure:

$ mkdir -p /System/Library/Frameworks/Ruby.framework/Versions/1.8-Current

Therein I have created a ‘usr’ directory structure that matches the one you’ll find in /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr . It’s all just symlinks into the appropriate /usr/local hierarchy. I then co-opted the Current symlink:

$ cd /System/Library/Frameworks/Ruby.framework/Versions
$ ls -l
1.8
1.8-Current
Current -> 1.8
$ rm Current
$ ln -s 1.8-Current Current

Consider that you may need to mimic certain aspects of the default Ruby framework configuration, which is the inverse of the mock framework setup that I just described. Specifically, /usr/local/lib/ruby takes after /System/Library/Frameworks/Ruby.framework/Versions/Current/usr/lib/ruby.

This is why I ended up with both a ‘/usr/local/lib/ruby/site_ruby’ and ‘/usr/local/lib/ruby/vendor_ruby’ subdir on my machine.

The standard location for the ruby executable is often used in a she-bang line for bash scripts. The standard location is:

#!/usr/bin/ruby

Which of course was still pointing at the default OS X Ruby framework. I renamed the existing symlink and replaced it with one to /usr/local/bin/ruby. I did not do this for gem, irb, rdoc (etc.) since I don’t have she-bang needs for them.

In Summary

I hope some of this information is useful to you. I was sitting at Cafe International and setting up RadRails when the HD on my two-week-old MacBook Pro 2008 went into beachball-of-death mode. I had to pick back up from there 2+ weeks later on a fresh laptop, which at least allowed me to keep these copious notes.

When you unpack the installer .dmg, it’s simply going to install the package. It will not automatically assume that you want to use it (not such a bad assumption, as it turns out). So there are a few caveats you need to deal with. I’m taking the approach of addressing all the details I know of, regardless of how obvious this may all seem :)

Post-install, Java 5 will still be in active use:

$ ls -l /usr/bin/java
/usr/bin/java -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java
$ Lunditer:~ root# java -version
java version "1.5.0_16"

You’ll see something much like this under /System/Library/Frameworks/JavaVM.framework/Versions :

1.3 -> 1.3.1
1.3.1
1.4 -> 1.4.2
1.4.1 -> 1.4
1.4.2
1.5 -> 1.5.0
1.5.0
1.6.0
A
Current -> A
CurrentJDK -> 1.5

Initially, I went and modified the symlink for CurrentJDK. Modifying the symlink for Current brought me nothing but pain. You can get the results you want by making the following change:

CurrentJDK -> 1.6.0

I’d had it set up that way until I wrote this post. But Apple highly discourages this practice. Instead, they recommend that you use the Java Preferences pane. Get rid of all your symlink shenanigans, and simply drag the ‘Java SE 6′ option to the top of the applet and application listings. It works like a charm. Start a new Terminal shell and you can confirm:

$ /usr/bin/java -version
java version "1.6.0_07"
$ /usr/bin/javac -version
javac 1.6.0_07

However … from my experience, this is a user-specific setting. Which means if you run Java as root, you’ll still get Java SE 1.5. Not a problem. We’ll stick with the Apple-sanctioned solution here.

Hooray, Java SE 6! Now let’s start Eclipse 3.4! Hooray … oops? Startup failure dialog:

JVM terminated.
Exit code=-1
...
-vm /System/Library/Frameworks/JavaVM.framework
...

This is a well documented issue. Both the Rob Kischuk and Stack Overflow blogs are very helpful in explaining that; Eclipse uses 32-bit SWT-Cocoa / Carbon, and Mac OS Java SE 6 only comes in a 64-bit flavor. I have not looked into the SoyLatte option, as suggested above, though that may prove to be an alternative approach.

I had some struggles getting their approach to work (until now, that is). Easiest way to start config testing is to run Eclipse from the command line:

$ cd /Applications/eclipse/Eclipse.app/Contents/MacOS/
$ ./eclipse -vm /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Commands/java

You can also provide /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0, which I extrapolated from the -vm setting in the failure message. If you still get an error dialog, you may see multiple instances of -vm, but that’s a red herring; it will work once properly configured.

Great. Once that’s working for you, let’s apply it to the App shortcut. There’s two ways you can go about this:

The blogs above recommend that you modify the Info.plist:

• edit /Applications/eclipse/Eclipse.app/Contents/Info.plist
• uncomment the following line:

  <string>-vm</string><string>/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Commands/java</string>

Alternately, you can modify eclipse.ini, where you might also configure your JVM memory configuration:

• edit /Applications/eclipse/Eclipse.app/Contents/MacOS/eclipse.ini
• add the following line:

  -vm /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Commands/java

Either dab will do ya. But if you didn’t know before, now you know where eclipse.ini lives :)

Things should be grand now. If they aren’t, I suggest trying some variations while running Eclipse from the command line. It took me a while to get the right combination, even though this post and (the ones it references) may seem very cookie-cutter. It will eventually work for you!

Now, I mentioned that the Apple-sanctioned approach seems to be user-specific. And I run Tomcat as root. So you will probably still want to add the traditional environment settings.

You can either make these changes to /etc/profile, or to the ~/.bashrc of each relevant account. No surprises here:

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
PATH="$JAVA_HOME/bin:$PATH"

My experiments to get /etc/paths to inject something into the path before ‘/usr/bin’ weren’t so useful, so I chose the brute force method.

If you’re going to use sudo to run Tomcat, you’ll want to make sure to load the bash environment.

sudo su -

to guarantee that root’s environment includes the custom /etc/profile overrides. No dash, and it no work. Tomcat Java 5 + App Java 6 =

SEVERE: Error deploying web application directory ...
java.lang.UnsupportedClassVersionError: Bad version number in .class file

With the environment in place, you should be good to go.

The writing of this post drove me to (a) drop the symlinks, (b) use Java Preferences pane and (c) get the right -vm config set up (since I’d been leaning entirely on the environment settings up to this point). So there are definitely a few ways to skin this cat.