Posts Tagged ‘ruby’

Twitter4R shifts RSpec onto my Front Burner

Thursday, February 5th, 2009

As usual, my day did not pan out as expected. But, also as usual, I learned a lot!

Coming Up To Speed on Rspec

So, learning rspec has been on my list for a while. I finally got around to it. Nice framework. I am familiar with EasyMock, and am aware of JMock. I never had an opportunity to get into Mockito, but I’d give it a glance next time I put on my Java hat.

There is some decent documentation on it out there. I found these links particularly helpful:

During my ramping-up, I took the usual meta-approach of creating a test suite — and yaaay, that’s what rspec is meant for! — which I then used to test out its own range of capabilities. The Modules in the rdoc which I’ve found to provide the most value are:

  • Spec::Expectations::ObjectExpectations for conditionals (eg. should & should_not)
  • Spec::Matchers for expectations (eg. equal(value), be_a(class), respond_to(method_sym), raise_error)
  • Spec::Mocks::Methods for mock method definition (eg. should_receive(method_sym))
  • Spec::Mocks::MessageExpectation for mock behaviour (eg. with(*args), once, exactly(n).times, any_number_of_times)
  • Spec::Mocks::ArgumentConstraints for mock arguments (eg. an_instance_of(class), anything)
  • Spec::Mocks::BaseExpectation for mock responses (eg. and_return(value), and_yield(&block))

I won’t got into the deep details, but here are some examples of conditionals and expectations that I paraphrased into my meta-test:

And here’s a little bit of silly mocking:

These are just ways I thought of to exercise the width and breadth of the library. Very nice. I hope that these are useful examples for people new to this gem.

I recommend the other references from above for filling in the missing details. Once you have a context / specify or a describe / it specification set up, you’ll be good to go. There’s much more to the library — Stories, for instance — but that’s for another day.

An Informative Walk through Twitter4R

No, I didn’t really want to spend time learning rspec — I mean heck, I’m busy — but I had a personal need to expand the Twitter4R gem. Specifically, I wanted to add on some Twitter search features, and I was very impressed with how this library has been built. Contribution-wise, the final step that Susan Potter recommends is to craft up some rspec tests.

Of course, mock testing is only as good as the framework you’re built upon. The assumption is that Net::HTTP is going to do it’s job, so mock it up and you can even test your Twitter features offline. When I built Bitly4R (given that name, my thinking has clearly been influenced), I did everything as full-on functional tests. It was easy; bit.ly has both shorten and expand commands, so I could reverse-test real values without having any fixed expectations.

However, Twitter is live and user-generated, so who knows what you’ll find. Mocking covers that for you. And of course not having to hit the service itself shortest testing time dramatically.

Here’s one of my tests, again foreshortened:

Plus a little mocking of Net::HTTPResponse:

Nothing like having some good inspiration to get you thinking about maintainability. I’m a not a strong adherent to any of the TDD or BDD denominations, but testability itself is close to my heart. Just ask the folks who’ll be looking at the micro-assertion Module that I wrote for my Facebook Puzzle submissions (which work locally under test conditions but fail miserably once thrown over their wall). Then again, I won’t need to write that (and test that) from scratch again.

So, back to talking glowingly about the Twitter4R architecture. Yes, I used that word. Yet regardless of that term’s heavyweightedness, it’s simply the result of carefully thinking out the formalization of a complex design. And once a non-author delves into extending an existing implementation, that impl’s demeanor becomes readily clear :)

Some of the interesting things I saw:

  • An inner bless-ish strategy, to inject the Twitter::Client into instanciated result classes. I’ve seen blessing as an OO-Perl-ism, for self-ication, and that metaphor carries over very nicely to this context.
  • Generous use of blocks / lambdas / closures in methods, for contextual iteration (eg. iterate through each Twitter::Message in a timeline response). Unnecessary, but an excellent convenience for a language that offers that optional capability in a very disposable fashion (from the caller’s perspective).
  • Retroactive sub-object population after construction. Twitter4R relies upon the json gem, which deals primarily in Hashes. Post-injection, the library itereates through Arrays of Hashes and transforms them into suitable Twiltter::User containers, etc. A great place to put such logic in the construction chain, and it doesn’t take long to get really tired of Hash hierarchies.

Good stuff, and learning with in a Ruby-centric mindset was invaluable for me. We all have to start somewhere, eh.

The Acute Long-Term Pain of Staticness

There was one issue that I ran into; static configuration. During my years of using the Spring Framework, my IOC-addled brain started thinking of everything in terms of instances — Factory Beans, POJOs, Observers & Listeners, Chain-of-Responsibility wrappers. Static configuration is a common metaphor, and in this case, there’s a static Twitter::Config instance. Convenient and centralized. Makes perfect sense.

I mean, the fact that it was a configurable library at all was awesome. I was able to easily set up a Twitter::Client to reference search.twitter.com. However, of course as soon as I did that, I whacked the ability for clients to talk to twitter.com in the process. Oops!

On GPs, I refused to modify the original code. And I wanted to make sure that my superficial tweaks to the library would be thread-safe — temporarily swaping out the global Twitter::Config in mid-operation would be an issue. Using Mutex.synchronize seemed like the perfect choice. After finding that the same thread can’t lock a Mutex instance twice — grr!, that’s a great trick if you can work it — I overrode the one method that cared the most about @@config:

It works like a charm. Please, everyone just line up to tell me how I could have done it better. And I don’t mean quicker or cheaper, I mean better. Believe me, I would not have sunk the time into this end-around approach, if not for the fact that:

  1. I don’t want to maintain a local gem code modification (even though my impl is closely coupled to the gem impl already)
  2. I intend to follow that practice, so every opportunity to pull and end-around is a Valuable Learning Experience.

So, now my local Twitter4R has search capability gracefully latched onto it (and implemented much in the flavor of the library itself). I have a mass of rspec examples to work off in the future.

Now, I haven’t spent a great amount of time testing the thread-safeness — no one in their right mind wants to do that — but my sundry Twitter::Client instances play nicely together in between searches and normal status operations.

And I had something useful to blog about!

Remote Scripting for AWS

Saturday, January 24th, 2009

When signing up for Amazon Web Services, you end up generating all of the following identifying information:

  • Your Account Number
  • An Access Key
  • A Secret Access Key
  • An SSL Certificate file
  • The Private Key for your SSL Cert

The various AWS command APIs and tools will require you to provide one more more of these pieces of information. Rather than actually host all of this information on my instances, I have instead chosen to build Ruby scripts using:

I’m small-scale for the moment, so I can keep a centralized list of my instance IDs. Without too much effort, I can look up the instance(s), identify their public DNS (plus everything else), and then open up an ssh connection and push data into a channel’s environment or upload files for bundling purposes.

Here’s a few things that I’ve learned in the process.

Channels

Most of your core work gets done on a Net::SSH::Connection::Channel, and sometimes asynchronously (as in the case of my Health Check script). There are a lot of library shorthands — the prototypical Hello World example is channel-less:

Net::SSH.start("localhost", "user") do |ssh|
	# 'ssh' is an instance of Net::SSH::Connection::Session
	ssh.exec! "echo Hello World"
end

But as with all examples, you soon find that surface-level tools won’t quite do the trick. The main thing you’ll need to have all the time is convenient access to the data come back from the channel, asynchronously or otherwise. So create yourself a simple observer:

Then during and after executions, you’ll have access to all the core information you’ll need to make conditional decisions, etc.

Pushing Your Environment

Many of the AWS tools will recognize specific environment variables. The usual suspects are:

  • AMAZON_ACCESS_KEY_ID
  • AMAZON_SECRET_ACCESS_KEY

So, you can use Net::SSH::Connection::Channel.env to inject key-value pairs into your connection. However, you’ll need to make some config changes first — and major thanks to the Net::SSH documentation for clarifying this in its own RDoc:

“Modify /etc/ssh/sshd_config so that it includes an AcceptEnv for each variable you intend to push over the wire”

#       accept EC2 config
AcceptEnv  AMAZON_ACCESS_KEY_ID AMAZON_SECRET_ACCESS_KEY
...

After you make your changes, make sure to bounce the ssh daemon:

/etc/init.d/sshd restart

Then your selected environment will shine through.

Running sudo Remotely

It’s reasonable to execute root-level commands using sudo, because you can provide a good amount of granular control.

The first thing that we’ll all run into is:

sudo: sorry, you must have a tty to run sudo

Fortunately, there’s Net::SSH::Connection::Channel.request_pty:

ch.request_pty do |ch, success|
	raise "could not start a pseudo-tty" unless success

	#	full EC2 environment
	###ch.env 'key', 'value'
	###...

	ch.exec 'sudo echo Hello 1337' do |ch, success|
		raise "could not exec against a pseudo-tty" unless success
	end
end

I’ve taken the approach of allowing NOPASSWD execution for everything I do remotely, after making darn sure that I had constrained exactly what could be done under those auspices (HINT: pounds of caution, tons of prevention). You can configure all of this by editing /etc/sudoers:

# EC2 tool permissions
username  ALL=(ALL)  NOPASSWD: SETENV: /path/to/scripts/*.rb
...

Also, if you intend to have those scripts consume the environment variables that you’ve injected into your channel, you’ll also need to annotate the line with SETENV, or they won’t export across into your sudo execution.

If you are more security minded, then there’s many other variations you can play with /etc/sudoers. I haven’t yet experimented within pushing a sudo password across the wire, but that may be where Net::SSH::Connection::Channel.send_data comes into play.

Port Forwarding

I wanted to easily do SSH tunneling / port-forwarding against an instance, referenced by my local shorthand. So I wrote my usual EC2 instance ID lookup and started an Net::SSH connection. Here’s how you put yourself in ‘dumb forward’ mode:

And then [Ctrl-C] will break you out.

Inconsistent EC2 Marshalling

I’d gotten all of my EC2 data parsing working great, then I started to expand my capabilities with the Amazon S3 API gem (aws-s3). Suddenly and magically, the results coming back from my EC2 queries had taken on a new form. Moreover, this was not consistent behavior across all operating systems; at least I do not see it on WinXP, under which my health monitoring checks execute.

The Amazon APIs are SOAP-driven, so I guessed that amazon-ec2 (and aws-3) would both leverage soap4r. Well, that is in fact not the case; Glenn has commented below on this known issue. He uses the HTTP Query API, and the cause of the discrepancy are (as of this writing) still undetermined.

Regardless, our job is to get things to work properly. So for the meanwhile, let’s use a quick & dirty work-around. The two core differences are:

  • the data coming back may be wrapped in an additional key-value pair
  • single value result-sets come back as an Object (vs. an Array with a single Object)

So I crafted a little helper singleton, which which still allows me to detect between ‘valid’ and ’empty’ requests (nil vs []):

That will address our immediate needs for the moment, until a benificent coding Samaritan comes along.

Learning Ruby through Assertions and Podcasts

Thursday, January 22nd, 2009

I’ve been working with the Ruby language since March 2008. So (as of this writing) I’m still on the n00b path.

Assertions

The first thing I did was to follow the great advice of Dierk Koenig, writer of Groovy in Action and general Groovy / Grails advocate. The book itself doesn’t use the typical print-the-result-based code examples; it encourages the reader to learn the language through assertions. And that’s how I learned Groovy; I took the examples from the book, paraphrased them, tried variations on a theme, and then asserted that my results were true. Now when I need to know how to use a language feature, I simply look back at my assertion code to re-edjumicate myself.

I learned the core Ruby language via Test::Unit. I spent three weeks (please don’t laugh) worth of my daily commute writing assertions for the core language, the standard packages, plus ActiveRecord and other common gems. It allowed me to get a handle on the concepts, syntax, semantics and the sheer range of capabilities of he language. I frequently look back at my language_test_unit.rb to figure out the best use of Array.collect, catch..throw, Regexp quirks, and using declared lambdas as blocks (etc). More importantly, I’ve already written code using all of those techniques, so it’s just a referesher.

I cannot recommend this technique enough for coming up to speed on a language!

Podcasts

With that under my belt, plus some command-line scripts and a solid Rails project, I’m not spending time back-consuming posts from the following Podcast blogs:

Rubyology

Site : Feed

I’m actively back-consuming a lot of content from this wealth that Chris Matthieu has provided. There are some great talks on Journeta, using EC2, great tutorials covering basic and RoR, and some scaling recommendations.

sd.rb Podcast

Site : Feed

Straight from the mouth of the San Diego Ruby Users group. A good variety of topics, focusing more on the Ruby language than on the Rails poster-child itself. Nice talks on rspec, MySQL clustering and Arduino, amongst many others.

Railscasts

Site : Feed

With 145 postings and counting, there’s a lot to be consumed here. However, this is the last on my list, because none of them download to my iPhone 3G :( . Lots of cross coverage on Capistrano, Partials, custom Routes, ruby-debug … the list goes on.

Ruby on Rails Podcast

Site : Feed

Geoffrey Grosenbach’s podcasts are seminal. I’ll leave it up to the reader to pore through the years of accumulated wisdom. How can you go wrong when you’re part of the rubyonrails.org domain!

In Summary

A number of these feeds provide screencasts and/or video. A few of the files are old-school QuickTime MOVs which are problematic for the iPhone, which is annoying (definitely not the podcaster’s fault… get your head in gear, Apple). And unfortunately when I break away to write down something in Evernote.app, and there’s any visuals associated with the cast, the iPhone halts playback. Grr. So I’m getting into the archaic habit of creating a Notes page and mailing it to myself :)

I recommend each and all of these podcasts. Be prepared to sink a lot of time into them, so you might as well upload them onto your iPhone and take them to the beach!

Custom Ruby and RadRails on Mac OS X

Tuesday, January 20th, 2009

Let’s break this down, shall we …

Building Ruby and RubyGems

Many thanks to Dan Benjamin for his excellent posting on doing a custom build of Ruby & RubyGems on OS X. I was just getting started with using a Mac at the time, and it was a very straight-forward and helpful guide. For most software, I’ll use MacPorts or standard Apple packages, but for Ruby I’ll make an exception.

For starters, I just want to expand a bit on his excellent tutorial. I won’t re-iterate his whole piece, but the outline is:

  • start with OS X Leopard, and XCode (from your Optional Installs DVD)
  • get /usr/local and its derivatives into your path
  • set aside a good long-term home for your custom build code
  • pull down the source for Ruby
  • run ./configure, then build and install with make
  • pull down the source for RubyGems, run its setup.rb

After that you can install all of the gems that your heart desires. I initially pulled down source for the Ruby 1.9 candidate release, but there are significant enough language changes that it destabilized a lot of tried-and-true gems. Instead, I went with the latest generation of 1.8.7.

I did however run into a little issue when installing the native mysql gem. Dan’s instructions weren’t valid for my case. Fortunately, the README.html in the gem suggested building it manually as such:

$ sudo gem install mysql -- --with-mysql-config
  or
$ ruby extconf.rb --with-mysql-config

That worked well, since I’d already installed MySQL and my /etc/mysql.cnf was in its standard location. Hooray!

Configuring RadRails to use the Custom-Built Componentry

Since I switch back and forth between Ruby and Java, I chose to stick with the Eclipse IDE. Aptana has created a nice development suite called RadRails which is highly componentized, and it works rather well for script development as well. Yes, if I don’t use TextMate I’m a heretic, but at some point I’ll hunker down and pony out the bucks for it :)

The install of Eclipse 3.4 Ganymede that I started out with contained several plugins / features from rubypeople.org. As well intended as they were, I ran into several areas of conflict between them and RadRails. I thus chose to remove these plugins from Eclipse, and stability ensued.

However, even after removed, I still have two Eclipse sections for ‘Ruby’. The one for RadRails is the one with fewer sub-sections (also known as ‘the one that looks less like the Java language configuration section’).

Since you’ve custom built your install of Ruby, you’ll want to make the following configuration changes to Eclipse / RadRails:

  • Under Ruby | Interpreters, add a new Generic Ruby interpreter. Its executable should be /usr/bin/local/ruby, and the rest will take care of itself (I didn’t specify any arguments).
  • Under Rails, set up the ‘rails’ and ‘mongrel_rails’ paths to reference the respective binaries in /usr/local/bin, since that’s where all of your installed gems will have been exposed.
  • Under Java | Build Path | Classpath Variables, you’ll want to add GEM_LIB, which should reference /usr/local/lib/ruby/gems/1.8/gems. It seems out of place under ‘Java’, but the IDE was crying out for it loudly.

That should address the majority of the IDE configuration. If anything doesn’t quite work correctly, you may want to examine the steps that I describe below as being ‘optional’, particularly the one where I created a mock Ruby.framework setup.

Optional (?) Configuration

I question whether they’re mandatory or not because this configuration has been stable on my machine for a little while now, and I have a tendency to get very detail-oriented, not all of which turns out to actually be necessary. In the spirit of completeness — yep, there’s detail-oriented for you right there! — I’ll lay out what I have put into play.

I’ve injected ‘/usr/local/bin’ and ‘/usr/local/sbin’ into /etc/paths; that works like a charm.

I consistently install gems as root, or else I end up with account-specific discrepancies driven by each ~/.gems folder. I merely point that out to illustrate that such user-specific directories exists :)

I’ve added the following unnecessary env var:

export GEM_LIB=/usr/local/lib/ruby/gems/1.8/gems

It matches the Classpath Variable you set in Eclipse (above). If nothing else, it’s an easy ‘$’ substitution when I need to go spelunking into gem source code.

There’s been at least once where I have also needed the following env var. It should be configured to point at the expanded source code of your custom Ruby build.

export RUBY_SOURCE_DIR=...

I’ve created the following directory structure:

$ mkdir -p /System/Library/Frameworks/Ruby.framework/Versions/1.8-Current

Therein I have created a ‘usr’ directory structure that matches the one you’ll find in /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr . It’s all just symlinks into the appropriate /usr/local hierarchy. I then co-opted the Current symlink:

$ cd /System/Library/Frameworks/Ruby.framework/Versions
$ ls -l
1.8
1.8-Current
Current -> 1.8
$ rm Current
$ ln -s 1.8-Current Current

Consider that you may need to mimic certain aspects of the default Ruby framework configuration, which is the inverse of the mock framework setup that I just described. Specifically, /usr/local/lib/ruby takes after /System/Library/Frameworks/Ruby.framework/Versions/Current/usr/lib/ruby.

This is why I ended up with both a ‘/usr/local/lib/ruby/site_ruby’ and ‘/usr/local/lib/ruby/vendor_ruby’ subdir on my machine.

The standard location for the ruby executable is often used in a she-bang line for bash scripts. The standard location is:

#!/usr/bin/ruby

Which of course was still pointing at the default OS X Ruby framework. I renamed the existing symlink and replaced it with one to /usr/local/bin/ruby. I did not do this for gem, irb, rdoc (etc.) since I don’t have she-bang needs for them.

In Summary

I hope some of this information is useful to you. I was sitting at Cafe International and setting up RadRails when the HD on my two-week-old MacBook Pro 2008 went into beachball-of-death mode. I had to pick back up from there 2+ weeks later on a fresh laptop, which at least allowed me to keep these copious notes.