Remote Scripting for AWS

When signing up for Amazon Web Services, you end up generating all of the following lovely identifying information:

Your Account Number
An Access Key
A Secret Access Key
An SSL Certificate file
The Private Key for your SSL Cert

And the various AWS command APIs and tools will require you to provide one more more of these secret-y pieces of information. But I’d rather not host all of this information on my instances. A connundrum indeed.

I chose to solve it with a Ruby toolkit using:

Glenn Rempe’s Amazon EC2 gem (amazon-ec2)
The net-ssh and net-scp gem suite

My needs are small-scale for the moment, so I can keep a hard-coded list of my instance IDs. The EC2 API allows me to fetch instance metadata, identify their public DNS (plus everything else), and

open up an ssh connection,
push data into a Channel’s environment,
or upload files for bundling purposes

I learned a lot.

Channels

It looked like a good plan to do most of the remoting over an Net::SSH::Connection::Channel, and sometimes asynchronously (as in the case of my Health Check script).

There are a lot of library shorthands – this “Hello World” example works great, and it is blissfully unaware of Channels:

Net::SSH.start("localhost", "user") do |ssh|
    # 'ssh' is an instance of Net::SSH::Connection::Session
    ssh.exec! "echo Hello World"
end

But as with all examples, you soon find that surface-level tools won’t quite do the trick. The main thing you’ll need to have all the time is convenient access to the data come back from the Channel, asynchronously or otherwise. So create yourself a simple observer:

def observer_struct @observer_struct ||= Struct.new(:channel, :stdout, :stderr, :exit_status, :exit_signal) end def observe_channel(ch, ops={}) observer = observer_struct.new(ch, '', '', 0, nil) is_puts = !!ops[:puts] # stdout ch.on_data do |ch, data| $stdout.print data if is_puts observer.stdout << data end # only handle stderr ch.on_extended_data do |ch, type, data| next unless [1, :stderr].include? type $stderr.print data if is_puts observer.stderr << data end ch.on_request("exit-status") do |ch, data| observer.exit_status = data.read_long end ch.on_request("exit-signal") do |ch, data| observer.exit_signal = data.read_long end observer end def open_channel(ssh, ops={}, &block) ch = if block_given? ssh.open_channel &block else ssh.open_channel end # somebody's watching you self.observe_channel ch, ops end

Then during and after executions, you’ll have access to all the core information you’ll need to make conditional decisions, etc.

Pushing Your Environment

Many of the AWS tools will recognize specific environment variables. The usual suspects are:

AMAZON_ACCESS_KEY_ID
AMAZON_SECRET_ACCESS_KEY

So, you can use Net::SSH::Connection::Channel.env to inject key-value pairs into your connection. However, you’ll need to make some config changes first – and major thanks to the Net::SSH documentation for clarifying this in its own RDoc:

“Modify /etc/ssh/sshd_config so that it includes an AcceptEnv for each variable you intend to push over the wire”

# accept EC2 config
AcceptEnv  AMAZON_ACCESS_KEY_ID AMAZON_SECRET_ACCESS_KEY
...

After you make your changes, make sure to bounce the ssh daemon:

% /etc/init.d/sshd restart

Then your selected environment will shine through.

Running sudo Remotely

It’s reasonable to execute root-level commands using sudo, because you can provide a good amount of granular control.

The first thing that we’ll all run into is:

sudo: sorry, you must have a tty to run sudo

Fortunately, there’s Net::SSH::Connection::Channel.request_pty:

ch.request_pty do |ch, success|
    raise "could not start a pseudo-tty" unless success

    #   full EC2 environment
    ###ch.env 'key', 'value'
    ###...

    ch.exec 'sudo echo Hello 1337' do |ch, success|
        raise "could not exec against a pseudo-tty" unless success
    end
end

I’ve taken the approach of allowing NOPASSWD execution for everything I do remotely, after making darn sure that I had constrained exactly what could be done under those auspices (HINT: pounds of caution, tons of prevention).

You can configure all of this by editing /etc/sudoers:

# EC2 tool permissions
username  ALL=(ALL)  NOPASSWD: SETENV: /path/to/scripts/*.rb
...

Also, if you intend to have those scripts consume the environment variables that you’ve injected into your Channel, you’ll also need to annotate the line with SETENV, or they won’t export across into your sudo execution.

If you are more security minded, then there’s many other variations you can play with /etc/sudoers. I haven’t yet experimented within pushing a sudo password across the wire, but that may be where Net::SSH::Connection::Channel.send_data comes into play.

Port Forwarding

I wanted to easily do SSH tunneling / port-forwarding against an instance, referenced by my local shorthand. So I wrote my usual EC2 instance ID lookup and started an Net::SSH connection. Here’s how you put yourself in ‘dumb forward’ mode:

def ports, offset = [], 0 # ... Net::SSH.start(hostname, user, :forward_agent => true, :verbose => :warn) do |ssh| ports.each do |port| ssh.forward.local port + offset, 'localhost', port end # loop, with an event loop every 0.1s, until Ctrl-C is pressed puts "[Ctrl-C] to terminate..." interrupted = false trap('INT') { interrupted = true } ssh.loop(0.1) { not interrupted } ports.each do |port| ssh.forward.cancel_local port + offset end end

And then [Ctrl-C] will break you out.

Inconsistent EC2 Marshalling

I’d gotten all of my EC2 data parsing working great, then I started to expand my capabilities with the Amazon S3 API gem (aws-s3). Suddenly and magically, the results coming back from my EC2 queries had taken on a new form. Moreover, this was not consistent behavior across all operating systems; at least I do not see it on WinXP, under which my health monitoring checks execute.

The Amazon APIs are SOAP-driven, so I guessed that amazon-ec2 (and aws-3) would both leverage soap4r. Well, that is in fact not the case; Glenn has commented below on this known issue. He uses the HTTP Query API, and the cause of the discrepancy are (as of this writing) still undetermined.

Regardless, our job is to get things to work properly. So for the meanwhile, let’s use a quick & dirty work-around. The two core differences are:

the data coming back may be wrapped in an additional key-value pair
single value result-sets come back as an Object (vs. an Array with a single Object)

So I crafted a little helper singleton, which which still allows me to detect between ‘valid’ and ‘empty’ requests (nil vs []):

class SOAPHelper class << self def data_from(data, wrapper) data[wrapper] || data end def items_from(set) unless set && set.item nil else items = set.item items = [items] unless Array === items items end end end end data = SOAPHelper.data_from( @ec2.describe_instances, 'DescribeInstancesResponse' ) reservations = SOAPHelper.items_from(data.reservationSet) || [] ...

That will address our immediate needs for the moment, until a benificent coding Samaritan comes along.

NOTE: If your screen reader is reading this, please contact me at admin@cantremember.com ... because it shouldn't. FIXME: build this dynamically based upon the maximum content in any sub-Element of this Element. I will call this my "Safari Reader Counterweight". In some of my Posts, I have huge code excerpts, etc. Safari Reader, at least in iOS, will identify the 'main Element', the one it features, based upon its content length. Sometimes those code excerpts get identified as the 'main Element', and the Post is borked in Safari Reader mode. This is a counter-weight; it gives the <main> Element additional content so that it gets featured, algorithmically. Yes, it increases the payload of every page (@see FIXME above). But not by that much. Then again, this is a guess as to how much content any given Element could contain. If it's not enough, BOOM, Safari Reader looks like crap. So, here's a great article on how to enable Safari Reader on your site. It's mostly guesswork, but those guesses helped me debug this obtuse goddamn problem. Oh, and look, you can enter and exit Reader programmatically. JavaScript can fix anything. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload. I promise I will never cut-and-paste lines of text simply to add Element payload.