CantRemembrances

okay, this is what happens when you rush forward into things …

i ponied up cash for the Apple Developer License a while back. and lo & behold, developers can download the DMG for the 3.0 Beta firmware update from the iPhone Dev Center! so i went and installed iPhone OS 3.0 Beta tonight … i wanted to check out the cut’n'paste capabilities, etc. downloaded and extracted the firmware update package, started iTunes, held down [Ctrl+Option] when clicking ‘Check for Update’ to bring up the file selector dialog, and installed the 3.0 Beta IPSW

great! hooray! it’s the usual slow process. however, once the device restarts, it goes into pre-activation mode … and iTunes rejects it as not behing a registered development device. open up the Device Management Portal, and it tells you how about Locating a Unique Device ID … “The 40 hex character string in the Identifier field is your device’s UDID.” i believe they refer to it as an ICCID in other places

alright, where can one find this magical string? well, it’s shown when your device is connected to iTunes … but of course, that’s the problem! it already won’t accept my device = meta-problem. so, then i read further … “Please DO NOT install the iPhone OS before registering device UDIDs, as installation on non-registered devices will render them inoperable.”

so i figured that i’m toast. i can easily find the Serial Number and IMEI, but not the UDID, at least according to Apple’s instructions

making a long story short … look at System Profiler (if you haven’t already) … under USB :: USB High Speed Bus :: iPhone :: Serial Number. register that as a developer device, and you’re good to go as soon as you re-boot the phone & iTunes. there’s a much more elaborate description of the process using Xcode, but it doesn’t seem to be mandatory cause i sure ain’t done that

and yes … cut’n'paste is really nice!

This sure was an interesting morning! I woke up to find that I’d unintentionally sent direct messages to all of the followers on my personal Twitter account. And I’d sent them out at 1a PST, which means that anyone who (a) uses SMS capabilities, and (b) has some text message notification sound set up would have been rudely interrupted in the middle of the night.

Fortunately, I haven’t lost any followers (yet). But this was a perfect case of how mixing business with pleasure can have unintended consequences.

What Have I Learned

Or rather, what have I re-learned …

Soft-disable features in Production at Launch Time

 
My Twitter engines are built with both an :enable_tweet and :enable_greeting config setting. In the git repo, they’re both true. When I did my local testing, I’d disabled them correctly. When I launched in Production, I neglected to make the quick-and-dirty changes; after all, everything worked great. And once started, my scripts correctly responded to the no-initial-state condition, and greeted everybody.

Launch preparation is critical, even for little projects. The start-up mentality is to move fast and lean, but there’s such as thing as too fast, and probably as too lean too. Gradual uptake migration is a wise strategy even for the ‘little things’.

Mock and Integration Testing Only Gets You So Far

 
I used rspec to mock out the full capabilities of the engine. Found some real-world issues, resolved them. I also wrote some core integration tests, ran them locally. Immediate failures. I had mocked documented features that didn’t actually exist. Fixed, re-mocked, re-tested, fixed again, etc .

Another great reminder that you can only mock something you trust, and how can you trust something you haven’t actually run under integration conditions to start with! Re-tested integration, and everything passed with flying colors. Sure, the features worked great now! And when I launched them, they did exactly what I asked.

So, as if we haven’t heard it enough times, be careful what you ask for!

All of this is familiar to anyone who has made a mistake in the software industry. It’s not like I haven’t successfully executed dozens of critical launches in the past, and most with virtually no issues at all. But what’s interesting is what happens when these mistakes happen in a public forum, and whom you expose them to — say, your friends :)

And who can say when two ounces of caution is more deserving than one … without the benefit of hindsight.

Just ask anyone who has a stringent backup policy how much time & effort they invest to avoid an event that may never actually happen. That stringency usually comes from that one unforgettable experience, and from there is born an extra layer of caution, and an additional time-sink (eg. mock & integration testing)

Heh. KISS. So, what exactly is simple? DRY. Isn’t that supposed to be a time-saver? Well, it depends on what you’re not trying to repeat. Strange how these cuddly and liberating acronyms can have more than one interpretation.

Experience taints everything.

As usual, my day did not pan out as expected. But, also as usual, I learned a lot!

Coming Up To Speed on Rspec

So, learning rspec has been on my list for a while. I finally got around to it. Nice framework. I am familiar with EasyMock, and am aware of JMock. I never had an opportunity to get into Mockito, but I’d give it a glance next time I put on my Java hat.

There is some decent documentation on it out there. I found these links particularly helpful:

• The formal documentation.
• David Chelimsky’s from-the-ground-up description
• rspec.info’s drill-down into Mocks and Stubs
• Luke Redpath’s cross-over into usage under Rails

During my ramping-up, I took the usual meta-approach of creating a test suite — and yaaay, that’s what rspec is meant for! — which I then used to test out its own range of capabilities. The Modules in the rdoc which I’ve found to provide the most value are:

• Spec::Expectations::ObjectExpectations for conditionals (eg. should & should_not)
• Spec::Matchers for expectations (eg. equal(value), be_a(class), respond_to(method_sym), raise_error)
• Spec::Mocks::Methods for mock method definition (eg. should_receive(method_sym))
• Spec::Mocks::MessageExpectation for mock behaviour (eg. with(*args), once, exactly(n).times, any_number_of_times)
• Spec::Mocks::ArgumentConstraints for mock arguments (eg. an_instance_of(class), anything)
• Spec::Mocks::BaseExpectation for mock responses (eg. and_return(value), and_yield(&block))

I won’t got into the deep details, but here are some examples of conditionals and expectations that I paraphrased into my meta-test:

	specify "knowledge of nil" do
		''.should_not be_nil
	end

	specify "numeric calculations" do
		(355.0 / 113).should be_close(Math::PI, 0.1)
	end

	specify "changes made by a closure" do
		array = []
		lambda {
			array << :a
		}.should change(array, :size)
	end

	specify "equality" do
		5.should eql 5
		5.should equal 5
		[:a].should_not equal [:a]
	end

	specify "raising of errors" do
		#	always construct fresh!!!
		#		otherwise it won't be re-wrappable
		def raiser(s=nil)
			lambda { raise(RuntimeError, s) if s }
		end

		raiser('x').should raise_error
		raiser('y').should raise_error(RuntimeError, 'y')
	end

	specify "closure satisfaction" do
		5.should satisfy {|n| (4..6).include?(n) }
	end

	specify "what lists do" do
		[].should respond_to(:each, :find, :size)
	end

And here’s a little bit of silly mocking:

	def expect_raise(type=Spec::Mocks::MockExpectationError, &block)
		abort 'block must be provided' unless block_given?
		block.should raise_error(type)
	end

	specify "the basics" do
		@mock.should_receive :hello
		@mock.should_not_receive :goodbye

		@mock.hello
		expect_raise { @mock.stay }
		expect_raise { @mock.goodbye }
	end

	specify "how many times, and with what" do
		@mock.should_receive(:one).once.with(1)
		@mock.one 1

		@mock.should_receive(:string).exactly(1).times.with(an_instance_of(String))
		@mock.string 'a string'

		@mock.should_receive(:anything).exactly(3).times.with(any_args())
		@mock.anything
		@mock.anything :again
		@mock.anything :third, 'time'

		@mock.should_receive(:array_ish).with(duck_type(:each, :find, :size))
		@mock.array_ish [:item]
	end

	specify "what i return, raise or throw" do
		@mock.should_receive(:get_one).and_return(1)
		@mock.should_receive(:put_one).with(1)
		@mock.put_one @mock.get_one

		@mock.should_receive(:increment).any_number_of_times.with(instance_of(Fixnum)).and_return {|i| i + 1 }
		@mock.increment(1).should equal(2)

		@mock.should_receive(:raises_string).and_raise('something runtime')
		expect_raise(RuntimeError) { @mock.raises_string }

		@mock.should_receive(:pitch).and_throw(:ball)
		lambda { @mock.pitch }.should throw_symbol(:ball)
	end

	specify "yielding in a complex fashion" do
		@mock.should_receive(:gimmee).exactly(3).times.and_yield(:x)

		holder = []
		@mock.gimmee {|value| holder << value }
		holder.should eql([:x])
		2.times { @mock.gimmee {|value| holder << value } }
		holder.should eql([:x, :x, :x])
	end

	specify "validation via closure" do
		(@mock.should_receive(:threely) do |value|
			value.to_s.size.should eql(3)
		end).exactly(3).times

		@mock.threely 'x' * 3
		@mock.threely :key
		@mock.threely 333
	end

These are just ways I thought of to exercise the width and breadth of the library. Very nice. I hope that these are useful examples for people new to this gem.

I recommend the other references from above for filling in the missing details. Once you have a context / specify or a describe / it specification set up, you’ll be good to go. There’s much more to the library — Stories, for instance — but that’s for another day.

An Informative Walk through Twitter4R

No, I didn’t really want to spend time learning rspec — I mean heck, I’m busy — but I had a personal need to expand the Twitter4R gem. Specifically, I wanted to add on some Twitter search features, and I was very impressed with how this library has been built. Contribution-wise, the final step that Susan Potter recommends is to craft up some rspec tests.

Of course, mock testing is only as good as the framework you’re built upon. The assumption is that Net::HTTP is going to do it’s job, so mock it up and you can even test your Twitter features offline. When I built Bitly4R (given that name, my thinking has clearly been influenced), I did everything as full-on functional tests. It was easy; bit.ly has both shorten and expand commands, so I could reverse-test real values without having any fixed expectations.

However, Twitter is live and user-generated, so who knows what you’ll find. Mocking covers that for you. And of course not having to hit the service itself shortest testing time dramatically.

Here’s one of my tests, again foreshortened:

	before(:each) do
		Twitter::Client.send :public, :create_http_get_request

		@client = Twitter::Client.new
		@client_clone = Twitter::Client.new
	end

	it "produces a querystring with all the crazy stuff Twitter imagined" do
		the_params = nil

		@client.should_receive(:create_http_get_request).and_return {|uri, params|
			the_params = params
			@client_clone.create_http_get_request(uri, params)
		}
		@client.should_receive(:http_connect).and_return Net::HTTPNotFound.mock(:code => '404')

		#	pass through a few we can pick back out
		#	also, side-effect ... nil on invalid response
		@client.search('query_value', :rpp => 3).should be_nil

		the_params.should_not be_nil
		the_params[:q].should eql('query_value')
		the_params[:rpp].should eql(3)
	end

Plus a little mocking of Net::HTTPResponse:

class Net::HTTPResponse
	def self.mock(ops={})
		#	defaults
		ops = {
			:http_version => '1.1',
			:code => '200',
			:message => 'OK',
			:body => nil,
		}.merge(ops || {})

		#	construct
		###clazz = CODE_TO_OBJ[ops[:code]]
		clazz = self
		response = clazz.new(ops[:http_version], ops[:code], ops[:message])

		#	inject
		ops.each do |k, v|
			response.instance_variable_set "@#{k}".to_sym, v
		end

		#	mockulate
		response.instance_eval %q{
			def body; @body; end
		}

		response
	end
end

Nothing like having some good inspiration to get you thinking about maintainability. I’m a not a strong adherent to any of the TDD or BDD denominations, but testability itself is close to my heart. Just ask the folks who’ll be looking at the micro-assertion Module that I wrote for my Facebook Puzzle submissions (which work locally under test conditions but fail miserably once thrown over their wall). Then again, I won’t need to write that (and test that) from scratch again.

So, back to talking glowingly about the Twitter4R architecture. Yes, I used that word. Yet regardless of that term’s heavyweightedness, it’s simply the result of carefully thinking out the formalization of a complex design. And once a non-author delves into extending an existing implementation, that impl’s demeanor becomes readily clear :)

Some of the interesting things I saw:

• An inner bless-ish strategy, to inject the Twitter::Client into instanciated result classes. I’ve seen blessing as an OO-Perl-ism, for self-ication, and that metaphor carries over very nicely to this context.
• Generous use of blocks / lambdas / closures in methods, for contextual iteration (eg. iterate through each Twitter::Message in a timeline response). Unnecessary, but an excellent convenience for a language that offers that optional capability in a very disposable fashion (from the caller’s perspective).
• Retroactive sub-object population after construction. Twitter4R relies upon the json gem, which deals primarily in Hashes. Post-injection, the library itereates through Arrays of Hashes and transforms them into suitable Twiltter::User containers, etc. A great place to put such logic in the construction chain, and it doesn’t take long to get really tired of Hash hierarchies.

Good stuff, and learning with in a Ruby-centric mindset was invaluable for me. We all have to start somewhere, eh.

The Acute Long-Term Pain of Staticness

There was one issue that I ran into; static configuration. During my years of using the Spring Framework, my IOC-addled brain started thinking of everything in terms of instances — Factory Beans, POJOs, Observers & Listeners, Chain-of-Responsibility wrappers. Static configuration is a common metaphor, and in this case, there’s a static Twitter::Config instance. Convenient and centralized. Makes perfect sense.

I mean, the fact that it was a configurable library at all was awesome. I was able to easily set up a Twitter::Client to reference search.twitter.com. However, of course as soon as I did that, I whacked the ability for clients to talk to twitter.com in the process. Oops!

On GPs, I refused to modify the original code. And I wanted to make sure that my superficial tweaks to the library would be thread-safe — temporarily swaping out the global Twitter::Config in mid-operation would be an issue. Using Mutex.synchronize seemed like the perfect choice. After finding that the same thread can’t lock a Mutex instance twice — grr!, that’s a great trick if you can work it — I overrode the one method that cared the most about @@config:

	@@config_mutex = Mutex.new
	def configuration_mutex
		@@config_mutex
	end

	alias :raw_http_connect :http_connect
	def http_connect(*args, &block)
		ops = (Hash === args.last) ? args.pop : {}

		result = nil
		self.configuration_mutex.synchronize do
			ops[:call_before].call if ops[:call_before]
			result = raw_http_connect *args, &block
			ops[:call_after].call if ops[:call_after]
		end

		result
	end

	#
	#	...
	#

	@@SEARCH_CONNECT_HOOKS = {
		:call_before => lambda { self.configure_apply(:search) },
		:call_after => lambda { self.configure_apply(:default) },
	}

	#
	#	...
	#

	#	broken out of closure for mocking
	req = create_http_get_request(uri, params)
	response = http_connect(@@SEARCH_CONNECT_HOOKS) {|conn|	req }
	return nil unless Net::HTTPOK === response

	model = bless_model(Twitter::Search.unmarshal(response.body))

It works like a charm. Please, everyone just line up to tell me how I could have done it better. And I don’t mean quicker or cheaper, I mean better. Believe me, I would not have sunk the time into this end-around approach, if not for the fact that:

1. I don’t want to maintain a local gem code modification (even though my impl is closely coupled to the gem impl already)
2. I intend to follow that practice, so every opportunity to pull and end-around is a Valuable Learning Experience.

So, now my local Twitter4R has search capability gracefully latched onto it (and implemented much in the flavor of the library itself). I have a mass of rspec examples to work off in the future.

Now, I haven’t spent a great amount of time testing the thread-safeness — no one in their right mind wants to do that — but my sundry Twitter::Client instances play nicely together in between searches and normal status operations.

And I had something useful to blog about!

When signing up for Amazon Web Services, you end up generating all of the following identifying information:

• Your Account Number
• An Access Key
• A Secret Access Key
• An SSL Certificate file
• The Private Key for your SSL Cert

The various AWS command APIs and tools will require you to provide one more more of these pieces of information. Rather than actually host all of this information on my instances, I have instead chosen to build Ruby scripts using:

• Glenn Rempe’s Amazon EC2 gem (amazon-ec2)
• The net-ssh and net-scp gem suite

I’m small-scale for the moment, so I can keep a centralized list of my instance IDs. Without too much effort, I can look up the instance(s), identify their public DNS (plus everything else), and then open up an ssh connection and push data into a channel’s environment or upload files for bundling purposes.

Here’s a few things that I’ve learned in the process.

Channels

Most of your core work gets done on a Net::SSH::Connection::Channel, and sometimes asynchronously (as in the case of my Health Check script). There are a lot of library shorthands — the prototypical Hello World example is channel-less:

Net::SSH.start("localhost", "user") do |ssh|
	# 'ssh' is an instance of Net::SSH::Connection::Session
	ssh.exec! "echo Hello World"
end

But as with all examples, you soon find that surface-level tools won’t quite do the trick. The main thing you’ll need to have all the time is convenient access to the data come back from the channel, asynchronously or otherwise. So create yourself a simple observer:

def observer_struct
	@observer_struct ||= Struct.new(:channel, :stdout, :stderr, :exit_status, :exit_signal)
end

def observe_channel(ch, ops={})
	observer = observer_struct.new(ch, '', '', 0, nil)
	is_puts = !!ops[:puts]

	#	stdout
	ch.on_data do |ch, data|
		$stdout.print data if is_puts
		observer.stdout << data
	end
	# only handle stderr
	ch.on_extended_data do |ch, type, data|
		next unless [1, :stderr].include? type
		$stderr.print data if is_puts
		observer.stderr << data
	end

	ch.on_request("exit-status") do |ch, data|
		observer.exit_status = data.read_long
	end
	ch.on_request("exit-signal") do |ch, data|
		observer.exit_signal = data.read_long
	end

	observer
end

def open_channel(ssh, ops={}, &block)
	ch = if block_given?
		ssh.open_channel &block
	else
		ssh.open_channel
	end

	#	somebody's watching you
	self.observe_channel ch, ops
end

Then during and after executions, you’ll have access to all the core information you’ll need to make conditional decisions, etc.

Pushing Your Environment

Many of the AWS tools will recognize specific environment variables. The usual suspects are:

• AMAZON_ACCESS_KEY_ID
• AMAZON_SECRET_ACCESS_KEY

So, you can use Net::SSH::Connection::Channel.env to inject key-value pairs into your connection. However, you’ll need to make some config changes first — and major thanks to the Net::SSH documentation for clarifying this in its own RDoc:

“Modify /etc/ssh/sshd_config so that it includes an AcceptEnv for each variable you intend to push over the wire”

#       accept EC2 config
AcceptEnv  AMAZON_ACCESS_KEY_ID AMAZON_SECRET_ACCESS_KEY
...

After you make your changes, make sure to bounce the ssh daemon:

/etc/init.d/sshd restart

Then your selected environment will shine through.

Running sudo Remotely

It’s reasonable to execute root-level commands using sudo, because you can provide a good amount of granular control.

The first thing that we’ll all run into is:

sudo: sorry, you must have a tty to run sudo

Fortunately, there’s Net::SSH::Connection::Channel.request_pty:

ch.request_pty do |ch, success|
	raise "could not start a pseudo-tty" unless success

	#	full EC2 environment
	###ch.env 'key', 'value'
	###...

	ch.exec 'sudo echo Hello 1337' do |ch, success|
		raise "could not exec against a pseudo-tty" unless success
	end
end

I’ve taken the approach of allowing NOPASSWD execution for everything I do remotely, after making darn sure that I had constrained exactly what could be done under those auspices (HINT: pounds of caution, tons of prevention). You can configure all of this by editing /etc/sudoers:

# EC2 tool permissions
username  ALL=(ALL)  NOPASSWD: SETENV: /path/to/scripts/*.rb
...

Also, if you intend to have those scripts consume the environment variables that you’ve injected into your channel, you’ll also need to annotate the line with SETENV, or they won’t export across into your sudo execution.

If you are more security minded, then there’s many other variations you can play with /etc/sudoers. I haven’t yet experimented within pushing a sudo password across the wire, but that may be where Net::SSH::Connection::Channel.send_data comes into play.

Port Forwarding

I wanted to easily do SSH tunneling / port-forwarding against an instance, referenced by my local shorthand. So I wrote my usual EC2 instance ID lookup and started an Net::SSH connection. Here’s how you put yourself in ‘dumb forward’ mode:

def ports, offset = [], 0
# ...

Net::SSH.start(hostname, user, :forward_agent => true, :verbose => :warn) do |ssh|
	ports.each do |port|
		ssh.forward.local port + offset, 'localhost', port
	end

	# loop, with an event loop every 0.1s, until Ctrl-C is pressed
	puts "[Ctrl-C] to terminate..."
	interrupted = false
	trap('INT') { interrupted = true }
	ssh.loop(0.1) {
		not interrupted
	}

	ports.each do |port|
		ssh.forward.cancel_local port + offset
	end
end

And then [Ctrl-C] will break you out.

Inconsistent EC2 Marshalling

I’d gotten all of my EC2 data parsing working great, then I started to expand my capabilities with the Amazon S3 API gem (aws-s3). Suddenly and magically, the results coming back from my EC2 queries had taken on a new form. Moreover, this was not consistent behavior across all operating systems; at least I do not see it on WinXP, under which my health monitoring checks execute.

The Amazon APIs are SOAP-driven, so I guessed that amazon-ec2 (and aws-3) would both leverage soap4r. Well, that is in fact not the case; Glenn has commented below on this known issue. He uses the HTTP Query API, and the cause of the discrepancy are (as of this writing) still undetermined.

Regardless, our job is to get things to work properly. So for the meanwhile, let’s use a quick & dirty work-around. The two core differences are:

• the data coming back may be wrapped in an additional key-value pair
• single value result-sets come back as an Object (vs. an Array with a single Object)

So I crafted a little helper singleton, which which still allows me to detect between ‘valid’ and ‘empty’ requests (nil vs []):

class SOAPHelper
	class << self
		def data_from(data, wrapper)
			data[wrapper] || data
		end

		def items_from(set)
			unless set && set.item
				nil
			else
				items = set.item
				items = [items] unless Array === items
				items
			end
		end
	end
end

data = SOAPHelper.data_from(
	@ec2.describe_instances,
	'DescribeInstancesResponse'
)
reservations = SOAPHelper.items_from(data.reservationSet) || []
...

That will address our immediate needs for the moment, until a benificent coding Samaritan comes along.

I’ve recently hosted my Spring / Hibernate webapp in the cloud thanks to Amazon Web Services. A future post will mention the monitoring I’ve put in place, but Tomcat keeps dying after about 36 hours. The first thing I need to do is enable debugging and JMX remoting so that I can put JConsole to work.

This turned out not to be as easy as I’d liked. Even with a bevy of useful resources available — lotsa people have run into these issues — it took a while to get the right combination. Let’s hope I can save you a bit of that pain …

JConsole / JMX Remoting via SSH Tunnels

As I mentioned, I’m hosting this solution in the cloud. So, when things go bad, I need to be able to debug remotely. I don’t want to open up my security group to general traffic, so using SSH tunnels is the best option. JConsole is a great tool for measuring current statistics and performance of your Java app, and relies on JMX remoting. It works great locally, or even within a trusted network, but once you’re behind a firewall with broad port-blocking, there are some significant issues. There are several core Java forum topics related to this discussion:

• http://forums.sun.com/thread.jspa?threadID=5267091
• http://forum.java.sun.com/thread.jspa?forumID=4&threadID=565139
• http://forum.java.sun.com/thread.jspa?forumID=4&threadID=5119655
• http://forum.java.sun.com/thread.jspa?forumID=58&threadID=703567

Daniel Fuchs has written several articles which illustrate these issues and provide good solutions. He explains that JMX remoting needs two ports: one for the RMI registry, and one for the RMI connection objects which are stubs used for remoting all the critical data. If you’re using the default JVM agent, you’ll tend to use the following JVM System.property’s on the server:

-Djava.rmi.server.hostname
-Djava.rmi.server.useLocalHostname
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port
-Dcom.sun.management.jmxremote.ssl
-Dcom.sun.management.jmxremote.ssl.need.client.auth
-Dcom.sun.management.jmxremote.authenticate
-Dcom.sun.management.jmxremote.access.file
-Dcom.sun.management.jmxremote.password.file

I’ll come back to these, but the one that’s important here is jmxremote.port. It allows you to specify the RMI registry port, the one that you’ll use to establish your remote connection via JConsole. However, the port for RMI export, which is used for all the critical data callbacks, is randomly chosen (on a session basis or JVM basis, not sure) and cannot be specified. And you can’t open a port for tunneling if you don’t know what it is.

You can see this issue if you crank up the debugging on JConsole. I was having issues getting the logging output so I took the double-barrel approach, using both the -debug argument and the custom java.util.logging descriptor, the contents of which I stole from here. Invoke it as follows:

jconsole -debug -J"-Djava.util.logging.config.file=FILENAME"

The quotes are optional. Provide the logging descriptor filename. You can call out the JMX Service URL or the hostname:port combination at the end if you like. Now you’ll eventually a debug output much like this:

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] connecting...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] finding stub...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] connecting stub...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] getting connection...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] failed to connect: java.rmi.ConnectException: Connection refused to host: IP-ADDRESS; nested exception is:
java.net.ConnectException: Operation timed out

PORT will be the RMI registry port you’re tunneling into. IP-ADDRESS is special, we’ll get to that, and it’s important to note that it’s a ‘ConnectException‘ occurring against that host.

This debugging information can show up rather late in the whole connection process, undoubtedly because it’s an ‘Operation timed out‘ issue, so don’t be surprised if it takes a while. Fortuntely, you can also see immediate verbose feedback when you set up your ssh tunnel connection (see below).

Addressing the Randomly-Generated RMI Export Port

The first problem I chose to resolve was the one relating to the random RMI export port issue. Daniel has provided a fine example of how to implement a custom ‘pre-main’ Agent which you can use to supplant the standard JVM one. There’s his quick’n'dirty version which doesn’t address security — which is where I started. And then there’s a more full-fledged version, which I modified to be configurable.

Most importantly, it builds its JMXConnectorServer with the following service URL:

service:jmx:rmi://HOSTNAME:RMI-EXPORT-PORT/jndi/rmi://HOSTNAME:RMI-REGISTRY-PORT/jmxrmi

Traditionally, you’ll see this service URL from the client perspective, where HOSTNAME:RMI-EXPORT-PORT is not defined and you just have ‘service:jmx:rmi:///jndi/rmi://...‘. JConsole will build that sort of URL for you if you just provide HOSTNAME:RMI-REGISTRY-PORT (eg. hostname:port) when connecting.

By calling out the RMI-EXPORT-PORT in the agent’s service URL, you can affix it and tunnel to it. You can use the same port as the RMI registry; this only requires you to open one port for tunneling.

On your client / for JConsole, the HOSTNAME will probably be localhost, where you’ve opened your tunnel like so:

ssh -N -v -L9999:REMOTE-HOST:9999 REMOTE-HOST

9999 is just an example port. REMOTE-HOST is the host you’re tunneling to. You can remove the -v argument, but it’s good to have that around so that you can see the network activity. You can also use the -l argument to specify the login username on the remote host. Note that you’re opening the same port locally as you’re hitting on the server, with no offset. You’ll need to open the same port because your agent on the server is going to need to know what port to callback to itself on for RMI export, and that won’t work if you have an offset. So you might as well use the same port for both the RMI registry and RMI export, and just keep that one port available locally.

On the server in your agent, the HOSTNAME part of the service URL can either be InetAddress.getLocalHost().getHostName(), or an IP address (127.0.0.1), or in my case ‘localhost’ just worked fine.

The major reason to create the custom agent is to build the port-qualifying service URL. As usual, the example code takes a lot of shortcuts. So, I built myself a more re-usable agent — influenced by the standard JVM agent’s System.property’s — which allowed me to configure the same sorts of things as mentioned above:

• hostname : to be used as the HOSTNAME value above
• port.registry : to be used for RMI-REGISTRY-PORT
• port.export : to be used for RMI-EXPORT-PORT, defaulting to the same as port.registry if not provided
• ssl : true to enable SSL
• authenticate : true to enable credentials / authentication
• access.file : specifies the location of the user credentials file
• password.file : specifies the location of the user password file

And so I was able to configure my agent service URL for localhost, using the same port for both RMI requirements, and using simple password-based auth. I did not go down the SSL route, though many of the posts from Daniel and others explain this as well. Do that once you’ve tackled the core problem :)

Another great post relating to this issue mentions that Tomcat has a custom Listener for setting up a similar agent. The example was:

<Listener className="org.apache.catalina.mbeans.JMXAdaptorLifecycleListener" namingPort="RMI-REGISTRY-PORT" port="RMI-EXPORT-PORT" host="localhost"/>

I didn’t look any deeper into this to see whether it supports SSL and/or basic authentication. But it seems clear that this *not* a Java agent, because you have to set those up via System.property’s. Here’s what I needed to add to Tomcat startup for my custom agent:

-D___.hostname=localhost
-D___.port.registry=RMI-REGISTRY-AND-EXPORT-PORT
-D___.authenticate=true
-D___.password.file=$CATALINA_HOME/conf/jmxremote.password
-D___.access.file=$CATALINA_HOME/conf/jmxremote.access
-javaagent:PATH/TO/AGENT.jar

I’ve omitted the namespace I used for my agent, and the formal name of the ‘pre-main’-compatible JAR file that I built using the instructions that Daniel provided. Tomcat won’t start up properly until the agent is happy; after that, then you’re golden.

So I got Tomcat running, started up an ssh tunnel, and invoked JConsole. And still no matter what I did, I still got ConnectException: Operation timed out. I tried to connect via JConsole in all the following ways:

HOSTNAME:PORT
service:jmx:rmi:///jndi/rmi://HOSTNAME:PORT/jmxrmi
service:jmx:rmi://HOSTNAME:PORT/jndi/rmi://HOSTNAME:PORT/jmxrmi

All of these are valid URLs for connecting via JConsole. For a while there I wasn’t sure whether you could use the same port for both the RMI registry and export, so I could see that the JConsole log was different when I called out the RMI export info explicitly in the service URL. Still, it didn’t seem to help.

Then I started to realize that there were two separate issues going on, although they tended to blend together a lot in the posts I’d been reading.

Addressing the RMI Export Hostname

The short version is, even if you’ve set up your JMX service URL properly on the server — yes, even if you’ve set its HOSTNAME up to be ‘localhost’ — you’ll still need to tell JMX remoting which hostname the RMI export objects should use for callbacks. This requires you to provide the following System.property’s as well:

-Djava.rmi.server.hostname=localhost
-Djava.rmi.server.useLocalHostname=true

The useLocalHostname may not be relevant, but it doesn’t hurt. All this time I’d thought that because I was configuring that information in the service URL that RMI would build the objects accordingly. But I was wrong … it doesn’t … you need to call that out separately.

What was not apparent to me — until I started to see the same articles pop up when I revised my search criteria — was the IP-ADDRESS in this exception dump:

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:PORT/jmxrmi] failed to connect: java.rmi.ConnectException: Connection refused to host: IP-ADDRESS; nested exception is:
java.net.ConnectException: Operation timed out

It was the IP address of my instance in the cloud. The callbacks were being made back to my VM, but they needed to be made to ‘localhost’ so that they could go through the tunnel that I’d opened. The ‘Operation timed out‘ was due to the port being closed, which is the whole reason you’re using ssh tunnels in the first place. Once the RMI exported objects know to use ‘localhost’, that addresses the problem. And magically, JConsole will connect and show you all the data in the world about your server.

So you must provide those System.property’s above regardless of what other configuration you’ve provided in your custom JMX agent.

Additional Concerns

There were a number of other red herrings that I followed for a while, but they were called out as being potential issues, so I kept note.

• If your server is running Linux, there are a couple of things you’ll want to check, to make sure that your /etc/hosts is self-referencing correctly, and that you’re not filtering out packets.
• You will have troubles stopping Tomcat when it has been started with your custom JMX agent; you’ll have to kill the process. Apparently agents don’t release their thread-pools very nicely. Daniel provides an example of an agent with a thread-cleaning thread — which still has some limitations, and raises the philosophical question ‘who cleans the thread-cleaner‘? He also provides an agent that can be remotely stopped — which is reasonably complex. I’ll save that one for a rainy day.
• If you want to use SSL in your authentication chain, read up on Daniel’s other postings, and use the following System.property’s on both the server and when starting JConsole:

  -Djavax.net.ssl.keyStore
  -Djavax.net.ssl.keyStorePassword
  -Djavax.net.ssl.trustStore
  -Djavax.net.ssl.trustStorePassword

• I have built some Ruby scripts which allow me to dynamically look up an AWS instance’s public DNS entry and then start up a Net::SSH process with port-forwarding tunnels. This works fine for HTTP and even for remote JVM debugging, but it did not work for JMX remoting. I’m not sure why, so you should stick with using ssh for setting up your tunnels.
• I started out this exercise using Wireshark for packet sniffing. I’m using OS X, and I installed Wireshark — the successor to Ethereal — via MacPorts. It runs under X11, which you’ll need to install from either Apple’s site or your Optional Installs DVD. I couldn’t get any Interfaces (ethernet card, etc.) to show up, until I learned that I should:

  sudo wireshark

  The app will warn you that this is unsafe, but it works. The Nominet team says that you can address this issue by providing:

  sudo chmod go+r /dev/bpf*

  However that is volatile, and has to be done whenever your Mac starts up. More config involved, so I took the easy path.

• If you’re using a script to start and stop Tomcat, you’ll need to somehow separate out the System.property’s that should be used on startup, and omit them when invoking shutdown. If you invoke shutdown with your debug and/or RMI ports specified, the shutdown will fail because those ports are already in use.

  I’m using the newest standard Tomcat RPM available for Fedora Core 8 — tomcat5-5.5.27 — and it’s uniquely nutty in terms of how it is deployed:

  /etc/init.d/tomcat5
  /usr/bin/dtomcat5
  /etc/tomcat5/*.conf

  That’s a very non-standard arrangement. The init.d script awks the *.conf files, and a whole array of other exciting things. I still haven’t gotten it to properly do an init.d restart due to how it blends the JAVA_OPTS handling. So that’s left as a case-specific exercise.

• The whole reason I went down this path was to address a memory leak relating to Hibernate sessions which I blogged about a long time ago. The fix required me to invoke Tomcat with the following System.property’s:

  -Djavax.rmi.CORBA.PortableRemoteObjectClass
  -Djava.naming.factory.initial

  The org.objectweb.carol JAR, which these settings were targeted at, is part of my weapp, so it’s available in its own Classloader. However, once I put the custom JMX agent in place, I got:

  FATAL ERROR in native method: processing of -javaagent failed
  Exception in thread "main" java.lang.reflect.InvocationTargetException
  Caused by: java.lang.ClassNotFoundException: org.objectweb.carol.jndi.spi.MultiOrbInitialContextFactory

  Attempting to create a symlink to the app-specific JAR in either common/lib, common/endorsed or shared/lib did not address the issue. I had to hack the JAR into the --classpath in order to get Tomcat to start. And yes, hack was the operative term (again).

In Summary

Frankly, all that discovery was enough for one day. And yes, it took me that long to find all of the corner-cases i was dealing with. I hope that if you find this article that it will make your path a bit easier. I know I’ll be glad that I blogged about the details the next time I bump into the issue!

Everbody loves cron. The classic basic scheduler, 80/20 flexibility, gets the job done. So, when I started with OS X, I went looking for cron.

Yes, you can cron if you want to. Or maybe, as the Mac children recommend, you can create Launch Daemons. It’s simply a custom launchd.plist, an XML file to define tasks in Apple’s terms. Sweet, I can do that…

The logrotated.plist Daemon

From what I could tell, out of the box, Leopard doesn’t give you logrotate. So, here it is as a Launch Daemon:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Disabled</key>
	<false/>
	<key>Label</key>
	<string>logrotate</string>

	<key>ProgramArguments</key>
	<array>
		<string>/opt/local/sbin/logrotate</string>
		<string>-f</string>
		<string>/etc/logrotate/*.conf</string>
	</array>
	<key>EnableGlobbing</key>
	<true/>

	<key>KeepAlive</key>
	<false/>
	<key>LaunchOnlyOnce</key>
	<false/>
	<key>RunAtLoad</key>
	<false/>
<!--
causes 'Throttling respawn: Will start in 10 seconds' in system.log
	<key>OnDemand</key>
	<false/>
-->

<!--
http://www.nabble.com/launchd-support-ranges-for-StartCalendarInterval - - td14857719.html
-->
	<key>StartCalendarInterval</key>
	<!-- 3:05a, 3:05p -->
	<array>
		<dict>
			<key>Hour</key>
			<integer>3</integer>
			<key>Minute</key>
			<integer>05</integer>
		</dict>
		<dict>
			<key>Hour</key>
			<integer>15</integer>
			<key>Minute</key>
			<integer>05</integer>
		</dict>
	</array>
</dict>
</plist>

Great. So what do I do with it? Name it anything ‘*.plist’ — logrotated.plist perhaps — and put it here:

• /System/Library/LaunchDaemons or
• /Library/LaunchDaemons

I tend to use System for any services I wan to launch on boot (chkconfig-ish), and anything that’s just a scheduled task is more pedestrian.

You register daemons through launchctl. When you change it, just:

$ launchctl unload FILE.plist
$ launchctl load FILE.plist

Sure, not a hard thing. What I did want to point out were some of the things that I learned:

• Provide each space-separated component of the command line as <string /> element in the ProgramArgument <array />. It’s the easiest way to go.
• I tend to keep the KeepAlive, LaunchOnlyOnce and RunAtLoad all in sync for each daemon. It seems to guarantee compatability.
• I still haven’t gotten quite the hand of OnDemand (c’mon, I’m new to this). When you are developing your daemons, keep a watch on /var/log/system.log — if you screw up, you’ll probably see it there in one fashion or another.
• StartCalendarInterval is the equivalent of the cron schedule mask. See the Nabble post i commented above. You can’t do ranges, but wildcards are do-able sorta by omission. If you want mulltiple disparate schedules, as you see above, you need to pass in an <array /> of <dict />s, not just a single <dict /> when you only need one mask.
• Yes, datatypes matter! I couldn’t figure out for the life of me why my daemon wouldn’t start, until I realized that my Hour and Minute values were configured as <string />s — vs <integer />s — because I’d been, well, you know, trying to do ranges and wildcards the cron-style way. launchctl didn’t complain, it just didn’t … work. So don’t do that.
• Any daemon which has a schedule that fires off when the system is down will be automatically executed shortly after the system re-awakens. I don’t know exactly how quickly, but I’ve seen it in action (though I’ve read some dispute about its efficacy in my searches).

The Trials and Tribulations of MacPorts

But of course this Epistle wouldn’t be any fun without a twist, right? Well, lucky us, because it wasn’t that easy.

I’m using MacPorts as my yum-my package manager. It’s great, and easy to set up, but it’s quite as stable as I’ve seen under Linux. Or rather, I’ve seen a disproportionate number of issues in the times I’ve used it. However, I’m eternally grateful because it saves me so much time … the successes far outweigh the problems.

When things do go bad, as they did when I built logrotate 3.7.7, you have to set up a custom Portfile and source repository so that you can effectively drive MacPorts to fetch and build your specific version. logrotate 3.7.1_1 turned out to be much more stable.

Techonova and Joe Homs go into very excellent and welcome detail in their posts on how to pull this off. A summary of what you need to do is:

• watch the build failure — debug with port -d — to identify where the dying source is located. That’ll be PATH/TO/PROJECT
• visit http://trac.macosforge.org/projects/macports/log/trunk/dports/PATH/TO/PROJECT and snag the Portfile. That’s a MacPorts spec file, and you can tweak it to do your bidding.
• create a source directory that you’ll keep around for a while (mine is /Users/Shared/macports).
• register that directory — your local Portfile repository — with /opt/local/etc/macports/sources.conf (using file:// protocol)
• copy the Portfile into a PATH/TO/PROJECT sub-directory structure (eg. sysutils/logrotate). Basically, the same path you’ll have snagged it from.
• pull down a previous source revision and snapshot it locally. That could come from SVN or git, tarball, whatever. Start search from the homepage setting of the Portfile.
• tweak the Portfile to ‘make sense’ for the source you’ve pulled down. If you based upon a close-enough version, it should be as easy as tweaking the version, etc. — no custom build tasks. I’m glossing over when I say that it’s usually a matter of version-based naming conventions and MD5 checksums.
• re-build the repository’s PortIndex (which you’ll need to do every time you make an addition / change)

Please feel free to consult the other two posts to fill in the details that I was so cavalier about.

In Summary

This was another one of those occasions when I was glad I kept track of what I was doing while I was in the moment. Pack your config files with comments, because they’re invaluable. And drop READMEs around where you’re likely to find them … I filled in a lot of the extra details for this post from those breadcrumbs I left for myself, things I could have easily wasted 15-20m on re-discovering by braille :)

Peace.