Archive for the ‘Learning’ Category

Virtualizing Windows XP

Sunday, April 5th, 2015

Over the past several months, in my copious free time, I have been working on a project to virtualize an old Windows XP box with a rather curious partition and boot configuration. This grand project involved

Exciting stuff, huh!

Actually, the process got pretty involved, and I ended up peering into some dark & dusty corners. Rather than try to write it all up as a simple Blog Post, I’ve created a Git repo:

cantremember/virtualizing-windows-xp

It’s got more hyperlinking in it than you can shake a stick at.

Please enjoy!

Upgrading your Rails Development Mac to Snow Leopard

Saturday, February 6th, 2010

Oh, there is such joy in the process of upgrading to Mac OS/X Snow Leopard for us developer folks. Me, I chose the in-place ugrade path … my co-worker, who chose the from-scratch path, was deprived of some of these pleasures. Then again, he had to reconstruct everything from scrach, so he had his own barrel of monkeys to contend with.

Here’s all the bumps I ran into, pretty much in reverse order as I tried (unsuccessfully) to do the minimum amount of work possible :) I had to go pretty much this entire process twice — once on my work Macbook Pro, once on my identical home verison — so I figured I might as well document all of this crap down in the hope that it may reduce the shock and awe of future migrators. Of course you may run into a mess of fun issues not described here … And pardon me if the instructions aren’t perfect, because I’ve tried to boil a lot of this down to the it-finally-worked steps extracted from the frenzied back-and-forth of my real-time upgrade experience :)

Backups

Yes, this is just common-sense developer stuff (as is a number of the other obvious things that I call out in this post).

You’ll probably want to do a full mysqldump before upgrading. You can dump your port install and gem list --local listings up front as well, or wait ’til you get to those respective steps below.

MacPorts

If you also chose the MacPorts library system, you’ll need to re-install it from scratch. You’ll need X11 from the Snow Leopard from the OS/X install disks, and download the latest version of Xcode. Follow the migration steps as outlined on their Wiki; it does the trick.

Save off your port install list as a reference. Now, your MacPorts install will be completely toast, so that command won’t work until you re-install. No problem though — all of your packages will still be listed even after you upgrade.

The port clean step in the Wiki will crap out in the http* range, but that’s fine … you can probably skip that step anyway. Re-install your core packages and you’re good to go. I suggest installing readline if you haven’t, because it’s very useful in irb or any Ruby console.

MySQL

It was not necesary for me to build MySQL from source. Instead, I just installed the x86_64 version of MySQL — the latest was mysql-5.1.42-osx10.6-x86_64.dmg af the time of this writing.

If this is a 5.x verison upgrade for you as well, the install will just re-symlink /usr/local/mysql, so your old data will still be in your previous install dir.

I didn’t make mysqldumps before I did the upgrade (handpalm) so I had to copy over my data/ files raw and hope that the new version would make friends with them. Initially I had problems with InnoDB. It wasn’t showing up under show engines;, and when I tried to manually install the plug-in — per this bug report, which explains the whole thing — it would fail on the ‘Plugin initialization function’. Turns out you need to do two things when you bring over raw data:

  • Whack your /var/log/mysql/*binary* / binary log files in order to get past mysqld startup errors.
  • Whack your ib_logfile* files too. Once you do that, there’s a good chance MySQL will regen them in recovery mode. Me, I had no choice (except rolling back with Time Machine). Miracle of miracles … it works!

Don’t try this at home kids. Make your backups. Note: here’s the correct link to the manual page on InnoDB tablespace fun.

x86_64 ARCHFLAGS

Snow Leopard is a lot more native 64-bit than previous OS/X versions, and when you do your manual builds & makes, you may want to set the following environment variable:

export ARCHFLAGS="-Os -arch x86_64 -fno-common"

You’ll see a set of similar (though mixed) recommendations in the blogs I reference below; this particular flagset worked for me.

Ruby

I built Ruby 1.9 at work, and 1.8.7 on my personal machine. Either path is fine, just pick up the latest source of your choosing. Chris Cruft’s blog post goes into some of the details I’m describing here as well. Basically, the README boils down to:

autoconf
./configure --with-readline-dir=/usr/local
make clean && make
make install

Though there’s no reason in the world that you’d want to — it has been superceded — do not install ruby 1.9.1p243. If you do, you’ll never get the mysql gem to work. Or wait, or was it mongrel? Well, it was one or the other … just trust me, it’s bad.

Gem

I re-built gem from source from scratch as well, just to be sure. Save off your gem env and gem list --local as a reference. And before you start installing gems, you’ll also probably want to make sure you’re fully up-to-date with gem update --system, though that’s probably redundant.

Uninstall and re-install all of your gems; if some won’t uninstall even though they’re listed, it may be an install path issue. Use gem list -d GEMNAME to find where your gem was installed, and then use gem uninstall -i DIRNAME GEMNAME to finish the job.

With the ARCHFLAGS in place, the vast majority of the native source gem builds will go smoothly, but there are some notable exceptions …

The mysql Gem

Uh huh, this is the one gem that gets me every time. And again, you won’t have needed to have built MySQL from source.

For starters, you may way want to glance over this very useful iCoreTech blog post to see if it works for you. But if you run into a lot of issues like I did, you may need to do it in two steps:

Fetch and Install the Gem

At the time of this writing, either mysql gem version 2.7 or 2.8.1 will do the trick.

gem fetch mysql --version VERSION
gem install mysql -- --with-mysql-dir=/usr/local --with-mysql-config=/usr/local/mysql/bin/mysql_config
gem unpack mysql

Sadly, it may fail, either during the build or when you try to test it. I was able to successfuly run the (included?) test.rb at my workplace, but as simple as that sounds, I swear I don’t remember how I did it ! The second time, at home, I only found the problems retroactively when I tried to get my Rails projects to boot. If you do find and run the test.rb, you’ll need to make sure that the standard MySQL test database exists.

Both times, one of the big blockers that I — and many other people — ran into was:

NameError: uninitialized constant MysqlCompat::MysqlRes

If so, try this:

Manually Re-build the Binary

Go into ext/mysql_api, make sure your ARCHFLAGS are exported as described above, and …

ruby extconf.rb --with-mysql-config=/usr/local/mysql/bin/mysql_config
make clean && make
make install

Hopefully your newly built & installed binaries will resolve the issue.

Mongrel

It took me a little effort to build mongrel on Ruby 1.9 with the x86_64 architecture. My memory is a little hazy — since my 1.8.7 build at home worked perfectly through standard gem install — but buried deep in this Is It Ruby contributory blog post are probably all the answers you’ll need.

Under Ruby 1.9, I did had to modify the source, which (to paraphrase) involved some global code replacements:

  • RSTRING(foo)->len with RSTRING_LEN(foo)
  • RSTRING(foo)->ptr with RSTRING_PTR(foo)
  • change the one-line case ... when statements from Java-esque : delimiters to then‘s.

And then the re-build:

ruby extconf.rb install mongrel
make
make install
cd ../..
ruby setup.rb
gem build mongrel.gemspec
gem install mongrel.gem

Conclusion

And that’s as far as I had to go. Whew! I certainly hope that my post has been of some assistance to you (and with a minimum of unintended mis-direction). Of course, I learned everything the I reiterated above by searching the ‘net and plugging away. And there’s plenty of other folks who’ve gone down this insane path as well. Good luck, brave soul!

Snake ‘n’ Bacon in The DDOS Caper!

Friday, August 7th, 2009

ah, come in! we’re so glad you’ve come Snake ‘n’ Bacon!
i’m crisp delicious bacon
sssss

glad you asked. it seems there’s a group of hackers, and we want you to go in under-cover
i go great on a sandwich
sssss


When Twitter came back online yesterday afternoon after their networking attacks, I got a torrent of @cr_snake_bacon tweets. Wasn’t sure why, but it seemed suspicious. Twitter’s API had flopped around for most of the day, so the logs were full of Exceptions and … oops! … re-connect attempts!

Of course I’d built the bots to re-tweet on an Exception. They’re all configured to wait 60 seconds, then try again. But of course until I fixed the configuration over night, they did exactly what a bot would doconspicuous

The service attacks on Twitter continued through today, and I’m sure that the birdy techs are furiously building black ice fortresses in Scala even now. Again, I saw a burst this afternoon from all of my bots. Pokey the Penguin, Conet Project, and Chewbacca all had several things to say, all at once. Obviously I had fucked something else up, so I hurriedly checked the logs. And nope … actually, my change had worked … Twitter had just un-blocked my IP.

*whew*

I’m not exactly sure how many bots are out there … here’s a nice wiki being kept of them. But I can imagine I’m not the only one who made that try-again coding mis-calculation. What’s sweet is that it’s un-done now, and my toys can continue prattling on.

Thanks, guys. Sorry we looked like a vicious autonoma for a while there. Glad to be back.

random problem in your cloud-hosted app? try a new instance!

Tuesday, May 26th, 2009

chalk this one up under ‘Time Sunk’.

my Ambience for the Masses app is a Spring / Hibernate / JSP stack, with a couple of other sweet components. i run it in on an AWS instance. it’s been purring along just wonderfully for months now. then, about ten days ago, it just stopped working

normally Java apps don’t die without throwing some sort of Exception. but that’s just what was happening. so i stripped out various components — thank you, Dependency Injection pattern! — and found that it would sometimes die instantly (if i was lucky) but usually it took a couple of hours. i don’t have a lot of free time to track down random intermittent bullshit like this, so it took me about a week to boil it down

it was somewhere in the Current Listener Map — my Shoutcast-listener-tracking geo-positioning statistics-gathering data sculpture back-end engine. i hear that the kids call them things mash-ups. the geo-lookup APIs were the most delicate part, and it seemed to work with them omitted. that red herring aside, it turned out to be the IP address resolution

that block was just a couple of lines until i’d added all the logging and desperate Exception handling. the app launches a lot of threads, so tracking down the issue was annoying … but eventually there it was in the traces. the stack just terminated when the app tried to getHostAddress (not during getByName though … must be a lazy-loading thing)

so i nearly had it all tracked down to that. then Tomcat inexplicably became unable to find basic JARs in /usr/share/java — i was using Fedora 8’s RPM version vs. raw Apache, and it’s organized real funny-like

so i threw up my hands and started up a fresh instance of my webapp AWS image. i’d rebooted the existing one, and that hadn’t helped at all. of course, the issue magically disappeared on the new instance. did anyone see that coming from a distance? ya probably did. cuz it’s ironic. and it’s the title line of the damn blog post

the hostname resoultion is surely a low-level OS thing. both Linux JavaSE 6 and IcedTea 7 just shat the bed when they got to that point, unlikely unless they both leveraged the same lib call. something must have gone wonk in the virtualization, and apparently a key part of the solution was running it inside of a different farm. i wasted a helluva lot of time to find that out

lesson !! if weird inexplicable freaky-ass things start happening to your cloud-hosted app, load it up on a new VM earlier than later. i’d taken a late-stage backup of the failed instance and assumed it would be corrupted with the Mystery Bug (read as: a waste of time to attempt). but oh no, it worked just great :) . next time it’ll be a cinch to just bundle the instance up, image it, and use it to launch a new one

and ultimately … it wasn’t a bug in my code !!!

being too rapid on the things that matter

Thursday, May 7th, 2009

it took me a while to come up with the title for this post. and it’s and Opinion Piece, not Techincal … so you’ll see why …

i’m working for a new company now, and they’re rocking it for RoR apps on the iPhone. sounds like a good place to be. one of the many reasons why this position works for me is because these guys are all about GTD and getting it out there. lean ‘n’ mean

whereas i’ve become very used to a holistic detail-orented, wisened test-backed process. great for Enterprise, but not so much for the reckless streets of Startup 3.0 . so i’m in a learning process. i’ve turned around some good stuff quickly, and it’s very satisfying

but i’ve screwed the pooch twice since i’ve been there. it’s totally a judgement call thing — i’m shooting too fast from the hip, and don’t feel like i really grasp the balance here …

first project i worked on was related to account management. they wanted a quick turn-around, i gave it a shot, had the whole thing backed with solid testing, and ready for on-time deployment with a smile. and in trying to keep track of all the new system permutations — i’d been there 2 weeks or so — i forgot one basic thing, and forgot to test for another. a nice little Perfect Storm. one emergency 1am database rollback later, we had a load of pissed customers and a helluva lot of explaining to do

so, then this past week, i went in to fix a minor rounding issue bug. those can be touchy. the right way to do it is with BigDecimal. yep, i’ve done that in Java too with BigDecimal. overall, it’s somewhat ponderous, detail-oriented, and can easily be polluted with Floats and the like. so i’d taken a shortcut, realizing that the low-level C impl was doing String conversion without the rounding issue. so i took the low-hanging fruit:

total.to_s.to_i

awesome !!1!. well, that is until you get into the 100-of-trillions area, otherwise shown as 1.0e+14. guess what happens when you parse that into a Fixnum? no database rollback this time, but Da Boss had to spend days sorting out the visceral impact of ridiculous sums of bogus exploit money pouring into our RPG

security, privacy and account management. payment calculations. not the sort of things to take shortcuts on. yet, if you’re embracing a culture that wants it done quickly and with minimum impact, it’s a risk you might be willing to take. it’s not like i didn’t have test scripts … i just forgot to head into scientific notation territory. just like i forgot to check for the implication of null password acceptance ( long story there, special account cases, etc. )

i’m putting these things up here for my fellow developers to laugh at.   “I mean, c’mon. All that’s totally obvious stuff.”   “I’d never miss that, that’s sophmore shit.”   good, get it out of your system, laughing boy

but believe me, when you’re on the other end of it, and had been in the middle of it and all full of all the other things that you needed to keep track of at that time, heh, well, that’s when you’ll really need to keep yerself laughing :)

When Broken Toys Impact your Friends

Friday, February 6th, 2009

This sure was an interesting morning! I woke up to find that I’d unintentionally sent direct messages to all of the followers on my personal Twitter account. And I’d sent them out at 1a PST, which means that anyone who (a) uses SMS capabilities, and (b) has some text message notification sound set up would have been rudely interrupted in the middle of the night.

Fortunately, I haven’t lost any followers (yet). But this was a perfect case of how mixing business with pleasure can have unintended consequences.

What Have I Learned

Or rather, what have I re-learned

Soft-disable features in Production at Launch Time
 
My Twitter engines are built with both an :enable_tweet and :enable_greeting config setting. In the git repo, they’re both true. When I did my local testing, I’d disabled them correctly. When I launched in Production, I neglected to make the quick-and-dirty changes; after all, everything worked great. And once started, my scripts correctly responded to the no-initial-state condition, and greeted everybody.

Launch preparation is critical, even for little projects. The start-up mentality is to move fast and lean, but there’s such as thing as too fast, and probably as too lean too. Gradual uptake migration is a wise strategy even for the ‘little things’.

Mock and Integration Testing Only Gets You So Far
 
I used rspec to mock out the full capabilities of the engine. Found some real-world issues, resolved them. I also wrote some core integration tests, ran them locally. Immediate failures. I had mocked documented features that didn’t actually exist. Fixed, re-mocked, re-tested, fixed again, etc .

Another great reminder that you can only mock something you trust, and how can you trust something you haven’t actually run under integration conditions to start with! Re-tested integration, and everything passed with flying colors. Sure, the features worked great now! And when I launched them, they did exactly what I asked.

So, as if we haven’t heard it enough times, be careful what you ask for!

All of this is familiar to anyone who has made a mistake in the software industry. It’s not like I haven’t successfully executed dozens of critical launches in the past, and most with virtually no issues at all. But what’s interesting is what happens when these mistakes happen in a public forum, and whom you expose them to — say, your friends :)

And who can say when two ounces of caution is more deserving than one … without the benefit of hindsight.

Just ask anyone who has a stringent backup policy how much time & effort they invest to avoid an event that may never actually happen. That stringency usually comes from that one unforgettable experience, and from there is born an extra layer of caution, and an additional time-sink (eg. mock & integration testing)

Heh. KISS. So, what exactly is simple? DRY. Isn’t that supposed to be a time-saver? Well, it depends on what you’re not trying to repeat. Strange how these cuddly and liberating acronyms can have more than one interpretation.

Experience taints everything.