The Trifecta of FAIL; or, how to patch Rails 2.0 for Ruby 1.8.7

It's an oft-stated fact that most disasters result not from a single point of failure but from a combination of failures reinforcing each other. I wouldn't term the problem I ran into last Friday a disaster, but it certainly cost me several hours of time trying to find a workaround.

Culprit #1: Rails

Rails' ActiveSupport added a handy little method called #chars to the String class. In and of itself this doesn't seem like such a bad thing, and a lot of other handy methods in ActiveSupport are built on top of #chars. However, as we'll see, taking advantage of Ruby's open classes to extend core types has a way of drawing the unwanted attention from the Law of Unintended Consequences.

Culprit #2: Ruby

It's not set in stone anywhere, but there's a fairly well accepted convention in open source projects that versions are divided into a major version, a minor version, and a tiny or patch version. New major versions indicate API-breaking changes. A new minor version may introduce new features, but existing code should continue to work as-is. And a new tiny version indicates that the API remains fixed; the only difference is that bugs have been fixed and security holes patched.

Ruby 1.8.7 is a minor release masquerading as a tiny release. Among the features backported into 1.8.7 from Ruby 1.9 is a new #chars attribute. Unfortunately, it is incompatible with the Rails 2.0 implementation of #chars. This, incidentally, is a prime example of one of the subtler ways that patching the core classes can bite you. Even if you are adding new methods rather than re-writing existing ones, the chances are good that someone else will have the same idea only with a slightly different implementation and semantics. Bang, incompatibility.

Culprit #3: MacPorts

We have an app which has not yet been ported to Rails 2.1. This, in itself, would not have been a problem; we can keep running it under Ruby 1.8.6 with Rails 2.0, no problem. However, I have a nasty habit of trying to keep my software up to date. So I run sudo port upgrade outdated periodically, and watch all the errors from unmaintained ports go scrolling across my terminal for 24 hours or so.

The last time I did this, one of the ports that did manage to build was Ruby. Version 1.8.7. The next time I ran our app, it of course promptly crashed.

This is the point at which I discovered something I hadn't realized about MacPorts: it has no downgrade path. Coming from the world of Debian, Ubuntu, and apt-get, I just expected any package management system to handle the case where the user specifies an older version to be installed.

In fact, there's a way to do it in MacPorts, but it's painful.


fail owned pwned pictures
So there I was with a broken app, no time in the iteration to upgrade it to Rails 2.1, and no easy way to get back to Ruby 1.8.6. Lame.


After bitching and moaning on Twitter for awhile, I decided Bob helps those who help themselves, so I took a look at the crash backtrace I was getting. I traced it back to a line in vendor/rails/activesupport/lib/activesupport/core_ext/string/access.rb:

In the Rails 2.0 version of String#chars, #chars returns an Array or Array-like object which can be subscripted with #[]. The Ruby 1.8.7 version, by contrast, returns an Enumerable::Enumerator.

“That's easy enough” thought I, and, fully expecting that patching this one issue would just reveal another incompatibility, and another, and another…, I changed the code to:

Lo and behold, the app worked perfectly.

Of course, YMMV. But as a quick kludge this one was surprisingly painless.

Lessons Learned

Here's what I took away from this experience:

  1. Be wary of adding methods to core classes. What could possibly go wrong? More than you think.
  2. Patch releases should be true patch releases. It's tempting to include a neat new feature as a bonus –
    again, what could possibly go wrong? Resist this urge.
  3. Macs are shiny, but for industrial-strength development support, nothing beats a Debian-based system with APT.
  4. Every now and then taking a clawhammer to vendor code is the shortest (short-term) way from point A to point B. Personally I prefer to either keep this kind of change local or, if necessary, version it with something like Piston, rather than maintaining it as a monkey-patch.

Phenomenal Cosmic Power

Jeff Atwood thinks monkeypatching should frighten you:

But if wielding that power doesn't scare and humble you a little, too, then maybe you should leave the monkeypatching to the really smart monkeys.

Reg Braithwaite thinks it should exhilarate you:

And ultimately, that is what this line of code says to me about Ruby. It says that this is a language where the fringe is inventing new things. And to embrace ruby is to embrace the idea of a language being propelled by its user base.

I'm okay with either reaction. Metaprogramming may be pure and godlike, but then again, thou art God. It's the people who have no reaction whatsoever that scare me. Use of power should be conscious and intentional. Otherwise you're like a woodworker who addresses a mismatched joint by just hammering harder.

Revision-Stamp Your Rails Apps

One of my biggest pet peeves in doing QA is when I spend half a day trying to track down why a bug I thought I'd fixed has cropped up again, only to discover that the reporter was looking at an old version of the application. Good revision tracking is vital to doing efficient QA. Here's a little trick for removing any uncertainty about what version of the app you're looking at.

Capistrano, after it exports an application from an SCM repository (such as a Subversion repository), writes an extra file to the application's root directory. The file is named “REVISION”, and, as you might imagine, it contains the revision number that was exported. Displaying this information in your application's UI is straightforward. Here's the addition I made to a view partial that is displayed on the application's home page:

And here's the app_revision helper method:

Of course, this isn't very efficient – it will re-load the file every time a page is rendered. Optimization is left as an exercise to the reader. But it accomplishes its purpose: now whenever someone reports a bug in the app, I can ask them to go to the application home page and tell me what the revision number is. No more trying to remember whether I pushed out the latest changes to the beta server.

Epic Fail, or, why the users hate us.

[I'm reposting this from my “personal journal”: as it has development applicability – in so much as it is a tale about how NOT to design software.]

So I've been trying to wire money to Kenya. No, I haven't been contacted by the wife of a former dictator who needs my urgent assistance in moving large sums of money. Just doing someone a favor.

I go to the website. I've used it before, so I figure I'll log in to my account. Try to log in with my usual credentials. Uh oh.

W0500 Sorry you're having trouble. Please try again later.

This does not bode well. I'm having trouble? News to me. And if the trouble is on my end, what good will trying again later do? Are they suggesting that I am having brain issues which will resolve themselves in a few hours?

OK, assume this is just their eccentric way of saying “login failed”. I have a dim recollection of not being able to use one of my regular passwords because they don't allow special characters in their passwords (why is it always institutions which handle large sums of money that have poor password policies?) Email customer service, and meanwhile, click the “retrieve password” link. Notice while emailing customer service the lengthy backlog of previous Western Union customer service mail from the last time I tried to wire money through them. Oh dear, I must have blocked those memories.

OK, well, chin up and keep trying. While waiting for customer service, hit the “forgot password” link. Go through the first step, email address, date of birth… that's odd. This is probably the first site I've ever seen where the “Date of Birth” field is a password-style field. Presumably so no potential identity thieves peering over my shoulder witness that top-secret information, and I have to go through all the hassle of changing my birth date to something new.

Anyway, moving on, ah, the Security Question.  OK, what's it going to be – good ole' mother's maiden name?  First pet?  Model of car in which I first got to third base? Nope, not good enough for Western Union security.  They've gone the extra mile and required me to guess the question before supplying the answer. Now that's secure!

Email customer service again. Looks like there's a response to my first query already.  They are prompt, I'll give them that.  Let's see…

We apologize for the inconvenience, due to security reasons the system blocks the access after 2 or more unsuccesfull attempts.

Reply to customer service informing them that that's nice, but this happened the very first time I'd logged in in months.

Oh look, a reply to my second problem.  They sure are on the ball over there.

We apologize for any problems you experienced with our site. However, if you are not being able to sign in with your current e-mail address and password, we may offer to deactivate your account so you may register again. Please keep in mind that no one at Western Union knows or can retrieve your password for you. If you wish to have your account deactivated, please send your request in an email to from the email address you used when you registered, and include your registered name, address and telephone number. Please note that if you have another e-mail address, you may use it to create a new account.

Ah, form letters detailing how to work around the flaws in the system.  Always a promising sign.

OK, let's try setting up a new account under a different email address.  Let's see, click “Sign UP”, enter name, rank, and serial number.  Like nearly every user interaction page in this system, the page has a CAPTCHA at the bottom.  Except this CAPTCHA has a difference: there's no CAPTCHA, just the alt text.

Click little “reload” icon, no good.  Click the “audible captcha” icon – leads to a blank page.  I think they've hit upon an unbeatable anti-spam strategy here – a CAPTCHA you have to interpret via mental telepathy.

Deep breath.  Let's take a time out and check our email, shall we?

Unfortunately, your e-mail has arrived blank.  Please resubmit your question or comment and we will be happy to assist you. Also, please keep in mind that we are unable to view attachments, enclosures, or hyperlinks.

Looks like they left something out of that list of things they are unable to view – replies to their own useless boilerplate messages.  It appears that Western Union has adopted the time-tested “hear no evil, see no evil”  approach to technical support.  If the techs can't view the problem, it doesn't exist!

While we're checking email, let's email them about their amazing vanishing CAPTCHA.  Prompt reply, as usual.

Usually when this happens, it is because of problems with the server. You may wish to double-check your Internet settings, or try again later. Please note that our site works best with Internet Explorer version 6.0. We are currently not compatible with Mozilla, Firefox nor Safari.

Western Union, after doing a costly internal audit to discover where their inefficiencies lay, discovered that 99% of their problems with online transactions stemmed from those annoying customers.  Excluding 20-30% of the web browsing population by limiting the web site to IE6 compatibility neatly alleviated the problem, and the assistant vice-president who thought of the idea was immediately promoted.  He is now working on a grand enterprise-wide plan to enhance the customer experience by requiring all requests to be submitted on Sumerian cuneiform tablets.

So let's review.  I can't log in.  I can't reset my password. And I can't register a new account.  Well, we gave it our best shot. Time to fall back on ye olde reliable wireless telephone.

Call Western Union… you can probably guess by now where this is going.  I'll spare you the play-by-play and sum up.  I called, I punched in my information. I was put on hold for 15 minutes while screechy distorted musaak drilled slowly into my ear canal.  Then they disconnected me.  I called back and discovered that while I was on hold they had helpfully cancelled my transaction.  I spoke to a manager, re-started the transaction.  Gave them all my information over again, then sat on hold for another ten minutes or so.  Finally, after a total of 50 minutes on the phone, they informed me that my transaction had been rejected by the ineffable machines which sometimes grant, and sometimes deny, and no one can say why… for security reasons.


Does anyone know of a method for transferring funds overseas that doesn't leave one with an urge to bash one's forhead repeatedly into a concrete post?

You should be on ruby-talk

Working as I do in the Rails world these days, I'm periodically reminded of the difference between me and most Rails programmers. That is, the fact that I came to Rails via Ruby, rather than vice-versa. Usually this happens when someone at work or in the blog world expresses delight (or perplexity) about some Ruby feature that I thought everyone knew about. This then prompts me to put on my best “old coot” voice and ramble on about “young whippersnappers” with their Rails and their fancy conferences and their big pants and… get off my lawn, ya darned kids!

One of the biggest disconnects for me is the fact that almost no one I know in the Railsverse reads “ruby-talk”: regularly. Having gotten into Ruby somewhere between five and seven years ago, it's difficult for me to imagine being a part of the Ruby community and not being privy to the discussions in ruby-talk. Ruby-talk was one of the first programming communities I ever regularly participated in (the other one was the Pragmatic Programmer mailing list). It was a warm and welcoming place compared to, say, the Perl communities of the time; full of the spirit of joyous discovery and show-and-tell that this amazing new language from Japan tended to inspire.

The S/N ratio has declined somewhat since some joker hooked up a conduit between “ruby-forum”: and the mailing list, leading in a steady stream of Rails newbies who have it confused with the RoR forum. But it's still one of the friendliest and most helpful programming communities out there. And nowhere else – not on blogs, not on Reddit, nor on Twitter – will you find better and more thoughtful discussions on Ruby style, idioms, alternate ways of accomplishing tasks, gotchas, and just plain fun Ruby tricks. And, because new Ruby libraries and frameworks are usually announced there, it's also a great place to find out about what's going on in the wider Ruby universe – beyond Rails, beyond web development entirely.

I'm going to make a bold statement here:

If you are a developer working in Ruby, you should be reading ruby-talk.

No, let me amend that:

If you are a developer working in Ruby, you should be -reading- contributing to ruby-talk.

Because apart from being a great place to pick up new techniques, it's also one of the easiest opportunities to give back to the Ruby community. Check in from time to time, and if a newbie asks a simple question you know the answer to, help them out! It takes five minutes, and it helps perpetuate the culture of supportiveness that is a big part of what makes Ruby so special.

It's easy to get involved with ruby-talk. There are multiple interfaces. There is the “mailing list itself”:, which is how I interact with it. Then there is the mail-to-news gateway, which enables you to read it as the Usenet newsgroup comp.lang.ruby. You can get to it through “Gmane”:nntp:// if your ISP doesn't provide NNTP service. Or you can interact via the “Google Groups frontend”: And finally, there's the aforementioned “ruby-forum gateway”:

Whichever route you choose, I hope to see you on ruby-talk soon!

Everything Old…

A passage in Charles Nutter's reaction to MagLev caught my eye today:

First off, they demonstrated its distributed object database automatically synchronizing globally-reachable state across multiple VMs. It's an amazing new idea that the world has never really seen…
except that it isn't. This is based on existing OODB technology that Gemstone and others have been promoting for better than a decade. It's cool stuff, no doubt, but it's been available in Gemstone's Smalltalk product and in their Java product for years, and hasn't seen widespread adoption. Maybe it's on the rise, I really don't know. It's certainly cool, but it's certainly not new.

True enough. But what really is new in software development, anyway? Quite a few people have noted that the feature sets of modern programming languages are gradually converging on the feature set of Lisp, a language that is over fifty years old. Ruby itself has been around as long as Java, believe it or not. And Ruby is essentially a Perl-ish syntax veneer over Smalltalk, a thirty-year-old language.

Why does Ruby seem to be succeeding in ways that Smalltalk didn't? Well first of all, Smalltalk did have a fair amount of success in it's time. It remains to be seen whether Ruby will have better staying power. My guess, though, is that it will. The momentum Ruby has accumulated is pretty impressive.

Why? I can think of a lot of possible reasons. Ruby came of age in a time when Moore's law had finally made the trade-off in increased development productivity versus relatively slow execution time economically justifiable. The primary Ruby implementation was free and open-source, not proprietary. For that matter, Ruby matured during a time when the Open Source community was exploding into relevance and even dominance, as opposed to the proprietary dominance of previous decades. And Ruby arrived at an inflection point in software development, a time when developers were still finding their legs on a new platform (the Web); and a new Web framework using the power of Ruby to accomplish Web programming tasks more easily had the opportunity to revolutionize the industry.

It could have been some combination of the above, or it could have been other factors entirely. Whatever the reason, a ten year old language based on thirty year old technology suddenly took off. Now turn to the field of data storage: developers are looking at novel new(?) document-oriented systems like CouchDB and StrokeDB, despite the fact that SQL is The Standard. Coders primed by Ruby and Rails to look to unfamiliar sources for solutions are questioning the conventional wisdom in other areas as well. Maybe it is time to reconsider the much-maligned OODB as well.

I guess the point I'm getting at, if there is one, is that sometimes the best new idea is an old one reconsidered in the light of current needs.

The Lotus Desk

I've been increasingly dissatisfied with office chairs both at the office and at home lately. I just can't seem to find a configuration that gives me the back support I need for hours-long coding sessions. I've considered joining the standing-desk crowd but I'm not sure that would be any more comfortable.

If there's one thing I've learned from meditation it's that the lotus (or half-lotus) position is remarkably stable and conducive to good posture over extended periods of time, at least for me. Combine that knowledge with a beautiful sunny day, and voila, the Lotus Desk. I still need to work out a few kinks, but I think I could get used to this. I wonder if I can find someone to do some custom woodwork for me…

Rock’em Sock’em Ockham

Inspired by “this talk by Jim Weirich”:


Image by “scottfeldstein”:, “some rights reserved”:

(For the shaving nerds, this poster features the Merkur Hefty Classic, a razor I recently acquired and am so far pretty happy with.)

On Beauty in Code

I was thinking about the topic of beautiful code this morning. There's a lot of disagreement about what constitutes beauty in code. I've watched Marcel Molina Jr. talk about “Plato and Pythagoras”:molina. O'Reilly has published a “whole book on the subject”:oreilly. On the other hand, “Jeff Atwood thinks that there's no such thing.”:atwood. I disagree with Jeff on this – I definitely think there is such a thing as beautiful code. But I'm not sure if my idea of what makes code beautiful is the same as others'.

What do I even mean when I say something is “beautiful”, anyway? Beauty is in the eye of the beholder, after all – there is no objective standard. For my purposes, I consider something beautiful when it triggers a certain emotional response. A flower is beautiful because I enjoy the simple act of observing it for its own sake.


When it comes to functional human-created objects, I've noticed that certain properties tend to trigger this emotional reaction. Here are a few products I find beautiful:

  • The “Mini Mag-Lite”:maglite
  • The “Fisher Bullet Pen”:fisher
  • “Chaco Sandals”:chaco
  • The “Apple MacBook Pro”:macbook

These are all products which are extroardinarily well engineered for their respective tasks. Out of that engineering emerges a kind of clean, austere beauty. I guess you could say my esthetic runs toward the “Bauhaus school”:bauhaus of thought.


By contrast, a lot of the code I have seen held up as “beautiful” has more in common with “MC Escher”:escher or a “Zen garden”:zen. In the case of the former, the beauty is all about intricate, brain-twisting “cleverness”:camping. In the latter case, “divine purity of expression”:fibonacci takes precedence over more worldly concerns.

To me, beautiful code – code that I can enjoy reading for its own sake – has the quality of expressing its function simply and clearly. Like a Mag-Lite, its form reflects its function in a way that is elegant, straightforward, and easy to grasp.


One of the projects that has recently impressed me as having this property in its code is the “Ramaze”:ramaze web framework. The Ramaze “source code”:ramaze_source is clean, straightforward, well-commented (but not excessively so). Most methods are only a few lines long, and the lines themselves are short. White space is in abundance, setting off stanzas of code. And the logic itself is usually easy to follow.

Routing is traditionally a thorny area in web application frameworks. Here is the Ramaze routing code, in its entirety (minus some documentation):

There is only one method of any length, @#resolve@. And that method has an easily recognizable cadence – if path matches some predicate, then perform some transformation, and return. Else move on to the next clearly-delineated stanza. This is another quality of beautiful code: each method body has a recognizable, almost archetypal “shape” which is not littered by special cases and digressions.


If you enjoy reading code, I recommend taking a stroll through the Ramaze “source code”:ramaze_source. It is very nicely presented online in a custom source browser. And most of it demonstrates a similar clean elegance to the code above.

p{clear: left}. So that's an example of what beautiful code means to me. What about you? What code do you consider beautiful?