Full Disclosure; or, What’s in your toolbox?

The one objection I haven't heard yet to my monkey patching rant is: you're a “hypocrite”:[http://utilitybelt.rubyforge.org/svn/lib/utility_belt/convertable_to_file.rb]!

Gasp, yes, I have written code that exploits Ruby's open classes. It even extends @Object@, the core-est of the core! And then shamelessly contributed it to a publicly available gem!

As anyone who read past the title of my article should know, I don't think monkey patches and open classes are unmitigated evil.  I even use them myself.  In this case, I happen to think it was justified. UtilityBelt is essentially a collection of MonkeyPatches. It's entire usefulness lies in providing convenience methods that require as few keystrokes as possible, since they will be used from the command line. And it is intended for a very special environment – IRB. Many Ruby hackers have hacks in their .irbrc that they would never include in a production application.

So does this mean I think that I should have power of approval over when monkey patching is justified?

No. Ruby is a powerful language. That's why I use it. That's one of the reasons that when I set out to find a new job, I looked for one where I could use Ruby every day. And with power comes “danger”:http://www.jamesbritt.com/2008/2/25/monkey-poaching-is-distracting-ruby.

But because I work with Ruby every day, on large, real-world projects that use a lot of third-party libraries, I also come in daily contact with the annoyances and hangups that are caused by the misuse of that power. And lately I've been noticing that the majority of those irritations stem from misuse of open classes, and that the majority of those misuses are completely avoidable.

I'm not going to tell people what features they can and can't use. But I'm also not going to hold back from commenting when I see a clear trend emerging, a trend with a completely predictable outcome. Pile all of your extensions from a myriad sources into a few pitifully overburdened classes, and there will be collisions, which will be a pain to track down when they start causing inexplicable bugs. And all of this will have been, once again… completely avoidable.

I'm not out to stifle anyone's “creativity”:http://weblog.raganwald.com/2008/02/1100inject.html. I don't believe, however, that there's necessarily any tension between promoting creativity and promoting wise practices.

All I'm saying is this: before re-opening a class, did you go through the rest of your toolbox first?

Followup to “Monkeypatching is Destroying Ruby”

My last article was intentionally provocative, and my expectations of response were exceeded. The “ruby-talk thread”:http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/292269?292062-292516 was full of good comments. I'm appreciative of everyone who has taken the time to respond either there or here on the blog.

I thought I'd write a quick followup and just respond to a few of the points that came up repeatedly in the ensuing discussion.

h5. “This is a Rails problem, not a Ruby problem.”

I've been coding in Ruby for something like seven years now. When I started ruby-talk was the only game in town, and Rails was years away. Now I work at a Rails shop now, and I'm the self-appointed “Ruby geezer” – I'm the guy who corrects people when they complain about some “Ruby” feature which is really a Rails extension.

I know the difference between the Rails community and the Ruby community. And the fact of the matter is, these days most of the payed work being done in Ruby is Rails work. For better or for worse, the coders of tomorrow's Ruby community are coming to it from Rails. Rails cultural problems will increasingly be Ruby cultural problems.

h5. “Monkey patching is a powerful and useful technique”

No argument there.  What I am calling for is that it be used sparingly, judiciously, and only when there are no other practical options.  In particular, I'm discouraging the use of monkey patching as a standard mechanism for extending core or third-party libraries – especially when those extensions will be packaged and distributed.

h5. “Bad code has always been with us.  There's nothing special about monkey patching”

What is special about monkey patching is that right now it is being used as a de-facto standard mechanism for class extension in gems and plugins,  to the exclusion of less invasive techniques.  Numerous gems add functionality to core classes like @Object@.  In some cases it's warranted, but in many it seems to be more because we can than because it's called for.  And in Rails, it is simply assumed that if you're going to extend, say, ActiveRecord with a plugin, you're going to do it by monkey patching @ActiveRecord::Base@ or some other core Rails class.

As an thoroughly modern OO language, Ruby offers almost an embarrassment of riches when it comes to  methods for extending functionality while maintaining encapsulation.  Inheritance, mixins, rediculously easy delegation.  Using monkey patching to extend classes throws all of that away, and takes us back to the C days of working all in one big undifferentiated namespace.  Except in C if you were lucky the compiler would complain when you redefined an existing symbol.  Ruby will happily let you override methods and overwrite instance variables without a peep.  This, of course, is part of the power of Ruby.  Power that should be reserved for prototyping and for special cases, not for everyday plugins and libraries.

h5. “Where's the code?”

This is a legitimate criticism. So far I've talked a lot about the evils of rampant monkey patching, and I haven't shown any code, either to demonstrate what not to do, or to demonstrate alternatives.  This is almost unforgivable on a programming blog.  I'm planning on remedying that ASAP.  I'm working on a series of posts covering alternatives to monkey patching.  Stay tuned.

Monkeypatching is Destroying Ruby

(The title of this post is intended to be deliberately provocative, as well as being a nod to Steven Colbert's “The People Destroying America” segments. It's provocative because I want to get people talking about this issue.  I don't actually think that monkey patching is “destroying” Ruby, but I do think the proliferation of the technique has real and troubling implications for Ruby's future.)

“Monkey patching”, for anyone who doesn't know, refers to the practice of extending or modifying existing code by changing classes at run-time. It is a powerful technique that has become popular in the Ruby community at least in part because the Ruby language makes it so easy. Any class can be re-opened at any time and amended in any way.

I believe the term first arose in the Python community, as a derogatory term for a practice which that community tended to frown on. The Ruby community, on the other hand, has embraced the term and the practice with enthusiasm. I'm starting to think that the Pythonistas' attitude may have been justified.

Here's what crystalized it for me. The other day I wrote a small Rails plugin (“NullDB”:nulldb). It was inspired largely by another plugin, “UnitRecord”:unitrecord. UnitRecord is by “Dan Manges”:[http://www.dcmanges.com/], a talented Rails developer whom I have a lot of respect for.

UnitRecord is implemented almost entirely as a set of monkey patches. When invoked, it dynamically modifies several standard Ruby and Rails classes, including @ActiveRecord::Base@, @Test::Unit::TestCase@.  As a result of this implementation, it is tightly coupled to the inner workings of ActiveRecord.  A small change to Rails and it could cease to work, and such a failure would be difficult to debug. Indeed, one of the reasons I decided to write “NullDB”:nulldb was because of just such a failure.

In writing “NullDB”:nulldb, I discovered that I could achieve the same functionality without resorting to monkey patching.  Instead of modifying existing classes, it implements the Rails “Database Adapter API”:[http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/AbstractAdapter.html].  The finished library is shorter than “UnitRecord”:unitrecord, is composed entirely of implemetations of public APIs, and contains zero monkey patches.  The code is also easier to understand, in my opinion, because of the lack of metaprogramming.

Now, I did not write this post to gloat about how my library is better than Dan's.  As I said before, I have a lot of respect for Dan.  He's a smart guy; I've learned a great deal from “his blog”:[http://www.dcmanges.com/]; and it's safe to say I would be a poorer Rails programmer if it weren't for hm.

And this is really the point.  Monkey patching is the new black.  it's what all the hip kids are doing.  To the point that smart, experienced hackers reach for a monkey patch as their tool of first resort, even when a simpler, more traditional solution is possible.

I don't believe this situation to be sustainable.  Where I work, we are already seeing subtle, difficult-to-debug problems crop up as the result of monkey patching in plugins.  Patches interact in unpredictable, combinatoric ways.  And by their nature, bugs caused by monkey patches  are more difficult to track down than those introduced by more traditional classes and methods.  As just one example: on one project, it was a known caveat that we could not rely on class inheritable attributes as provided by ActiveSupport.  No one knew why.  Every Model we wrote had to use awkward workarounds.  Eventually we tracked it down in a plugin that generated admin consoles.  It was overwriting @Class.inherited()@.  It took us months to find this out.

This is just going to get worse if we don't do something about it.  And the “something” is going to have to be a cultural shift, not a technical fix.  I believe it is time for experienced Ruby programmers to wean ourselves off of monkey patching, and start demonstrating more robust techniques.

I see the problem as one of convenience: sure, Ruby provides us with an immense toolbox, bigger than that of most other languages. But we're like lazy carpenters: sure, we have a whole garage full of tools, but there's a hammer laying right there on the floor next to us, and it's easier to reach for the hammer instead of getting up and fetching the right tool for the job.  But those tools do exist.  Indeed, most of the patterns in the “software pattern literature”:[http://c2.com/ppr/index.html] are inspired at least partly by the need to manage extension in robust and maintainable ways.

I look at the Emacs community as an inspiration.  Emacs, as many of you probably know, is essentially just a lisp machine.  Emacs Lisp, the language it is written in, is every bit as dynamic as Ruby.  Functions can be replaced at any time, by code in any package.  And because all Emacs Lisp code is intended to extend the Emacs editor, it is not uncommon for hundreds of different Emacs Lisp packages to be running in the same process, all coexisting and interacting. And yet, for the most part this melting pot of extensions all function together smoothly and without breakage.

Why?  I think the biggest reason is community conventions.  I've read a lot of ELisp code, and I have very rarely seen the equivalent of Ruby-style monkey patching.  Even the @advice@ mechanism, an AOP(Aspect Oriented Programming)-like feature which enables functions to be dynamically wrapped and chained, somewhat like Rails' @alias_method_chain@, is used sparingly.  Instead, every mature Emacs extension exposes a plethora of “hooks”, extension points that other packages can attach their own handlers to.  Other packages add their handlers to these hooks, and to hooks that the core Emacs code provides, and thus they cooperate largely without collisions.

Hooks are just one of the techniques available to us for robustly handling extension in a language as dynamic and powerful as Ruby.   Another is using the dependency injection style[1] to decouple classes from each other and allow third parties to substitute their own classes without monkey patching.  Another un-sexy but important practice is simply writing clear and comprehensive documentation of our classes' APIs and extension points.

But the most important thing we can do is set an example.  This is my call to action, my line in the sand: as experienced Ruby programmers, let us demonstrate solid, sustainable practices in our own code.  Let us use monkey patching sparingly, as a tool for exploration and experiment, and only in production code as a last resort.  Let's produce APIs that are amenable to extension without having to be patched.  And let us begin to develop some community-wide conventions and tools for writing extensible classes.

fn1. UPDATE A commenter on ruby-talk felt that my use of “Dependency Injection” revealed an “Enterprisey” bias.  Let me be clear that when I say “dependency injection style”, I'm not talking about using elaborate DI frameworks.  I'm just talking about writing classes that allow their collaborators to be passed in, either via constructors or setters, rather than having hard-coded collaborators.

[nulldb]http://avdi.org/projects/nulldb/

[unitrecord]http://unit-test-ar.rubyforge.org/

Announcing NullDB 0.0.1

I spent the afternoon coding an alternative to the ARBS and UnitRecord database-elimination plugins.  Definitely scratching a personal itch, as I had a project where I wanted to use one of the above-mentioned libraries but i just couldn't seem to get them to work.  I'm releasing it in hopes others will find it useful too.

Here's the text of the README:

What

NullDB is a Rails database connection adapter that interprets common database operations as no-ops. It is the Null Object pattern as applied to database adapters.

How

Once installed, NullDB can be used much like any other ActiveRecord database adapter:

NullDB needs to know where you keep your schema file in order to reflect table metadata. By default it looks in RAILS_ROOT/db/schema.rb. You can override that by setting the schema option:

There is a helper method included for configuring RSpec sessions to use NullDB. Just put the following in your spec/spec_helper.rb:

You can also experiment with putting NullDB in your database.yml:

However, due to the way Rails hard-codes specific database adapters into its standard Rake tasks, you may find that this generates unexpected and difficult-to-debug behavior. Workarounds for this are under development.

Why

NullDB is intended to assist in writing fast database-independant unit tests for ActiveRecord classes. For why you would want to test your models without the database, see: www.dcmanges.com/blog/rails-unit-record-test-without-the-database.

NullDB was inspired by the ARBS and UnitRecord libraries. It differs from them in a couple of ways:

  1. It works. At the time of writing both ARBS and UnitRecord were not working for me out of the box with Rails 2.0.
  2. It avoids monkey-patching as much as possible. Rather than re-wiring the secret inner workings of ActiveRecord (and thus being tightly coupled to those inner workings), NullDB implements the same [semi-]well-documented public interface that the other standard database adapters, like MySQL and SQLServer, implement.
  3. UnitRecord takes the approach of eliminating database interaction in tests by turning almost every database interaction into an exception. NullDB recognizes that ActiveRecord objects typically can‘t take two steps without consulting the database, so instead it turns database interactions into no-ops.

One concrete advantage of this null-object pattern design is that it is possible with NullDB to test after_save hooks. With NullDB, you can call +save+ and all of the usual callbacks will be called – but nothing will be saved.

Limitations

  • It is not an in-memory database. Finds will not work. Neither will reload, currently.
  • It has only the most rudimentery schema/migration support. Complex migrations will probably break it.
  • Lots of other things probably don‘t work. Patches welcome!

Who

NullDB was written by Avdi Grimm <avdi@avdi.org>

Where

Professionalism

I've been thinking a lot about the subject of professionalism in the context of software development lately.  As I see it, there are two strains of professionalism.  There are, for lack of a better term, “professionalistic” developers.  To be professionalistic is to adopt the trappings of professionalism while missing out on the underlying attitude.  These are the developers who know all the acronyms and buzzwords and will happily tell their managers when asked that they are “87.3% complete” on their current project.

Then there is the other kind, the true professionals.  I don't feel I have a good enough handle on the topic to distill what it means to be professional  down to a few pithy phrases.  But I “know it when I see it”, as the saying goes.  I can begin to describe it by example.

Professionalism is…

  • Understanding that software is hard, but that the hardest problems you face are social, not technical, and do not have technical solutions.
  • Rewriting the cleverest, most difficult code you've written until it appears so trivial that any idiot could have written it.
  • Asking your colleagues for feedback even when you are sure that you are right.
  • Answering emails as soon as you get them, even if it's just to say “I don't have time to look at this right now”.
  • Explaining a solution in simple terms, even when you know all the buzzwords and academic terms for it.  And even when you're certain that your audience ought to know those terms.
  • Evaluating every potential change in terms of business value and risk as well as elegance and programmer happiness.
  • Proving your point with code and statistics, rather than rhetoric.
  • Not spending hours writing code and collecting statistics, just for the sake of proving a point.
  • Asking questions even when you think you know the answer.
  • Investing constantly in your knowledge portfolio, whether your job forces you to or not.
  • Going out of your way to help n00bs, without behaving as if it's a chore.
  • Researching alternatives, but time-boxing your research.  Forcing yourself to stop at the end of the allotted time even though you are oh so close to the perfect solution.
  • Joining your coworkers for lunch periodically.
  • Following, contributing, or taking charge, rather than arguing.
  • Discarding your faith in your own gut-level estimates, and measuring your performance empirically.  Then adjusting your estimates accordingly.
  • Rigidly controlling the scope of changes.  Resisting the urge to rewrite for the sake of rewriting.
  • Reporting actual project status,not what you think your manager wants to hear.
  • Thinking of the customer as a an asset, not an enemy.
  • Recognizing that you are not only fallible, but probably overlooking something vital at this very moment.
  • Not expecting the world to fall neatly into categories, systems, and procedures the way code does.
  • Keeping your perspective.
  • Staying organized.  It doesn't matter so much what system you use, whether you're a Franklin-Covey type or a GTD nerd or just keep copious notes in a spiral-bound notebook.  Knowing what you were doing yesterday, what you are doing right now, and what responsibilities are currently on your plate is one of the true marks of a professional.
  • Turning off your workstation and going home at the end of the work day, whether the problem is solved yet or not.

I'm sure I could think of many others, if I kept at it.  I'm no paragon; I've been guilty of violating all of these.  Some of them I still violate on a regular basis.  Part of my goal in writing this list is to take a look at my own comportment and as a developer and discover where I am falling short.

Utility Belt

Utility Belt is a grab-bag of IRB extensions to make your Ruby REPLing easier.

It's worth the install just for the edit command, which allows you to shell out to your favorite editor in order to edit more complex input.  But there's a lot of other good stuff there too.

Virtuous Language Design

Eric.weblog() reminded me of an excellent interview with Anders Hejlsberg on generics in C#, Java, and C++. Anders is the kind of person you want in charge of designing a language. In the interview he demonstrates a deep knowledge of the how competing languages implement generics, and the implications of those design decisions. While he is obviously biased towards his own choices, his biases are clearly based in a solid understanding of the alternatives. Matz, the creator of the Ruby language, has a similar grounding; in his communications on the Ruby mailing list he demonstrates an almost encyclopedic knowledge of other programming languages past and present. This contrasts starkly with the attitude I have occasionally seen from certain other language camps (“I don't know how the other guys do it, but it can't be better than our way…”).

Eric‘s entire series on moving from C# to Java is worth reading, BTW.