So what’s the deal with Ruby refinements, anyway?

If you’ve been following Ruby developments for the past couple of years, chances are you’ve heard about refinements. You may have heard that they are controversial, confusing, or even “broken”.

It’s true that refinements had some growing pains in their early, experimental versions. But having spent some time exploring the feature as it now exists in Ruby 2.1 and onward, I think they’re a valuable addition to the language. In a language where any class can be re-opened at any time, refinements provide a nice mix of language malleability tempered by locality and visibility.

Back in October I released a video to my RubyTapas subscribers which set out to introduce and demystify this powerful new ability. Today I’m releasing it for free in hopes that it will help clear up some of the confusion around refinements.

Here’s the video:

By the way, if you want to see an example of a real-world use of refinements, check out the Sequel core_refinements library. Sequel has a set of core class extensions which are traditionally added globally. But with refinements, it’s possible to make use of these convenience extensions without either forcing or requiring them to be globally available. This is particularly handy within gems, where we might want the ease of use of the extension methods, but we don’t want to force global extensions on our library clients.

If you prefer reading over watching, the script is below. And remember, if you like this, please consider subscribing and supporting the creation of more videos like this!


 

In the last episode (#249), we developed some code for unindenting text inside of heredocs.

def unindent(s)
  s.gsub(/^#{s.scan(/^[ \t]+(?=\S)/).min}/, "")
end

module Wonderland
  JABBERWOCKY = unindent(<<-EOF)
      'Twas brillig, and the slithy toves
      Did gyre and gimble in the wabe;
      All mimsy were the borogoves,
      And the mome raths outgrabe.

    -- From "jabberwocky", by Lewis Carroll
  EOF
end

puts Wonderland::JABBERWOCKY

# >>   'Twas brillig, and the slithy toves
# >>   Did gyre and gimble in the wabe;
# >>   All mimsy were the borogoves,
# >>   And the mome raths outgrabe.
# >>
# >> -- From "jabberwocky", by Lewis Carroll

This method seems like a perfect candidate to be made into an extension to the String class. That way instead of wrapping the heredoc in a call to unindent() , we could append it as a message send instead.

JABBERWOCKY = <<-EOF.unindent
# ...

So let's go ahead and do that. We reopen the String class and define unindent() inside it. References to the method's argument become implicit references to self instead.

class String
  def unindent
    gsub(/^#{scan(/^[ \t]+(?=\S)/).min}/, "")
  end
end

module Wonderland
  JABBERWOCKY = <<-EOF.unindent
      'Twas brillig, and the slithy toves
      Did gyre and gimble in the wabe;
      All mimsy were the borogoves,
      And the mome raths outgrabe.

    -- From "jabberwocky", by Lewis Carroll
  EOF
end

puts Wonderland::JABBERWOCKY

# >>   'Twas brillig, and the slithy toves
# >>   Did gyre and gimble in the wabe;
# >>   All mimsy were the borogoves,
# >>   And the mome raths outgrabe.
# >>
# >> -- From "jabberwocky", by Lewis Carroll

Now, if you've watched episode #226, a little alarm bell might be going off in your head right now. In that episode I talked about how even when we add brand new methods to core classes, these methods are conflicts waiting to happen. This is doubly true of methods like this one: it is not only possible, but likely, that someone else will have the idea to add an #unindent method to String. And, in fact, I know of at least one Rubygem which adds exactly this method to String. If our implementation of #unindent differs slightly from the conflicting definition, then whichever one "wins" based on the program's load order will cause subtle and difficult-to-track-down bugs in the code expecting different semantics.

"But Avdi!" you might object. "I just won't include libraries that include conflicting definitions in my project!" The difficulty comes when some unrelated gem you need—for instance, an API wrapper around some remote service—has an implicit dependency on a gem that extends a core class with a conflicting method definition.

How can we be sure that our extension to String is the only one our code will use, while also ensuring that our extensions won't interfere with third-party code? This is a question which has vexed Ruby programmers for many years. And it has lead some of us to come to the conclusion that extensions to core classes—or any classes we don't ourselves own—are not worth the cost.

However, Ruby 2.0 introduced an experimental answer to this question: a feature called "refinements". In Ruby 2.1 it ceased to be experimental. In a nutshell, refinements are a way to limit the scope of an extension to a class to only the code we control.

Let's convert our extension to a refinement. But first, let's create a conflicting String extension. This definition won't actually unindent anything; it'll just return an obvious flag value to tell us when we are invoking the conflicting definition.

Moving on, we create a new module to contain our refinements, calling it StringRefinements. Then we move our extension—including the reopened String class—inside this module. Then we switch the String class definition into a refine declaration, including a do keyword.

At this point, we've declared our refinement, but it hasn't taken effect anywhere. Inside our Wonderland module, we add a using declaration, with the name of our refinements module as an argument. This tells Ruby that inside the Wonderland module, the refinments we defined should take effect. Remember that inside this module we call String#unindent, and we are hoping it will invoke our version of unindent.

Outside of the Wonderland module, we use String#unindent on another heredoc and assign the result to a constant.

With all that done, we then output the contents of our unindented string constant, followed by the contents of the second heredoc. The output tells the story: inside the Wonderland module, the refinements defined within the StringRefinements module took precedence. But outside that module, the global definition of String#unindent was in effect. Our string class extensions are no longer in conflict.

class String
  def unindent
    ""
  end
end

module StringRefinements
  refine String do
    def unindent
      gsub(/^#{scan(/^[ \t]+(?=\S)/).min}/, "")
    end
  end
end

module Wonderland
  using StringRefinements
  JABBERWOCKY = <<-EOF.unindent
      'Twas brillig, and the slithy toves
      Did gyre and gimble in the wabe;
      All mimsy were the borogoves,
      And the mome raths outgrabe.

    -- From "jabberwocky", by Lewis Carroll
  EOF
end

UNREFINED =<> Refined:
# >>   'Twas brillig, and the slithy toves
# >>   Did gyre and gimble in the wabe;
# >>   All mimsy were the borogoves,
# >>   And the mome raths outgrabe.
# >>
# >> -- From "jabberwocky", by Lewis Carroll
# >> Unrefined:
# >> 

It is important to understand that the effect of a using statement is strictly lexically scoped. To see what this means, let's reopen the Wonderland module and define another unindented string constant. Note that this time, we do not declare that we are using StringRefinements.

When we output the contents of the string, we can see that the unrefined definition of String#unindent was in effect. This is despite the fact that we declared we were using Stringrefinements in another definition of this same module. What this tells us is that declaring that we are using a refinement module in one location does not "infect" other code defined inside the same module. Just like local variables, the refinements are in effect only up to the end of the module block in which they are used.

class String
  def unindent
    ""
  end
end

module StringRefinements
  refine String do
    def unindent
      gsub(/^#{scan(/^[ \t]+(?=\S)/).min}/, "")
    end
  end
end

module Wonderland
  using StringRefinements
  JABBERWOCKY = <<-EOF.unindent
      'Twas brillig, and the slithy toves
      Did gyre and gimble in the wabe;
      All mimsy were the borogoves,
      And the mome raths outgrabe.

    -- From "jabberwocky", by Lewis Carroll
  EOF
end

module Wonderland
  TWINKLE = <<-EOF.unindent
    Twinkle, twinle little bat...
  EOF
end

puts Wonderland::TWINKLE

# >> 

And this is a very good thing. Refinements exist to address some of the confusing and surprising consequences of being able to extend any class at any time. The fact that refiements are strictly lexical means we cannot change the behavior of other code "at a distance". Anywhere that a a refinement is in effect, we will be able to scroll the file up in our editor and see that the refinement is in effect.

And that's it for today. Happy hacking!

[boilerplate bypath="rubytapas-sample"]

12 comments

    1. Three reasons come to mind why people are negative about it:

      1) Performance. Charles Nutter wrote a great summary of why refinements prevent a lot of optimizations. They are a significant slowdown.

      2) Encouraging monkeypatching, which many people are down on in the first place.

      3) It’s attempting to solve a complex, hard-to-reason-about problem with more complexity. This is related to things like another commenter’s observation that refinements don’t apply to indirect calls like send() and respond_to?(). It’s not clear exactly what the semantics should be in many places, and we’re adding another complex and varied scoping system on top of Ruby’s several existing complex and varied scoping systems (local variables, constants, class variables, etc.) which already have different rules from each other, and from refinements.

      To be fair, #3 is still a very Ruby way to solve the problem. We do a lot of solving complexity with more complexity. But at some point, that makes for a very unscalable language — you would never want to manage a single million-line program in Ruby, for instance, though Java and C do better at that. The debate is just about where that point should be.

      1. I still think that with regard to #2, refinements are far preferable to monkeypatching. And I’m in the camp that believes that despite the dangers, “making the language your own” is one of Ruby’s strengths. It’s also one of its weaknesses, but I think locking Ruby down would make it a language other than Ruby. We choose languages partly based on their spirit and features, and I think when we choose Ruby, we’re choosing malleability.

        Regarding performance: I’m certainly sensitive to the pain of implementers, as well as of early adopters of un-optimized features. But Matz has been clear on his intent for the language (and I have this straight from him): Ruby prioritizes making the developer’s life easy with language features. If a feature is slow, it’s the implementor’s challenge to find a way to make it fast. (This was in reference to dynamic #extend, which caused a lot of perf issues back when there was a global method cache).

        In other words, for better or for worse, where there is tension between a feature and performance, Matz would rather see that tension resolved by pushing on the implementation technology, not by pushing back on the feature. Plenty of other languages go the other direction, and that’s fine.

        BTW thanks for explaining why people are critical, that was needed in this discussion.

        1. My take would be something like “old Ruby tried for fully malleable, new Ruby tries for a reasoned balance of malleability and scalability.” Not performance — you’re right, Matz has made his stance very clear on that.

          But scalability in the sense that a program can get very hard to reason about very quickly, and Ruby does do a fair bit of trying to balance that. It just often does it by adding complexity, in the sense of adding new features (weird constant scoping, global variables, modules) which makes for a dizzying variety of potential interactions.

          Reg Braithwaite once summarized it as “Ruby: Programmer Happiness Through Featuritis.” The longer I use Ruby, the more I agree with him.

          1. Seems like a fair assessment. And to be honest, sometimes I just want parentheses and lambdas. Thank goodness for variety 🙂

      2. Thanks for explain all those points, that adds more light into the discussion.

        #1: As Avdi as stated in other comments, Ruby is not focused in Performance but in developer happiness.

        #2: I don’t think refinements encourage monkey patching, it just offer one more tool to do something similar but in a safer way. BTW monkey patching and refinements should be one of the last resources to solve a problem in my opinion.

        #3: This one is a really good point. Add more complexity to solve a problem is just one of many trade-off. I hope refinements will apply to indirect calls in a next release as the ruby docs point.

        1. For #2, refinements encourage a specific flavor of monkeypatching — which refinements essentially are. They limit the scope, but they also increase the perception of safety, which will result in more monkeypatching (if refinements count.)

          The scope is one problem with monkeypatching, but not the only one. So we’ll get less scope-related conflicts in monkeypatching, but more of the other conflicts (unpredictability, complexity, hard-to-look-up code and so on.)

          For #1, sure, Ruby doesn’t focus on complexity. But just as Ruby 1.9 took a very long time to be adopted (but not as long as Python 3!), later versions of Ruby, if they become sufficiently unusable, will have trouble being adopted. It would be lovely not to go through that again, you know?

          Fixing the indirect call problem is interesting. It’s not entirely clear what refinements should apply to. This is a case where the scope is going to wind up working in weird ways no matter how they solve it — constant scoping is similarly weird if you ever look up the actual rules for it. In practice it’s usually right, so we only very occasionally have to worry about it. But that “in practice” also assumes “no conflicts.” That’s problematic for a feature like refinements that exist specifically to reduce or avoid conflicts.

          But it may work. I would have bet money that Ruby couldn’t get its existing four-or-so different scopes to pretty much work, and they do. Speaking as a guy who gets paid to investigate weird Ruby edge cases, it’s at least good for my job security.

          But I have the usual distrust of features where I can say, “well, it’s good for my job security.” C++ taught me where that path leads.

Leave a Reply to Avdi Grimm Cancel reply

Your email address will not be published. Required fields are marked *