The other day I put an app in production with (gasp) no automated tests. I’m careful to say no automated tests, because of course I had been testing it manually throughout the ~12 hours it took me to write the initial version of the app. In lieu of tests, I made sure to log copious amounts of information, which I then collected from my deployed app into Papertrail.

For a single day’s work this was acceptable. But I knew that once I had a version in production, pushing out even the smallest, seemingly benign change would be an extremely dicey proposition without some automated regression tests in place.

I had also deliberately written the app in a deliberately, uh, “un-factored” fashion. Building the app was an exploratory process, and I wanted my design to evolve out of use rather than impose design assumptions up-front.

An element in choosing to write the app this was way was the knowledge that I have the tools available to me to refactor the design incrementally. But in order to do that, I needed tests to assure me that I hadn’t broken anything. Those tests needed to be extremely high-level, so that they could continue running unchanged even as I heavily refactored the app’s internals.

So I set out to write an initial smoke test, one that would put the app through its paces in a “happy path” scenario. Because this was a Sinatra app, I needed to do a fair amount of setup to get this smoke test running. I’d done this before in other Sinatra apps, so a lot of it was a matter of cribbing off of those apps.

Since some of these steps were non-obvious, and because getting over the “first test” hump can be a pretty daunting prospect, I thought I’d document the process.

RSpec

First off, I needed a testing framework, and as a matter of habit I use RSpec. So I added a :testing group to my Gemfile, and added a dependency on the RSpec gem.

group :test, :development do
  gem "rspec"
end

Then I ran rspec –init to populate my project with starter spec/spec_helper.rb and .rspec files.

I left these mostly unchanged. My project has a top-level environment.rb file which sets up basic stuff like Bundler and Dotenv. Every test would need this environment set up consistently, so I added a line to the top of spec/spec_helper.rb:

require File.expand_path("../../environment", __FILE__)

I also made one change to .rspec. By default, RSpec configured this file to enable Ruby’s warnings mode. Sadly, my project includes some gems which are not warningfree. So I removed this line from the file:

--warnings

The test

It was time to write a test. I like to keep high-level tests in a spec/features directory, so I created it. Inside it, I created the file spec/features/smoke_spec.rb.

I wrote the file incrementally, but I’m going to paste in its final contents here, somewhat elided.

RSpec.describe "happy path", feature: true do
  let(:example_ipn_data) {
    {
      "payer_email"          => "johndoe@example.org",
      # ...
    }
  }

  specify "a customer buying a book" do
    test_github_team_id = ENV.fetch("GITHUB_TEAM_ID")
    test_github_login   = ENV.fetch("GITHUB_TEST_LOGIN")
    test_github_uid     = Integer(ENV.fetch("GITHUB_TEST_UID"))

    expect(db[:tokens]).to be_empty
    expect(db[:users]).to be_empty

    client = Octokit::Client.new(access_token: ENV.fetch("GITHUB_APP_TOKEN"))
    client.remove_team_member(test_github_team_id, test_github_login)
    expect(client.team_member?(test_github_team_id, test_github_login)).to be_falsey

    authorize "ipn", ENV.fetch("IPN_PASSWORD")
    post "/ipn", example_ipn_data
    expect(last_response.status).to eq(202)

    auth_hash = {
      provider: "github",
      uid: test_github_uid,
      info:     {
        name:  "John Doe",
      }
    }
    OmniAuth.config.mock_auth[:github] = OmniAuth::AuthHash.new(auth_hash)

    open_last_email_for("johndoe@example.org")
    click_first_link_in_email

    expect(client.team_member?(test_github_team_id, test_github_login)).to be_truthy

    email = open_last_email_for("johndoe@example.org"
      expect(email).to have_body_text(%r(https://github.com/ShipRise/rfm))
    end
  end

I’ll be going over this code one piece at a time over the course of this post. A few notes to start out with:

First, I’m using the new monkeypatch-free syntax from RSpec 3.0 to declare a test:

RSpec.describe "happy path", feature: true do

Second, in that last line you can see some RSpec metadata: feature: true . There’s nothing special about the name feature , it’s just a tag that I picked. I’ll be using this later to target some RSpec configuration to only apply to example groups with this tag.

Feature spec helper

You might have noticed that the test does not reference the spec_helper.rb file directly. Instead, it has this:

require "feature_spec_helper"

One of the practices advocated by the RSpec team is to limit the number of dependencies that have to be loaded before every test. One way they advise for doing this is to keep spec/spec_helper.rb file minimal, only requiring libraries in it which are required by every test. If some tests need more than the baseline level of support, they can require a specialized spec helper instead.

I opted to follow this advice, and have my feature specs require a specialized spec/feature_spec_helper.rb. Initially, this file just referenced the main spec helper:

require "spec_helper"

I made various additions to this file as I implemented the test.

Environment variables

The test starts out by fetching some volatile and/or sensitive data from the system environment. I keep configuration info like this in environment variables to make it easy to change; for ease of deploy to Heroku; and to make it easier to set up remote Continuous Integration without committing sensitive information to my Github repo.

test_github_team_id = ENV.fetch("GITHUB_TEAM_ID")
test_github_login   = ENV.fetch("GITHUB_TEST_LOGIN")
test_github_uid     = Integer(ENV.fetch("GITHUB_TEST_UID"))

I use ENV.fetch to grab all of the variable values my test needs. The test will fail if any of these variables are missing, so I want to know right away if there’s an unset variable. ENV.fetch will raise a KeyError if it doesn’t find the variable, letting me know exactly what var I forgot to add to my .env file or to my CI configuration.

The last variable is expected to be an integer. I don’t just want to ensure that the variable is set; I want to be assured that it is a valid integer as well. So rather than using the lenient #to_i > method, I use the Integer() conversion function. This function will raise an exception if the value it is given can’t be sensibly interpreted as an integer.

Database

Since this is my very first test, I am making no assumptions. Thus, the next stanza of my test does a sanity check that the database tables I’m interested in start out empty. This is important, because if I later check for the existence of some record, that check is meaningless unless I also know that the record didn’t exist at the beginning of the test.

expect(db[:tokens]).to be_empty
expect(db[:users]).to be_empty

These lines also force me to set up database integration for my tests.

I’m using Sequel as the database layer for this project. In order to make the database connection available to my tests, I add some code to the spec/feature_spec_helper.rb file.

The first line requires a file named db.rb. This file lives at lib/db.rb in my project, and is responsible for setting up a global DB constant which refers to a Sequel database connection.

require "db"

Next I define a module for database-related test helper methods. It defines a db attribute.

module DbSpecHelpers
  attr_accessor :db
end

And then I update the RSpec config to include this module into any example tagged with feature: true . I also add a before hook to initialize the db attribute with the value of the global DB constant.

RSpec.configure do |c|
  # ...
  c.include DbSpecHelpers, feature: true
  c.before feature: true do
    self.db = ::DB
  end
  # ...
end

I could avoid all this by referencing the DB constant directly in my tests. But I’m not sure I’m going to keep using a global constant, and I like to be able to override the default DB for individual tests.

Test setup

Next up in my test, I do some setup. The software that is being tested is supposed to add a user to a Github team when they purchase a product. In order to meaningfully test this behavior, I have to first be sure that the test user isn’t in the team to begin with. So I use Octokit to get the scenario started in the right state. Since I’m not certain if an exception will be raised on failure, I verify that the test user is not a team member after making the change.

client = Octokit::Client.new(access_token: ENV.fetch("GITHUB_APP_TOKEN"))
client.remove_team_member(test_github_team_id, test_github_login)
expect(client.team_member?(test_github_team_id, test_github_login)).to be_falsey

This setup illustrates a general rule in tests: it’s better to enforce a particular state of affairs before the test than to try to return things back to a “clean slate” state after the test. Tests can often be interrupted in the middle, especially when you’re first writing them. A robust test makes sure everything is where it is expected to be before getting started.

Rack::Test

Now that my setup is done, I need to kick off the test proper. The first step is to simulate a product purchase. To do this, I need to hit a webhook action with some simulated IPN data.

In order to simulate a POST to my Sinatra app, I need to set up Rack::Test. First, I add it to my Gemfile.

group :test, :development do
  # ...
  gem "rack-test"
  # ...
end

Then I bundle install.

Next I require rack/test in the spec/feature_spec_helper.rb. In order to use Rack::Test , I need my Sinatra app to be accessible, so I require that as well. It’s found in a top-level project file called app.rb.

# ...
require "rack/test"
# ...
require File.expand_path("../../app", __FILE__)

I create a helpers module for Rack::Test-enabled examples.

module RackSpecHelpers
  include Rack::Test::Methods
  attr_accessor :app
end

The app attribute is used by Rack::Test to find the Rack app to be tested.

I add some RSpec config to include and configure this new module in feature tests.

RSpec.configure do |c|
  c.include RackSpecHelpers, feature: true
  c.before feature: true do
    self.app = Sinatra::Application
  end
  # ...
end

Now I can simulate an authorized IPN POST using Rack::Test helper methods.

authorize "ipn", ENV.fetch("IPN_PASSWORD")
post "/ipn", example_ipn_data
expect(last_response.status).to eq(202)

OmniAuth

Shortly this test is going to be simulating a user clicking a link in an email. But before that can happen, I need to do a little more setup. The link is going to redirect them to a Github OAuth authentication page. My test can’t automate actions performed on sites outside of the Sinatra app being tested, so I need a way to fake out this authentication step.

Thankfully, I’m using OmniAuth, and OmniAuth has built-in support for faking authentication. First off, I go over to my spec/feature_spec_helper.rb and add this line:

# ...
OmniAuth.config.test_mode = true
# ...

This should be pretty self-explanatory.

Next, in my smoke test I fake up an authentication hash. Normally OmniAuth calls the app back with auth data sourced from the authentication provider. But in this test, it’s just going to call the app back with the auth data that I give it.

auth_hash = {
  provider: "github",
  uid: test_github_uid,
  info:     {
    name:  "John Doe",
  }
}

I tell OmniAuth to call back with this fake data the next time the app triggers authentication.

OmniAuth.config.mock_auth[:github] = OmniAuth::AuthHash.new(auth_hash)

EmailSpec and Capybara

I’m almost ready to have the app simulate a user opening their email and answering an invitation. But again, I can’t test systems outside my own, which means I need a way capture emails that the system sends. To do this, I’ll use EmailSpec. EmailSpec in turn depends on Capybara to simulate a human clicking on email links, so I’ll need to add that to the project as well.

I’ll start with Capybara. I add it as a project dependency, adding EmailSpec at the same time while I’m at it.

group :test, :development do
  # ...
  gem "email_spec"
  gem "capybara"
  # ...  
end

Then I update the spec/feature_spec_helper.rb file to set up Capybara/RSpec integration. I require the library, tell Capybara what app to use, and include some helper modules into feature example groups.

require "capybara/rspec"
# ...
Capybara.app = Sinatra::Application
# ...
RSpec.configure do |c|
  # ...
  c.include Capybara::DSL, feature: true
  c.include Capybara::RSpecMatchers, feature: true
  # ...
end

Now for EmailSpec. I require the library, and include some helper modules into feature example groups. Then I add a before hook to feature specs, resetting the delivery bin. If I neglected to do this, emails from one test might “bleed” over into others.

require "pony"
require "email_spec"

# ...

RSpec.configure do |c|
  # ...
  c.include EmailSpec::Helpers, feature: true
  c.include EmailSpec::Matchers, feature: true
  c.before :each, feature: true do
    reset_mailer
  end
  # ...
end

Notice that I require the pony library before email_spec. EmailSpec monkeypatches the two most common Ruby mail-sending libraries, ActionMailer and Pony, to send their email into a fake test delivery bin instead of to a mail server. It automatically detects which library to patch, based on what is available. I use Pony in my app, so I make sure that the pony library is loaded before email_spec.

I’m finally able to add some lines to the test, simulating a user opening their email and clicking on a link they find there.

open_last_email_for("johndoe@example.org")
click_first_link_in_email

Assert

In theory, the user clicking on the link in the welcome email should trigger a series of actions: they should authenticate with Github, which will be simulated as we saw earlier. Then the app will add them to a Github team. Finally, it will send them an email welcoming them to the team.

The rest of the test asserts that this is, in fact, what happens.

First, that the user has been added to a Github team:

expect(client.team_member?(test_github_team_id, test_github_login)).to be_truthy

And second, that they have a welcome email in their inbox:

email = open_last_email_for("johndoe@example.org")
expect(email).to have_body_text(%r(https://github.com/ShipRise/rfm))

DatabaseCleaner

All this works… exactly once. The second time I run the test, it fails at the beginning, where I checked that the database starts out empty.

expect(db[:tokens]).to be_empty
expect(db[:users]).to be_empty

In order to make sure the database starts out clean every time, I’m going to add DatabaseCleaner to the project. I’ve written about DatabaseCleaner before, so I’m not going to go into detail about this part. But for the record, here’s the configuration.

First, I need to add it to the Gemfile (and then bundle install).

group :test, :development do
  # ...
  gem "database_cleaner"
  # ...
end

And then I add DatabaseCleaner setup to spec/feature_spec_helper.rb.

# ...
require "database_cleaner"
# ...
RSpec.configure do |c|
  # ...

  c.before :suite do
    DatabaseCleaner[:sequel, {connection: ::DB}].strategy = :transaction
    DatabaseCleaner[:sequel, {connection: ::DB}].clean_with(:truncation)
  end
  # ...
  c.before feature: true do
    DatabaseCleaner[:sequel, {connection: ::DB}].start
  end

  c.after feature: true do
    DatabaseCleaner[:sequel, {connection: ::DB}].clean
  end
  # ...
end

Notice that I am specific about which database connection to use:

DatabaseCleaner[:sequel, {connection: ::DB}]

Files

Here is full addition made to Gemfile :

group :test, :development do
  gem "rspec"
  gem "rack-test"
  gem "email_spec"
  gem "database_cleaner"
  gem "capybara"
end

And here is the completed spec/feature_spec_helper.rb :

require "spec_helper"
require "db"
require "rack/test"
require "pony"
require "email_spec"
require "database_cleaner"
require "capybara/rspec"

require File.expand_path("../../app", __FILE__)

Capybara.app = Sinatra::Application

OmniAuth.config.test_mode = true

module DbSpecHelpers
  attr_accessor :db
end

module RackSpecHelpers
  include Rack::Test::Methods
  attr_accessor :app
end

RSpec.configure do |c|
  c.include RackSpecHelpers, feature: true
  c.before feature: true do
    self.app = Sinatra::Application
  end

  c.before :suite do
    DatabaseCleaner[:sequel, {connection: ::DB}].strategy = :transaction
    DatabaseCleaner[:sequel, {connection: ::DB}].clean_with(:truncation)
  end

  c.include DbSpecHelpers, feature: true
  c.before feature: true do
    extend DbSpecHelpers
    self.db = ::DB
    DatabaseCleaner[:sequel, {connection: ::DB}].start
  end

  c.after feature: true do
    DatabaseCleaner[:sequel, {connection: ::DB}].clean
  end

  c.include EmailSpec::Helpers, feature: true
  c.include EmailSpec::Matchers, feature: true

  c.before :each, feature: true do
    reset_mailer
  end

  c.include Capybara::DSL, feature: true
  c.include Capybara::RSpecMatchers, feature: true
end

As the app grows I will probably begin to split this setup out into multiple files.

Conclusion

And there you have it, a first smoke test for a Sinatra app. This was a lot of setup, but it also accomplishes a lot. It simulates an external API call and a user at a browser; it fakes out both OAuth authentication and sending emails; and it interacts with a real external API.

Since I wrote this test I’ve added VCR to speed things up and keep me from hitting the API too often from, tests. I’ve also set up CircleCI to only push my app to Heroku if this smoke test passes. I can now make changes—little ones, or sweeping design renovations—with confidence, knowing that I haven’t broken the essential functionality of my application.

EDIT: As someone pointed out in the comments, this test is awful by a number of measures of test quality. The name talks about “buying a book”, but there’s no point in the test where a book appears to be purchased. Assertions are mixed in with actions. The test is long. If I had to read this test without any familiarity with the codebase, it would take a while before I fully understood what it was testing.

And yet it is still 1000x better than the tests which preceeded it, which is to say no tests at all. Before, I couldn’t change anything in this app, for fear of screwing up a working steady state. Now I can make changes a fair degree of confidence.

I have a bad habit of agonizing over my tests. I want them to be beautiful and well organized. And at least in some contexts there are some good reasons for wanting these qualities, for reasons I’ll go into in another post. But sometimes the most important thing is to simply get some kind of repeatable, automated test in place. Toward that goal, this test succeeds with flying colors.

Published by Avdi Grimm

8 Comments

  1. “The software that is being tested is supposed to add a user to a Github team when they purchase a product. ”

    The description says: specify “a customer buying a book” do

    but there’s no reference to a “book” in the test. There’s a mention of an IPN, and parsing an email to make sure a reference to a github repo is in there… and there are multiple expects after each step. Just looking at the tests alone, it’s hard to see what’s important here and what’s actually being tested. Does the user have to just buy the book, or buy the book and click a link in the email? Or do they have to buy the book, THEN login, THEN click the email? And how many emails are being sent?

    I don’t meant to be pedantic, but I just want to throw out a warning about “smoke tests…” It’s very, very easy for them to spiral out-of-control. I don’t think they’re meant to test everything, they’re just meant to serve as a simple integration test — one that goes through the entire process and verifies a couple things. Or perhaps even verifies nothing! If something raises an exception or makes continuing down the “happy path” impossible, the smoke test will fail.

    Testing minor details, like the content of an email, may be better kept in separate unit tests.

    Reply
  2. Good stuff, @avdi:disqus .

    I’m curious about why you bothered to bring in the extra dependency on DatabaseCleaner when Sequel already provides a transaction/rollback API that, IMO, is far simpler to integrate:

    RSpec.configure do |config|
    config.around(:example, :db) do |ex|
    ::DB.transaction(rollback: :always, &ex)
    end
    end

    On a side note, the .rspec file generated by rspec --init in RSpec 3 includes a --require spec_helper line that should ensure that file’s always loaded without the need to explicitly require it from your feature_spec_helper. You may still want to require it manually to make the dependency explicit, of course.

    Reply
    • Running all the test and app code inside a db transaction isn’t quite the same as running it outside a transaction.

      Probably fine though if you aren’t using fancy stuff like multiple connections, threads, deferred triggers, care about the value of now(), or any of the other differences.

      But I like my tests to mirror as close as possible how the code will actually run, so I don’t put wrap test code inside a transaction.

      Reply
    • Just to add to your comment, Myron, this won’t work if your examples also use transactions (either directly or by calling some code that does use it). This is what I have for my spec helpers (I use PostgreSQL):

      transaction_options = {auto_savepoint: true, savepoint: true, rollback: :always}
      config.around(:each) {|e| DB.transaction(transaction_options){ e.run } }
      config.around(:all_nested) {|g| DB.transaction(transaction_options){ g.run_examples } }

      The all_nested is specific to my application as I want to be able to create records in before(:all) blocks and be able to rollback after the context is finished. In case you’re curious about it:

      https://github.com/rosenfeld/rspec_around_all/blob/config_around/lib/rspec_around_all.rb

      Reply
  3. A few minor suggestions:

    1. Use require_relative. So instead of:

      require File.expand_path(“../../environment”, FILE)

    use:

    require_relative “../environment”

    1. Add APP_ROOT or something similar in your environment.rb to make it easy to find the project root directory. Assuming you use a Pathname, this will allow you to change:

      require File.expand_path(“../../app”, FILE)

    to:

    require APP_ROOT + “app”

    Reply
  4. Avdi, I just wanted to say thanks as always for the countless ways you help other developers. I came across this while starting to dig more in-depth into some refactoring of specs and, as always, your thoughts have greatly improved my code. Thanks again!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *