Zero to Smoke Test with Sinatra

First off, I needed a testing framework, and as a matter of habit I use RSpec. So I added a :testing group to my Gemfile, and added a dependency on the RSpec gem.

group :test, :development do
  gem "rspec"
end

Then I ran rspec –init to populate my project with starter spec/spec_helper.rb and .rspec files.

I left these mostly unchanged. My project has a top-level environment.rb file which sets up basic stuff like Bundler and Dotenv. Every test would need this environment set up consistently, so I added a line to the top of spec/spec_helper.rb:

require File.expand_path("../../environment", __FILE__)

I also made one change to .rspec. By default, RSpec configured this file to enable Ruby’s warnings mode. Sadly, my project includes some gems which are not warningfree. So I removed this line from the file:

--warnings

It was time to write a test. I like to keep high-level tests in a spec/features directory, so I created it. Inside it, I created the file spec/features/smoke_spec.rb.

I wrote the file incrementally, but I’m going to paste in its final contents here, somewhat elided.

RSpec.describe "happy path", feature: true do
  let(:example_ipn_data) {
    {
      "payer_email"          => "johndoe@example.org",
      # ...
    }
  }

  specify "a customer buying a book" do
    test_github_team_id = ENV.fetch("GITHUB_TEAM_ID")
    test_github_login   = ENV.fetch("GITHUB_TEST_LOGIN")
    test_github_uid     = Integer(ENV.fetch("GITHUB_TEST_UID"))

    expect(db[:tokens]).to be_empty
    expect(db[:users]).to be_empty

    client = Octokit::Client.new(access_token: ENV.fetch("GITHUB_APP_TOKEN"))
    client.remove_team_member(test_github_team_id, test_github_login)
    expect(client.team_member?(test_github_team_id, test_github_login)).to be_falsey

    authorize "ipn", ENV.fetch("IPN_PASSWORD")
    post "/ipn", example_ipn_data
    expect(last_response.status).to eq(202)

    auth_hash = {
      provider: "github",
      uid: test_github_uid,
      info:     {
        name:  "John Doe",
      }
    }
    OmniAuth.config.mock_auth[:github] = OmniAuth::AuthHash.new(auth_hash)

    open_last_email_for("johndoe@example.org")
    click_first_link_in_email

    expect(client.team_member?(test_github_team_id, test_github_login)).to be_truthy

    email = open_last_email_for("johndoe@example.org"
      expect(email).to have_body_text(%r(https://github.com/ShipRise/rfm))
    end
  end

I’ll be going over this code one piece at a time over the course of this post. A few notes to start out with:

First, I’m using the new monkeypatch-free syntax from RSpec 3.0 to declare a test:

RSpec.describe "happy path", feature: true do

Second, in that last line you can see some RSpec metadata: feature: true . There’s nothing special about the name feature , it’s just a tag that I picked. I’ll be using this later to target some RSpec configuration to only apply to example groups with this tag.

You might have noticed that the test does not reference the spec_helper.rb file directly. Instead, it has this:

require "feature_spec_helper"

One of the practices advocated by the RSpec team is to limit the number of dependencies that have to be loaded before every test. One way they advise for doing this is to keep spec/spec_helper.rb file minimal, only requiring libraries in it which are required by every test. If some tests need more than the baseline level of support, they can require a specialized spec helper instead.

I opted to follow this advice, and have my feature specs require a specialized spec/feature_spec_helper.rb. Initially, this file just referenced the main spec helper:

require "spec_helper"

I made various additions to this file as I implemented the test.

The test starts out by fetching some volatile and/or sensitive data from the system environment. I keep configuration info like this in environment variables to make it easy to change; for ease of deploy to Heroku; and to make it easier to set up remote Continuous Integration without committing sensitive information to my Github repo.

test_github_team_id = ENV.fetch("GITHUB_TEAM_ID")
test_github_login   = ENV.fetch("GITHUB_TEST_LOGIN")
test_github_uid     = Integer(ENV.fetch("GITHUB_TEST_UID"))

I use ENV.fetch to grab all of the variable values my test needs. The test will fail if any of these variables are missing, so I want to know right away if there’s an unset variable. ENV.fetch will raise a KeyError if it doesn’t find the variable, letting me know exactly what var I forgot to add to my .env file or to my CI configuration.

The last variable is expected to be an integer. I don’t just want to ensure that the variable is set; I want to be assured that it is a valid integer as well. So rather than using the lenient #to_i > method, I use the Integer() conversion function. This function will raise an exception if the value it is given can’t be sensibly interpreted as an integer.

Since this is my very first test, I am making no assumptions. Thus, the next stanza of my test does a sanity check that the database tables I’m interested in start out empty. This is important, because if I later check for the existence of some record, that check is meaningless unless I also know that the record didn’t exist at the beginning of the test.

expect(db[:tokens]).to be_empty
expect(db[:users]).to be_empty

These lines also force me to set up database integration for my tests.

I’m using Sequel as the database layer for this project. In order to make the database connection available to my tests, I add some code to the spec/feature_spec_helper.rb file.

The first line requires a file named db.rb. This file lives at lib/db.rb in my project, and is responsible for setting up a global DB constant which refers to a Sequel database connection.

require "db"

Next I define a module for database-related test helper methods. It defines a db attribute.

module DbSpecHelpers
  attr_accessor :db
end

And then I update the RSpec config to include this module into any example tagged with feature: true . I also add a before hook to initialize the db attribute with the value of the global DB constant.

RSpec.configure do |c|
  # ...
  c.include DbSpecHelpers, feature: true
  c.before feature: true do
    self.db = ::DB
  end
  # ...
end

I could avoid all this by referencing the DB constant directly in my tests. But I’m not sure I’m going to keep using a global constant, and I like to be able to override the default DB for individual tests.

Next up in my test, I do some setup. The software that is being tested is supposed to add a user to a Github team when they purchase a product. In order to meaningfully test this behavior, I have to first be sure that the test user isn’t in the team to begin with. So I use Octokit to get the scenario started in the right state. Since I’m not certain if an exception will be raised on failure, I verify that the test user is not a team member after making the change.

client = Octokit::Client.new(access_token: ENV.fetch("GITHUB_APP_TOKEN"))
client.remove_team_member(test_github_team_id, test_github_login)
expect(client.team_member?(test_github_team_id, test_github_login)).to be_falsey

This setup illustrates a general rule in tests: it’s better to enforce a particular state of affairs before the test than to try to return things back to a “clean slate” state after the test. Tests can often be interrupted in the middle, especially when you’re first writing them. A robust test makes sure everything is where it is expected to be before getting started.

Now that my setup is done, I need to kick off the test proper. The first step is to simulate a product purchase. To do this, I need to hit a webhook action with some simulated IPN data.

In order to simulate a POST to my Sinatra app, I need to set up Rack::Test. First, I add it to my Gemfile.

group :test, :development do
  # ...
  gem "rack-test"
  # ...
end

Then I bundle install.

Next I require rack/test in the spec/feature_spec_helper.rb. In order to use Rack::Test , I need my Sinatra app to be accessible, so I require that as well. It’s found in a top-level project file called app.rb.

# ...
require "rack/test"
# ...
require File.expand_path("../../app", __FILE__)

I create a helpers module for Rack::Test-enabled examples.

module RackSpecHelpers
  include Rack::Test::Methods
  attr_accessor :app
end

The app attribute is used by Rack::Test to find the Rack app to be tested.

I add some RSpec config to include and configure this new module in feature tests.

RSpec.configure do |c|
  c.include RackSpecHelpers, feature: true
  c.before feature: true do
    self.app = Sinatra::Application
  end
  # ...
end

Now I can simulate an authorized IPN POST using Rack::Test helper methods.

authorize "ipn", ENV.fetch("IPN_PASSWORD")
post "/ipn", example_ipn_data
expect(last_response.status).to eq(202)

Shortly this test is going to be simulating a user clicking a link in an email. But before that can happen, I need to do a little more setup. The link is going to redirect them to a Github OAuth authentication page. My test can’t automate actions performed on sites outside of the Sinatra app being tested, so I need a way to fake out this authentication step.

Thankfully, I’m using OmniAuth, and OmniAuth has built-in support for faking authentication. First off, I go over to my spec/feature_spec_helper.rb and add this line:

# ...
OmniAuth.config.test_mode = true
# ...

This should be pretty self-explanatory.

Next, in my smoke test I fake up an authentication hash. Normally OmniAuth calls the app back with auth data sourced from the authentication provider. But in this test, it’s just going to call the app back with the auth data that I give it.

auth_hash = {
  provider: "github",
  uid: test_github_uid,
  info:     {
    name:  "John Doe",
  }
}

I tell OmniAuth to call back with this fake data the next time the app triggers authentication.

OmniAuth.config.mock_auth[:github] = OmniAuth::AuthHash.new(auth_hash)

I’m almost ready to have the app simulate a user opening their email and answering an invitation. But again, I can’t test systems outside my own, which means I need a way capture emails that the system sends. To do this, I’ll use EmailSpec. EmailSpec in turn depends on Capybara to simulate a human clicking on email links, so I’ll need to add that to the project as well.

I’ll start with Capybara. I add it as a project dependency, adding EmailSpec at the same time while I’m at it.

group :test, :development do
  # ...
  gem "email_spec"
  gem "capybara"
  # ...  
end

Then I update the spec/feature_spec_helper.rb file to set up Capybara/RSpec integration. I require the library, tell Capybara what app to use, and include some helper modules into feature example groups.

require "capybara/rspec"
# ...
Capybara.app = Sinatra::Application
# ...
RSpec.configure do |c|
  # ...
  c.include Capybara::DSL, feature: true
  c.include Capybara::RSpecMatchers, feature: true
  # ...
end

Now for EmailSpec. I require the library, and include some helper modules into feature example groups. Then I add a before hook to feature specs, resetting the delivery bin. If I neglected to do this, emails from one test might “bleed” over into others.

require "pony"
require "email_spec"

# ...

RSpec.configure do |c|
  # ...
  c.include EmailSpec::Helpers, feature: true
  c.include EmailSpec::Matchers, feature: true
  c.before :each, feature: true do
    reset_mailer
  end
  # ...
end

Notice that I require the pony library before email_spec. EmailSpec monkeypatches the two most common Ruby mail-sending libraries, ActionMailer and Pony, to send their email into a fake test delivery bin instead of to a mail server. It automatically detects which library to patch, based on what is available. I use Pony in my app, so I make sure that the pony library is loaded before email_spec.

I’m finally able to add some lines to the test, simulating a user opening their email and clicking on a link they find there.

open_last_email_for("johndoe@example.org")
click_first_link_in_email

In theory, the user clicking on the link in the welcome email should trigger a series of actions: they should authenticate with Github, which will be simulated as we saw earlier. Then the app will add them to a Github team. Finally, it will send them an email welcoming them to the team.

The rest of the test asserts that this is, in fact, what happens.

First, that the user has been added to a Github team:

expect(client.team_member?(test_github_team_id, test_github_login)).to be_truthy

And second, that they have a welcome email in their inbox:

email = open_last_email_for("johndoe@example.org")
expect(email).to have_body_text(%r(https://github.com/ShipRise/rfm))

All this works… exactly once. The second time I run the test, it fails at the beginning, where I checked that the database starts out empty.

expect(db[:tokens]).to be_empty
expect(db[:users]).to be_empty

In order to make sure the database starts out clean every time, I’m going to add DatabaseCleaner to the project. I’ve written about DatabaseCleaner before, so I’m not going to go into detail about this part. But for the record, here’s the configuration.

First, I need to add it to the Gemfile (and then bundle install).

group :test, :development do
  # ...
  gem "database_cleaner"
  # ...
end

And then I add DatabaseCleaner setup to spec/feature_spec_helper.rb.

# ...
require "database_cleaner"
# ...
RSpec.configure do |c|
  # ...

  c.before :suite do
    DatabaseCleaner[:sequel, {connection: ::DB}].strategy = :transaction
    DatabaseCleaner[:sequel, {connection: ::DB}].clean_with(:truncation)
  end
  # ...
  c.before feature: true do
    DatabaseCleaner[:sequel, {connection: ::DB}].start
  end

  c.after feature: true do
    DatabaseCleaner[:sequel, {connection: ::DB}].clean
  end
  # ...
end

Notice that I am specific about which database connection to use:

DatabaseCleaner[:sequel, {connection: ::DB}]

Here is full addition made to Gemfile :

group :test, :development do
  gem "rspec"
  gem "rack-test"
  gem "email_spec"
  gem "database_cleaner"
  gem "capybara"
end

And here is the completed spec/feature_spec_helper.rb :

require "spec_helper"
require "db"
require "rack/test"
require "pony"
require "email_spec"
require "database_cleaner"
require "capybara/rspec"

require File.expand_path("../../app", __FILE__)

Capybara.app = Sinatra::Application

OmniAuth.config.test_mode = true

module DbSpecHelpers
  attr_accessor :db
end

module RackSpecHelpers
  include Rack::Test::Methods
  attr_accessor :app
end

RSpec.configure do |c|
  c.include RackSpecHelpers, feature: true
  c.before feature: true do
    self.app = Sinatra::Application
  end

  c.before :suite do
    DatabaseCleaner[:sequel, {connection: ::DB}].strategy = :transaction
    DatabaseCleaner[:sequel, {connection: ::DB}].clean_with(:truncation)
  end

  c.include DbSpecHelpers, feature: true
  c.before feature: true do
    extend DbSpecHelpers
    self.db = ::DB
    DatabaseCleaner[:sequel, {connection: ::DB}].start
  end

  c.after feature: true do
    DatabaseCleaner[:sequel, {connection: ::DB}].clean
  end

  c.include EmailSpec::Helpers, feature: true
  c.include EmailSpec::Matchers, feature: true

  c.before :each, feature: true do
    reset_mailer
  end

  c.include Capybara::DSL, feature: true
  c.include Capybara::RSpecMatchers, feature: true
end

As the app grows I will probably begin to split this setup out into multiple files.

And there you have it, a first smoke test for a Sinatra app. This was a lot of setup, but it also accomplishes a lot. It simulates an external API call and a user at a browser; it fakes out both OAuth authentication and sending emails; and it interacts with a real external API.

Since I wrote this test I’ve added VCR to speed things up and keep me from hitting the API too often from, tests. I’ve also set up CircleCI to only push my app to Heroku if this smoke test passes. I can now make changes—little ones, or sweeping design renovations—with confidence, knowing that I haven’t broken the essential functionality of my application.

EDIT: As someone pointed out in the comments, this test is awful by a number of measures of test quality. The name talks about “buying a book”, but there’s no point in the test where a book appears to be purchased. Assertions are mixed in with actions. The test is long. If I had to read this test without any familiarity with the codebase, it would take a while before I fully understood what it was testing.

And yet it is still 1000x better than the tests which preceeded it, which is to say no tests at all. Before, I couldn’t change anything in this app, for fear of screwing up a working steady state. Now I can make changes a fair degree of confidence.

I have a bad habit of agonizing over my tests. I want them to be beautiful and well organized. And at least in some contexts there are some good reasons for wanting these qualities, for reasons I’ll go into in another post. But sometimes the most important thing is to simply get some kind of repeatable, automated test in place. Toward that goal, this test succeeds with flying colors.