Customizing the Ruby TestRunner

We all know we should write tests. That’s nothing new.

After those great tests get written, they have to be run. Lets figure out how this works so we can make running those tests as awesome as possible. We’re going to write a custom test runner to scratch an itch.

The Test

Here’s a TestUnit TestCase, it’s pretty fabulous:

require 'test/unit'

class ExampleTest < Test::Unit::TestCase
  def test_assert_equal
    assert_equal 1, 1
  end

  def test_lies
    assert false
  end

  def test_exceptions
    raise Exception, 'Beware the Jubjub bird, and shun the frumious Bandersnatch!'
  end

  def test_truth
    assert true
  end
end

The TestRunner

There should be nothing startling here, it’s just a quick fabricated test case. Copy it and save it as example_test.rb, and then run the test case with ruby. As you can see below it should complete pretty quick and give you an error, a failure, and two passing tests:

ruby example_test.rb  Loaded suite example_test
Started
.EF.
Finished in 0.016 seconds.   1) Error:
test_exceptions(ExampleTest):
Exception: Beware the Jubjub bird, and shun the frumious Bandersnatch!
    example_test.rb:21:in `test_exceptions'
2) Failure:
test_lies(ExampleTest) [example_test.rb:17]:
<false> is not true.
4 tests, 3 assertions, 1 failures, 1 errors

Now we know what to expect. So what can we do to scratch that itch I mentioned? Wait a sec, what was that itch again? Oh yeah, slow tests. Every now and then we will run our tests and see a failure at the top of our list, only to know we need to wait three more minutes before the tests will finish. So here is how you scratch an itch:

# Usage:
#   ruby -rfast_fail_runner [test] --runner=fastfail
require 'test/unit'
require 'test/unit/ui/console/testrunner'

class FastFailRunner < Test::Unit::UI::Console::TestRunner
  def add_fault(fault)
    @faults << fault
    nl
    output("%3d) %s" % [@faults.length, fault.long_display])
    output("--")
    @already_outputted = true
  end

  def finished(elapsed_time)
    nl
    output("Finished in #{elapsed_time} seconds.")
    nl
    output(@result)
  end
end

Test::Unit::AutoRunner::RUNNERS[:fastfail] = proc do |r|
  FastFailRunner
end

So what have we done? We have overridden the default test runner to output faults as soon as they are collected. “add_fault” and “finished” are two methods that respond to events which the tests emit. The last part registers this as an alternative test runner. I’ll explain the architecture of this all a bit more in another post. For now just save the example as fast_fail_runner.rb and find out what happens.

ruby -rfast_fail_runner example_test.rb --runner fastfail
Loaded suite example_test
Started
.   1) Error:
test_exceptions(ExampleTest):
Exception: Beware the Jubjub bird, and shun the frumious Bandersnatch!
    ../test/example_test.rb:13:in `test_exceptions'
--      2) Failure:
test_lies(ExampleTest) [../test/example_test.rb:9]:
<false> is not true.
--
.
Finished in 0.031 seconds.

4 tests, 3 assertions, 1 failures, 1 errors

It’s a little messy, but it scratches our itch.

So why fiddle with your test runner? Because you can make everything just a little bit more awesome. You could instrument your tests to gather performance data, wrap the tests in a nicer gui, or perhaps emit XML and JSON so you can consume it with some other process.

The Tease

Here’s a peek at something for next time:

TextMate Test Runner

So there it is, post number one.

end of line.

3 Comments

Filed under ruby, testing

3 responses to “Customizing the Ruby TestRunner

  1. I keep stumbling across this post (which is also useful in its own right) and wondering where the “next time” TextMateRunner is. I’d really like to tweak the way TextMate runs my tests, but not quite enough to forge out on my own. Any chance this article will show up soon? (Yes, I realize it’s been 2 years.)

    • Adam Sanderson

      Oh shucks, I didn’t really think anyone was particularly interested in it.
      I’ll post my textmate test runner on GitHub when I have a chance🙂

  2. dubek

    Take a look at the turn gem, it might work for what you’re looking for: https://github.com/TwP/turn

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s