Customizing the Ruby TestRunner

We all know we should write tests. That’s nothing new.

After those great tests get written, they have to be run. Lets figure out how this works so we can make running those tests as awesome as possible. We’re going to write a custom test runner to scratch an itch.

The Test

Here’s a TestUnit TestCase, it’s pretty fabulous:

require 'test/unit'

class ExampleTest < Test::Unit::TestCase
  def test_assert_equal
    assert_equal 1, 1
  end

  def test_lies
    assert false
  end

  def test_exceptions
    raise Exception, 'Beware the Jubjub bird, and shun the frumious Bandersnatch!'
  end

  def test_truth
    assert true
  end
end

The TestRunner

There should be nothing startling here, it’s just a quick fabricated test case. Copy it and save it as example_test.rb, and then run the test case with ruby. As you can see below it should complete pretty quick and give you an error, a failure, and two passing tests:

ruby example_test.rb  Loaded suite example_test
Started
.EF.
Finished in 0.016 seconds.   1) Error:
test_exceptions(ExampleTest):
Exception: Beware the Jubjub bird, and shun the frumious Bandersnatch!
    example_test.rb:21:in `test_exceptions'
2) Failure:
test_lies(ExampleTest) [example_test.rb:17]:
<false> is not true.
4 tests, 3 assertions, 1 failures, 1 errors

Now we know what to expect. So what can we do to scratch that itch I mentioned? Wait a sec, what was that itch again? Oh yeah, slow tests. Every now and then we will run our tests and see a failure at the top of our list, only to know we need to wait three more minutes before the tests will finish. So here is how you scratch an itch:

# Usage:
#   ruby -rfast_fail_runner [test] --runner=fastfail
require 'test/unit'
require 'test/unit/ui/console/testrunner'

class FastFailRunner < Test::Unit::UI::Console::TestRunner
  def add_fault(fault)
    @faults << fault
    nl
    output("%3d) %s" % [@faults.length, fault.long_display])
    output("--")
    @already_outputted = true
  end

  def finished(elapsed_time)
    nl
    output("Finished in #{elapsed_time} seconds.")
    nl
    output(@result)
  end
end

Test::Unit::AutoRunner::RUNNERS[:fastfail] = proc do |r|
  FastFailRunner
end

So what have we done? We have overridden the default test runner to output faults as soon as they are collected. “add_fault” and “finished” are two methods that respond to events which the tests emit. The last part registers this as an alternative test runner. I’ll explain the architecture of this all a bit more in another post. For now just save the example as fast_fail_runner.rb and find out what happens.

ruby -rfast_fail_runner example_test.rb --runner fastfail
Loaded suite example_test
Started
.   1) Error:
test_exceptions(ExampleTest):
Exception: Beware the Jubjub bird, and shun the frumious Bandersnatch!
    ../test/example_test.rb:13:in `test_exceptions'
--      2) Failure:
test_lies(ExampleTest) [../test/example_test.rb:9]:
<false> is not true.
--
.
Finished in 0.031 seconds.

4 tests, 3 assertions, 1 failures, 1 errors

It’s a little messy, but it scratches our itch.

So why fiddle with your test runner? Because you can make everything just a little bit more awesome. You could instrument your tests to gather performance data, wrap the tests in a nicer gui, or perhaps emit XML and JSON so you can consume it with some other process.

The Tease

Here’s a peek at something for next time:

So there it is, post number one.

end of line.