Real unit tests, and bugs that go THWACK
Jason Gorman posted about “The Future Of TDD - Real-Time Feedback As You Type?” on his blog. He says that unit tests should give us feedback as fast as the feedback he got when dropping his security badge -
“On my way back from lunch today, I was walking down the stairs when I heard a loud “thwack” and turned round to see that my security pass had come loose from its clip and fallen to the floor.”
Wouldn’t it be great if bugs made a thwacking sound the moment they landed in our code? He goes on to imagine an IDE someday running unit tests as we’re typing code so we get real-time feedback. He suggests that if it takes five minutes to run unit tests, that’s far too long, so we’ll have to wait until computers get much faster before his idea is feasible.
But I would suggest that if we have unit tests that take five minutes to run, we may not be doing unit testing properly. I recently wrote a unit test for a Perl module using Perl’s xUnit - Test::Unit. There are 18 tests, achieving close to 100% coverage, and they run in about 200 milliseconds of cpu time and well under a second of elapsed time. I like Michael Feathers’ criteria for unit tests, which say that they don’t talk to a database, don’t communicate across a network, don’t touch the filesystem, don’t require configuration files, and if a test case takes longer than 1/10 of a second, it’s glacially slow (see Working Effectively with Legacy Code). I did violate one of these, because my tests do create and delete files several times; I haven’t found a good way to mock file access in Perl. But I’m happy with the subsecond run time.
It’s conceivable that a large system could have five minutes worth of well-written unit tests. Jason hinted that he’s thinking of running all of the system’s unit tests when he says that we could try to take a shortcut and only run the unit tests that are dependent on the code that changed. But that doesn’t sound like unit testing to me, it sounds more like integration testing. I do have suites of subsystem tests and system tests that take more than a minute to run, but I don’t call them unit tests. So we’re probably using different definitions of what a unit test is. I know that many people use the term “unit test” very loosely, which tends to mask the fact that people often aren’t really getting the benefits of good unit testing.
So I think that Jason’s desire for very quick feedback is closer to our grasp than he thinks. We need to test our code in extreme isolation from the rest of the system, so that as many bugs as possible will make a loud “thwack” in the unit test environment, before we pull out the slow integration tests.
Update: seeing the inelegance of “close to 100%”, I added a few tests and now have 100% branch and condition coverage for my class. But I still have much to learn about migrating my black box testing skills to unit testing.
April 2nd, 2008 at 1:02 am
Wow - that’s fast.
So where do the slow integration tests fit in? Should they be run before a check in? Daily by an automated process?
April 2nd, 2008 at 8:24 am
Yes, fast is good, and I think more developers should be going for this kind of speed. If I could find a trick for staying out of the filesystem, my tests would be much faster still. Granted, many developers face some real challenges in getting legacy code running in a unit test environment.
The integration tests can be run between coding sessions and before checkin in the development environment, and after checkin in an integration test environment. The project I’m referring to doesn’t have an automated continuous integration process because it’s a fairly small project.
April 2nd, 2008 at 10:35 pm
I hear you - legacy code is painful.
So what do you mean by a development environment and an integration test environment? Does (should) an integration test environment use xUnit?
Further, how long should integration tests take to run? (Is 1 hour or 2 hours too long?)
April 3rd, 2008 at 8:48 am
I said “development environment” meaning a computer that would be convenient for the developer to test their latest code changes. An “integration environment” in this context usually refers to a separate machine from the development environment, where the code is checked out from version control and where any unknown configuration dependencies that were masked on the development environment are more likely to be found. But integration testing could also be run in a development environment.
The choice of tools in an integration environment varies widely. You could use either a black box test tool or a unit test tool. If you’re testing a partially integrated part of the system, you’d probably either need to use a unit test tool or build drivers that can execute the software when it doesn’t have a user interface.
Run time for integration tests also varies. Many projects have no intermediate level tests between unit tests and system tests (and some only have one type of test). If your integration tests take more than about 10 minutes to run, you might want to choose a subset of them that can be run in the development environment so that developers can easily test a day’s or few hours’ worth of work. You can the full set of tests in an integration environment, independent of the development effort. The first step to worry about, before improving run time, is making the integration environment easy enough to use so that all the developers can feasibly use it. E.g., if they rely on a single shared database server, coordinating test runs will take some effort.