Open Testware Reviews
JUnit
Copyright 2003 by Tejas Software Consulting
- All rights reserved.
Contents
Overview -- Maturity
-- Project activity -- Platforms -- Support -- Documentation -- Installation
-- Implementation -- Performance -- Similar
tools -- Limitations -- Observations -- Appendix:
Sample JUnit test
Overview
Reviewed: 2003-June-30
Version reviewed: 3.8.1,
2002-Sept-03
Maintainer: junit.org
URL: http://www.junit.org/
Testingfaqs.org category:
Unit Test Tools
License: Common Public License
User interface: GUI, command
line
JUnit is probably the most widely used unit test tool on the planet. If you
have Java code to test, and developers willing to take responsibility for
the quality of their own code, then you should consider jumping on the bandwagon.
The only things that make me nervous about JUnit is the lack of a complete
tutorial, and a strange lapse in the release cycle over the last several
months. Read on for details.
Here's a screen shot of the GUI interface for JUnit, after running the the
test suite from the Appendix. The red bar is showing. The goal for the developer
is to achieve the celebrated green bar shortly after it goes red.
Maturity
5 - Production (on a scale
of 1-5)
The JUnit team has classified the tool on SourceForge as production quality,
and from what I've seen so far, I agree. Its broad usage base increases my
confidence that the code has been exercised in a wide variety of scenarios.
Project activity
4 - Active (on a scale of 1-5)
There were six releases of JUnit between January 2000 and September 2002,
but this pace stops there. There hasn't been another release since September
2002, and a quick scan of the CVS archives show no code changes checked in
during the interim. But because the current release is less than a year old,
and the junit mailing list continues to be very active, JUnit earns an "Active"
rating.
Platforms
Java tools rarely document their supported platforms. SourceForge
says the JUnit project is "OS Independent," which seems to be true to a large
extent, because Java runs on a wide variety of platforms. It's fairly safe
to assume that JUnit will run anywhere that Java runs.
Support
The mailing list for JUnit users is the JUnit list on Yahoo Groups.
An alternative outlet is the JUnit Forum on
jguru.com. There are also the JUnit announcements
and JUnit
developer lists hosted on SourceForge, but both are plagued by spam and
little else, so they're of no value. The tool's maintainers did not reply
to email I sent them directly, perhaps because the wide popularity of the
tool generates too many messages for them to handle personally.
SourceForge hosts databases for Bugs,
Support
Requests, Patches,
and Feature
Requests. Personally, I find it confusing when all of these databases
are used on a project, but it's good to know that there's a formal mechanism
to track bug reports.
The JUnit project uses the CVS server on SourceForge for revision control.
The README file gives
a nice set of release notes for recent minor releases, but there's practically
no documentation on the 3.8.1 bug fix release.
I couldn't find any information about commercial support for JUnit, though
there are probably hordes of Java contractors who are familiar with JUnit
and could provide help. There is a JUnit training course listed at junit.org.
Documentation
There are several places you can turn to for information about JUnit. The
main web page is junit.org. Several links
there lead to the SourceForge page, junit.sourceforge.net.
Those who are familiar with SourceForge would have an easier time navigating
the services hosted by SourceForge by going directly to the summary page,
sourceforge.net/projects/junit.
There are several documents provided for JUnit:
"Test Infected" is the seminal JUnit article that's often considered required
reading for Java programmers. I found as a whole, though, that the available
JUnit documentation is not very cohesive, especially for a beginning Java
programmer like me. I dug through document after document that mentioned
and illustrated JUnit's GUI, but only in the "Frequently asked questions"
file did I find the incantation required to actually start the GUI.
There is a full set of JavaDocs, which has comprehensive low-level coverage
of the JUnit classes with documentation extracted from the source code. Also
available is a long list of articles about various aspects of JUnit and its
extensions.
Another source of documentation is the book Test-Driven Development: By
Example by Kent Beck, though it's more of a discussion about the development
process than a tutorial introduction to JUnit.
Installation
JUnit installs from a 432 kilobyte zip file that expands to just over a
megabyte. The only other installation required is adding the path to the
junit.jar file to the Java classpath, plus the path to where your test classes
will be located. I was very frustrated with trying to get the classpath right
and figuring out the package names relative to the classpath, but that's
because I'm a Java newbie.
Implementation
JUnit source code is available in a src.jar file in the binary distribution,
which can be extracted using your favorite zip utility, assuming you can
convince it that it's okay to read a .jar file. JUnit consists of 47 Java
files for a total of 3673 non-comment lines of code, plus 986 comment lines.
JUnit is able to test itself, always a notable feature of a test tool. JUnit
comes with 91 test cases implemented in 35 Java files, adding up to 1184
non-comment source lines and 131 comment lines. And yes, all tests pass on
both platforms I tried JUnit on, finishing in well under 10 seconds.
Performance
There's not much to report concerning performance. JUnit is basically bound
to the performance of the Java run-time environment.
If you compare unit testing to the typical black-box test framework, you're
likely to notice a dramatic difference in performance. Simple and well-written
black-box tests are likely to run for several seconds each. But in the unit
test performance mentioned in the previous section, I was able to run more
than 10 unit tests per second. This is because unit tests focus on the code
with pinpoint precision. There's no need to initialize the entire application
in order to run one test.
Similar tools
JUnit is one of the first of a large class of unit test frameworks collectively
called "xUnit." Dozens of unit test tools for other languages are patterned
after JUnit. By the way, legend has it that JUnit was based on SUnit, a unit
test framework for Smalltalk.
In the April 2003 Unit
Test Tool Survey, there is one other Java unit test tool listed, GrandTestAuto,
that doesn't have nearly the following that JUnit does. There are a handful
of tools that deal with mock objects in Java, plus a few others related to
web testing that deal with Java in one way or another. There are also many
JUnit extensions available.
Though most unit test frameworks are freeware, there are a few commercial
Java test tools listed on the testingfaqs.org Unit Test Tools list, AQtest
and Test Mentor - Java Edition, and several more will be added soon.
Limitations
At the time I started the review, there were 47 open bugs in the bug database,
and 48 open feature requests. There hasn't been any appreciable progress
on the bug backlog since the past JUnit release.
Earlier I mentioned that the documentation can be frustrating for beginners.
A few people have expressed concerns about the fact that JUnit stops after
the first failure. Actually, JUnit will continue running test cases if a
test case fails, as you can tell by the screen shot above. But it won't continue
a test case after the first assertion that fails within that test case. The
FAQ discusses why this is okay, and explains that if developers keep their
test cases small, then this won't cause a problem.
However, having small tests means that you could end up end a large number
of test methods, all hand-coded. This doesn't lend itself well to data-driven
testing, or other scenarios where many tests are produced by a small amount
of test code that loops. If one JUnit test case is a wrapper on top of 1000
tests from a test database, then stopping at the first sub-test failure is
a large limitation. (This concern is certainly not unique to JUnit.) A possible
workaround is for the test generation process to spit out fully formed Java
test methods as a step before you compile the test.
Observations
JUnit is a wildly popular unit testing framework, used primarily by developers.
It is only relevant for Java code, unless you link object code from another
language into a Java program, or run your test in an external process. Most
of JUnit's functionality is accessed only from a Java API, but there is also
a command line and a graphical interface for running tests. JUnit is a fairly
simple toolset; what's powerful is that it has facilitated a surge in the
practice of unit testing.
JUnit is a good choice for a unit test tool if you have Java code and developers
who are willing to write unit tests. Ideally, the developers are doing "test-driven
development," where the unit testing is an integral part of the development
process. If you're developing in more than one language, maintaining a unit
test framework for each one will take extra work. I haven't thoroughly investigated
which unit test frameworks or integrated development environments support
multiple languages, but it's likely that there are commercial tools that
can do this.
A selling point for JUnit is that fact that the unit tests are developed
in the same language as the product code, so developers can work in an environment
that they're comfortable in. This selling point might not be as strong if
you're wanting an independent test team to develop black-box tests. If the
testers are already familiar with a different automation language, and they
don't need white-box access to the product implementation, then they would
likely be more productive with a tool designed for black-box testing.
It's exciting to see developers willingly doing unit testing, and doing it
well. JUnit can be a valuable toolset to facilitate the unit testing process.
Appendix: Sample JUnit test
The code below implements the minimal structure for a JUnit test case. There
are several subtest methods that illustrate various types of assertions.
Every method with a name starting with "test" is wrapped up into a suite
through the magic of the TestSuite constructor in the suite method. The user
interface is provided by the GUI or the command-line test runner, as in "java
junit.swingui.TestRunner MyJUnitDemo" or "java junit.textui.TestRunner MyJUnitDemo,"
so we don't even necessarily need a main() method.
import junit.framework.*;
public class MyJUnitDemo extends TestCase {
public void testAssertTrue1() {
assertTrue(1 == 0);
fail("we never reach this assertion");
}
public void testAssertTrue2() {
assertTrue("truth", 1 == 0);
}
public void testAssertEquals1() {
assertEquals("integer equality", 1, 0);
}
public void testAssertEquals2() {
assertEquals("string equality", "one", "zero");
}
public void testAssertEquals3() {
assertEquals("double equality", (double) 1, (double) .5, .2);
}
public void testFail() {
fail("do-it-yourself");
}
public static Test suite() {
return new TestSuite(MyJUnitDemo.class);
}
}