Open Testware Reviews

Black Box Test Driver Survey

Reviewed: 2003-Mar-22
Testingfaqs.org category: Non-GUI Drivers and Test Suite Managers

This time around, I survey tools that provide a generic mechanism to run suites of black-box functional tests. They may be called test drivers, test harnesses, test management tools, test scaffolds, and probably many other terms. I'm specifically excluding GUI test tools because they're in a testingfaqs category of their own. I'm also excluding unit test tools, because there are enough of them to also form their own category. I list some unit test tools under "Test Implementation Tools" on testingfaqs.org, but soon I expect to create a new category for them.

The test driver tools are listed below. Click the tool name to see the testingfaqs.org entries. (I added three new test driver entries to the site during the course of this survey.) The User Interface column describes the interface that the tester uses to access the tool. This does not reflect on the user interface of the software under test. So while these tools can't automate testing for a GUI interface, a few tools do have a GUI interface themselves.

Tool	Platforms	User Interface	License
Aegis	Unix	Command line, read-only web interface	GNU GPL
Canoo WebTest	Platforms that support the Java JDK version 1.2 or higher.	Ant command line or IDE, API	Equivalent to The Apache Software License, Version 1.1
cg-eye	Any platform supporting Perl and the URI::Escape module. Parts of the tool are hosted on the Web Design Group's web page.	Command line, web	GNU GPL/Artistic license
CONVEX Integrated Test Environment (CITE)	ConvexOS, SPP-UX, HP-UX 10	Command line (GUI front end available but obsolete)	Similar to the Apache Software License, Version 1.1
DejaGnu	Unix, Cygwin/Windows NT (maybe Windows 2000+)	Command line	GNU GPL v2
QMTest	Linux, Windows NT/2000/XP, IRIX 6.5, Most Unix-like operating systems	Web, command line	GNU GPL v2
Software Testing Automation Framework (STAF)	Win95/98/Me/NT/2000/XP, Linux Intel,PPC64,S/390, Solaris 2.6+, AIX 4.3.3.0+, HP-UX 11.00+, Irix, AS/400	Command line, GUI provided by the STAX front end	GNU LGPL v2.1
Test Case Manager (TCM)	Windows + Microsoft Access	GUI	custom, source provided
Test Environment Toolkit (TET)	AIX Release 4.1.1, 4.2, 4.3, HP-UX Release 10.01, 10.10, Solaris Release 2.4, 2.5, 2.6, 7, 8, Linux, Red Hat 5.1, 6.0, 6.2, UnixWare Release 2.1.2, 7.0, OSF/1	Command line, API	modified Artistic license, additional licensing applies to contrib directory

I did have the Expect scripting language listed in this category, but I have moved it to the Test Implementation Tools list. You can use Expect to build a test driver, but by itself, Expect is not a test driver. I may eventually create a new category for scripting languages and move it yet again. Note that DejaGnu is indeed a test driver that's implemented using expect.

Of these tools, STAF and QMTest peak my interest the most. One uses a convenient web interface, the other has some sort of optional GUI front end. Both are general-purpose and run on a broad set of platforms. Also, TCM has a bit of a following, and might be a tool you'd be comfortable with if you're only testing on Windows and you have a database mentality.

TET was originally written to support standards testing, and has been around for a long time. It still uses only a command line interface, unless you pony up for the commercial version, called TETware. DejaGnu also only has a command-line interface, and may be a good choice for testers who already know tcl or need to use Expect to test interactive command-line programs. Aegis is actually a complete command line-based configuration management system that includes a test driver. The test driver would likely only be useful if you need the full range of Aegis' functionality.

There are two web-specific tools. Canoo WebTest is designed to test web applications, and has the unique quality of using the Ant build system as its user interface. Cg-eye focuses on testing CGI scripts. I need to look into it further to decide whether to classify it as a unit test tool instead.

Then there's the beloved CITE toolset that I used for a number of years at Convex Computer Corporation, where we had thousands of tests under its control. During its heyday, several people outside the company requested a copy, but the company refused. After collecting dust for several years, it has now been released under an open source license by Hewlett-Packard, the current owner. I don't know if anyone has tried to compile it since it was released as open source. There are actually several different tools in the toolset that span most testingfaqs.org categories, including an old analog-style GUI test tool. It's not clear whether there's any value in this toolset compared to the other options that are available, but it's nice having it available now so we can decide for ourselves.