Open Testware Reviews

Test Design Tool Survey

Reviewed: 2003-July-28
Testingfaqs.org category: Test Design Tools

This time around I survey a category not often explored by testers, test design tools. Newly posted to testingfaqs.org for this survey are AllPairs.java, ADL, tsl.

About test design tools

Test design tools can automatically specify test cases. That doesn't necessarily mean that the tool will give you a completed automated test or a manual test script, though at least one does claim to generate test code. Generally, you will use the output of a test design tool as part of a test design specification, and you would do additional work to convert the design ideas into working test cases.

These tools can help with building up a list of tests. A few of them focus on paring down an astronomically large set of possible tests to a feasible list. Whichever of these is your need, these tools can be a big help in the test design process. However, you need to have a good understanding of what these tools do in order to know the risks they introduce and to know what other tests are needed for your product.

Most test generation tool do not include an "oracle," that is, they generate the test input, but they do not generate the expected results to compare against. If you generate a large number of tests, you may have to create a large number of expected results by hand. See Harry Robinson's Model-Based Testing web site for links to much more information.

About the matrix

This time I've incorporated notes about each tool into the matrix. Most tools listed here use equivalence classes as a foundation, that is, they have you represent the possible values for input parameters with a few ranges and boundaries that only need one test each. Three use the all-pairs or all-tuples approach, the others divide up the solution space in different ways.

Tool	Platforms	Notes
ALLPAIRS	most (any platform supported by Perl)	Generates test specs using the all-pairs method, which reduces combinations of attributed to a manageable but effective set. Reviewed in the February 2003 issue of Open Testware Reviews.
AllPairs.java	most (any platform supported by Java)	An all-pairs tool written in Java, based loosely on ALLPAIRS.
Assertion Definition Language (ADL)	POSIX (Unix, Linux, BSD, Cygwin/Windows NT)	Automatically generates tests based on API specifications for C/C++, IDL and Java. From The Open Group, commercial support available.
DGL	Unix and Windows ActiveX control	A venerable entry on testingfaqs.org. Originally designed to general functional level tests for VLSI designs, but reportedly works just as well for software. Can randomize test data.
jenny	most (binary for Windows, C code can be compiled on Unix)	Another all-pairs tool that recently emerged. Can also handle triples, and all-tuples in general, as well as allowing you to specify impossible combinations that should be avoided. The output can be difficult to interpret, though.
Multi	most (any platform supported by Java)	Give it a boolean or relational expression, and it'll suggest tests that exercise the boundaries of the expression.
tsl	HP-UX 9 & 10, SPP-UX, ConvexOS	Starts with sets of equivalence classes like you would when using all-pairs, and generates the complete cross product of all possible test combinations. Can be very CPU-intensive. Allows you to use constraints to pare down the list. Hasn't been used in years, may take some work to compile.

There are two tools that are not on the chart but that deserve mention. One is called FASTER. I haven't posted it on testingfaqs.org yet because it's likely to be alpha quality and the release hasn't been consistently available lately. FASTER generates tests from an XML schema using equivalence class analysis. Then there's Segue's SilkPerformer .NET Explorer, which is listed as freeware on testingfaqs.org, but I'm re-evaluating its freeware status. I believe that it only works if you have a license for SilkPerformer. While it makes sense to me to list a free add-on for a commercial tool as freeware if it were available from a different organization, if the add-on is available from the same vendor as the main tool, then it seems that it's logically just a separately packaged part of the same tool.