Open Testware Reviews

QMTest

Overview -- Maturity -- Project activity -- Platforms -- Support -- Documentation -- Installation -- Implementation -- Performance -- Similar tools -- Limitations -- Observations

Overview

Reviewed: 2003-August-30
Version reviewed: 2.0.3, 2003-February-25
Maintainer: CodeSourcery
URL: http://www.codesourcery.com/qmtest
Testingfaqs.org category: Test Drivers and Test Suite Management Tools
License: GNU GPL v.2
User interface: GUI, command line

After dealing with half-baked test harnesses and unit test tools with limited features, QMTest was a pleasant surprise. QMTest's feature set reminds me of my favorite features of a home-grown test harness I used to use. Tools like QMTest are useful for black box functional testers who want a tool that they can plug a suite of tests into and run the tests in various subsets and configurations. Common tasks such as checking return codes, comparing output to a recorded expected result, and summarizing test results are handled by QMTest so the test code can focus on exercising the application. Python programmers can extend QMTest in a number of ways using QMTest's extension capabilities. The user interface will adapt to the specific types of data that are needed by the newly defined modules.

While its limitations will put a crimp in how widely it's used for the time being, the tool is worth learning about if for no other reason than to study its design. I've heard that many users are using a more recent build of QMTest than the latest official release.

Rumor has it that QMTest came about as a winning entry in the Software Carpentry contest, which seems to have faded from existence as fast as it appeared. But QMTest very much lives on.

Maturity

3 - Alpha (on a scale of 1-5)

I had sincerely hoped that I could grant beta or production status to QMTest, but I have to rate the current release as alpha. This is mainly because of serious limitations with results verification on Windows, and data corruption and parallel execution problems on Linux (see the Limitations section).

I heard a recommendation to try the latest sources from the CVS server, which seems like wise advice since it has been several months since the last release. It does appear that a number of issues are fixed in the latest sources. I did encounter a spurious build warning that at first led me to believe that the build failed. And the check-rsh and check-threads test suites both hang on Red Hat 9. I'm told that check-rsh requires some setup that isn't documented, and check-threads may be hitting a Linux bug. Given the heavy activity on the CVS server, your experience will likely vary widely.

Project activity

5 - Very Active (on a scale of 1-5)

This rating is based mainly on the frequent patches that go out to the qmtest mailing list. Users are actively discussing design changes and many of them are submitting code changes. The development team is very responsive to submitted patches, and they follow a well-defined process for validating a patch.

The frequency of official releases has slowed over the past several months. The major goal for the next release is being able to scale to hundreds of thousands of tests, and stabilizing the API for extensions.

Platforms

CodeSourcery supports QMTest on Linux, Windows NT/2000/XP, IRIX 6.5, and "most UNIX-like operating systems." The documentation states that you shouldn't run QMTest from a Cygwin shell on Windows. It seems that the Cygwin incompatibility is isolated to certain configuration details, for example, when you use the Cygwin build of Python rather than a native Windows Python. I was able to start QMTest from a Cygwin shell.

Support

CodeSourcery, LLC, the copyright holder for QMTest, offers support, training, and consulting for the tool. No details about these services are given on the CodeSourcery web pages.

There is no public bug tracking database, though users are encouraged to post bug reports to the qmtest mailing list. The mailing list is very active and a good source of help. CodeSourcery representatives are active on the mailing list. However, the list's archives are not searchable, so it's hard to see what's there.

Source code for the 2.0.3 release is available in a 1.5 MB gzipped tar file. Source code is also available from an anonymous CVS server, were you can pick up the latest changes since the last official release. The source includes a detailed change log and release notes for each version.

Documentation

QMTest has a 41 page manual, available in html and PDF format both on the web page and in the local installation directory. The html manual is linked from the tool's web interface. The manual is fairly complete and accurate.

Other documentation is available from a "--help" command line option, README files, and other miscellaneous documentation files that install with the tool. There is also a link to "internal documentation" on the CodeSourcery web site.

If you build QMTest from the CVS sources, you won't get the documentation by default. The documentation link in the web interface will then be broken. When I tried to build the documentation, it didn't work because I didn't have the pdfjadetex tool installed. This tool isn't mentioned in the documentation.

You can see a bit about how one organization has set up QMTest at http://spi.cern.ch/extsoft/qmtest.html.

You can get some insights into the market positioning for QMTest by reading a discussion thread "QMTest and the G++ testsuite" started by QMTest ringleader Mark Mitchell in 2002.

Installation

Binary distributions are available in these formats:

Windows - 1.2 MB self-extracting executable
Linux - 1 MB Red Hat 8.x rpm
Linux - 1.2 MB gzipped tar file

I installed QMTest on Windows XP Home Edition without incident. It did not install any shortcuts on the Start menu or on the desktop. A shortcut to a program that would allow you to choose a repository and start the GUI interface would be nice. There appears to be no uninstall option for Windows. Uninstallation probably just requires removing the "\Program Files\QM" folder.

I successfully installed the rpm file on Red Hat Linux 9. For other systems, presumably you can build and install from the source package.

Implementation

QMTest 2.0.3 is about 23,000 total lines of non-comment code according to sloccount (counting test code and other auxiliary files). I couldn't use my favorite metrics tool, sclc, because it can't parse Python code. Most of the code is Python, with a smattering of shell scripts and C code. I count only 17,000 lines of code in a newer version of QMTest that I pulled from the CVS archives. This drop seems to be attributable to a big reorganization of the zope-dtml files; the core QMTest implementation has grown quite a bit since the last release.

There is a detailed style guide for QMTest developers, plus a rigorous procedure for submitting changes that seems to be strictly enforced.

There are four test suites that are executed if you run "make check." The source also includes a throughput benchmark.

Performance

The performance of a test harness like QMTest generally is not an issue. The test cases that the harness executes will occupy the bulk of the execution time. So the design of the tests and the performance of the system under test are the largest determining factor for performance.

Similar tools

Open Testware Reviews featured a survey of black box test drivers in the March 2003 issue. The general-purpose tools in the survey besides QMTest were Aegis, CITE, DejaGnu, STAF, TCM, and TET. However, native GUI or web interfaces like QMTest has are rare in freeware test harnesses.

Since publishing the survey, I've run across FIT (Framework for Integration Test), which also has a web-based interface. FIT is worth a look.

There are a number of commercial tools that are likely more sophisticated than QMTest or tailored for specific technologies (QMTest's extension features give you the opportunity to fill in many of the gaps). Some of them also work well for managing suites of manual test scripts, which is not something QMTest is suited for.

Limitations

I ran across a few serious bugs while running QMTest (most or all of these are fixed in the code in CVS) -

Twice I hit a page saying "You Have Hit a Bug in QMTest!" when trying to view test case details. It appears to be a data corruption issue, because I could stop QMTest and restart it, only to get the same error on the troublesome test cases. I noticed that the newer version of QMTest I built from CVS could read the corrupted test cases without error.
When running two-way parallel, once I got an "All tests have not yet been run. These results will be updated every few seconds" message that never went away. I was running a very small set of tests. The Run->Stop menu item did not stop the execution.
Stdout comparisons on Windows will always fail because of issues with Windows line endings.
Exit code comparison doesn't work on Windows, apparently because of a limitation in Windows. This is documented, but still possibly surprising because there are no cues in the interface about this limitation.

The web interface does not expose all of the same functionality that is available from the command line interface. The differences I found were -

I can't find out which version of QMTest is running.
I can't find out which test database I'm using unless I match the web server port with the output on the command line.
I can tell the harness to run tests in parallel on the command line when I start the web interface, but I can't adjust this after I start the tool.
I can only use the option to run tests in random order when running tests from the command line.

There are a few glitches in the interactions between QMTest and Netscape on Windows. If the last Netscape window in focus was not a Navigator window, it'll fail to open the web page. Or if the Netscape windows are minimized, they'll stay minimized even after loading the web interface. This is a Netscape problem I've seen in other situations. When I run QMTest from Cygwin, it tries to start Mozilla, which isn't installed. Luckily, when any of a number of glitches like this occurs, it's easy to enter the URL manually into any web browser. One more browser issue - if Javascript is not enabled in your browser, none of the menus in the browser-based interface will work, and there's no warning about what the problem is.

I have a few concerns about concurrency issues and remote access to the web interface -

There's appears to be no protection against two people accessing and modifying the test database at the same time, from multiple accesses to the same web interface, multiple web servers in the same test database, or command-line accesses to the same database.
While using the web interface remotely, I can't load or save expectations files that are in the test database without remote access to the filesystem.

Here are some random observations about QMTest functionality -

It's difficult to know how many total tests are in a directory structure or suite without running the tests.
Timeouts are set on a per-testcase basis, with no global default. I'd like to be able to set a timeout globally, to make sure no test can hang up the test execution.
There's no way to set up exclusive access to a resource when more than one test is running at the same time.
When the stdout or stderr doesn't match the expected results, I don't get diff output that pinpoints where the difference is.
I have to be careful about saving test results and noting manually when the tests were run and which version of the system under test they were run against. I don't get an automatic date stamp in the test results.
Expectations are based on a high-level category. If a test starts failing in a different way within the same category, it's still an expected result.

Observations

When I first looked at QMTest, I was excited to find all of the basic features that I like to have in a test driver for black box functional tests, like comparing standard output and standard error, organizing tests into suites in a flexible way, and keeping a record of expected results after test failures had been analyzed. I had been looking at unit test tools before then, and I very much missed these familiar features that are so rarely available in a unit test environment. However, it turns out that the devil is in the details, as you can see in the Limitations section above.

QMTest has several test "classes" available by default, including one that exec's the test program directly, one that calls the test through a shell, and one that embeds an entire shell script into the test case. There are also a handful of test classes that allow you to embed Python code directly into the test database. Users can also write their own classes. So I have hope that I could learn just enough Python to write a class for Perl. :-)

QMTest can run tests in parallel on one machine, or it can spread tests across multiple machines. It can run tests in random order to look for dependencies between product features. It can rerun the tests that had unexpected outcomes in a previous test run, so you can see if a new version of the system under test has improved. You can set "expectations" as a way to filter out failures that have already been analyzed and reported.

There is a resources feature that is intended to allow for setting up complex resources only once for a test run. The only resource class that's currently defined is designed to set up a scratch directory, but extensions could expand upon this. In fact, several times I started to complain about certain limitations, but then I realized that the limitations were in extensions that could be replaced with a little programming effort.

QMTest has a command line interface that you can use for non-interactive test runs, and a web interface that shields you from many of the low-level details. While the web interface is more limited in its design than a native GUI would be, it's also more flexible in terms on running from a very wide variety of desktop systems.

I'd recommend QMTest for Linux and Unix users who are comfortable with pulling new versions from CVS in order to take advantage of bug fixes over the last several months. Windows users who are accomplished developers may benefit from trying a current build of QMTest on Windows. QMTest provides a well-rounded set of features for basic functional testing, plenty of extensibility, plus commercial support if you need it.