Open Testware Reviews - Background Information

Last updated: 2003-12-02

This background information will help you understand the content of Open Testware Reviews.

1. General concepts

Why pick on open source developers?

It may not seem fair to review the work of an open source project. After all, the developers are usually volunteers. But the tone of Open Testware Reviews is different from a review of a commercial tool. The focus is on helping potential users of the tool decide whether they should use it. It is relevant whether someone is actively maintaining the tool, since not all users are programmers, and many users can't take the time to learn the code.

Open source developers have no obligations to maintain their code. Any contributions to the open source movement enrich the community, even if the code is a prototype from a research project that is no longer active. If people or companies are volunteering to maintain the tool, that's a wonderful bonus that we can benefit from. But a key point is that with an open source tool, we have the freedom to look at the source code and make changes ourselves, or we can pay someone with the proper skills to modify the tool for us.

The aim for Open Testware Reviews is to be uncompromisingly honest. Reviews of commercial software in the popular press are overwhelmingly positive, often ignoring any negative aspects. While Open Testware Reviews will focus on the tools that are most likely to be useful, which will result in an overall positive outlook, when we choose a tool to review, we want to document both the positives and the negatives as thoroughly as possible. There will be no advertisers who might sway us from this objective.

Scope - what is freeware?

Freeware is one of those terms that can mean many different things. I use it as a blanket term describing tools that anyone is free to acquire and use without payment. There can be no limits on: who can use it, the length of time it can be used, or the number of people who can access it. If any features are disabled, the tool must still be able to perform a useful task. Most demo versions of commercial software do not meet these criteria.

Freeware is even more useful if the source code is freely available, if you're allowed to modify the code, and if you're allowed to redistribute the tool and your modification to anyone else. Then it meets the criteria for "free software" as defined by the Free Software Foundation and is likely to qualify as Open Source software as defined by the Open Source Initiative.

Why the monkey?

The logo I chose for Open Testware Reviews includes a picture of a langur monkey. The monkey represents the curiosity that I exhibit while exploring free test tools. Langur monkeys were once thought to be rare with limited distributions, but recent research shows that they are widespread across a variety of habitats. I'm drawing the same conclusions about free tools.

2. Review content

Typical content for each section of the detailed reviews is described below.

Overview

This section documents the date the review was completed, the latest version of the tool that is covered by the review, both version and release date (sometimes earlier versions are also evaluated, and the text of the review will make it clear when this is the case), the name and email address of the individual or organization that maintains the tool, the tool's web site, the testingfaqs.org category where the tool is listed, the type of license, and the user interface (command-line, API, GUI).

This section also includes a summary of what the tool does and a general impression of the tool's usefulness.

Maturity

The maturity of a tool is judged using a five-level scale. The highest relevant rating will be used - for example, a production-quality tool that is having new features designed will be listed as "Production" as long as a production-quality version is still available.
  1. Planning - Working on requirements/architecture/design.
  2. Construction - Coding is underway.
  3. Alpha - Some features implemented, but users are likely to encounter failures.
  4. Beta - Most of the functionality is working well, ready for real-world testing.
  5. Production - Suitable for a broad end-user audience.
Generally, the higher the maturity level, the more likely the tool will work well for you.

Project activity

The project activity rating is based on how actively the tool is maintained and whether there is a community of users and developers. This rating is only loosely coupled with the maturity of the tool.
  1. Orphaned - No one is maintaining the tool, no visible user community.
  2. Inactive - No changes to the tool or documentation in two years or more. Few users.
  3. Stable - Changes are uncommon, but it's not hard to find people who understand the code.
  4. Active - New releases are made one to four times a year. Active user community.
  5. Very active - New releases more than four times a year. Active design and development efforts.
Depending on your preferences, you may consider either Stable or Active to be the most desirable. A Very Active project is probably Construction or Alpha maturity.

Platforms

The Platforms section documents the hardware and operating system requirements for the tool, along with any other prerequisites. Also, mentions how the tool integrates with other tools.

Support

Describes the following items, in terms of getting others to help you use the tool or being able to fix problems yourself:

Documentation

Discusses the documentation for the tool, including online help, documents installed with the tool, documents on the tool's web site, and a sampling of independent documentation and papers that are relevant.

Installation

Describes the installation process, whether it's an automated installation or a simple zip file.

Implementation

For those interested in working with the source (if available), this section will include the implementation languages used for the tool, the number of non-comment source lines, and a general impression of the comments in the code. Any issues with building the tool from source will be discussed. There will be discussion of whether available quality-control options from the compiler or interpreter are used, and whether any test cases are included.

Performance

Sets expectations regarding the size of project that can be feasibly tackled with the tool, in terms of time and machine resources.

Similar tools

Compares the tool to other available tools, both free and commercial, that perform similar functions.

Limitations

Lists problems encountered while reviewing the tool. Many of the issues documented here should be considered bugs, but others may be differences of opinion about the design of the tool. Summarizes any collection of known issues in an existing bug tracking system or mentioned in other documentation. Any issues that are resolved in a released version of the tool before the review is published will be listed separately.

Observations

An expansion of the overview, including information you should know about the tool that isn't included in other sections, as well as discussion about the overall user experience and what situations the tool might be useful for you.

Appendices

May include test code or extended logs showing the usage of the tool, if relevant.

3. Survey content

The format of the tool surveys continues to evolve. The date that the survey was completed is given, along with the relevant testingfaqs.org category. The tools will be listed in a table, with the name hyperlinked to the associated entry on testingfaqs.org. Other information that may be in the table includes the platforms, user interface, brief notes, and other data that's relevant to the type of tools listed.