Open Testware Reviews Blotter 2003

Copyright 2003 by Tejas Software Consulting - All rights reserved.
2003-December-26

Andrew Sliwkowski provided some valuable feedback on the review of The Grinder.

Wow, that is quite the review!  After the first read, I have to agree with Phil and say the portrayal of the value of Grinder is clouded by detailed usability experience. I agree with a lot of points in terms of the Out of the Box Experience, but the review misses the fact that Grinder provides many hundreds of users, if not thousands, extremely high value for the initial investment.

Granted, Grinder has its weaknesses and strengths and like every other load testing tool, and it takes experience to bring out the optimum value. I have worked with most every commercial and freeware load testing tool known to man, and every one of them requires a 1 - 6 month learning curve before deriving real benefits.

It also was clear that nothing was at stake when you reviewed Grinder. It would have been nice to create an atmosphere of urgency and realism when defining the goals of the review. For example, most folks who use Grinder need to solve a particular problem urgently. I.e. Customer X calls up and says "Why is my web site slow?" or your boss says "We are losing $10,000 a day because of the response time/scalability of our web site... Find a way to fix it." In this way Grinder becomes a vehicle to help solve a problem rather than an arbitrary experience that sometimes fails to convey why users should consider putting Grinder in their testing toolbox.

Very good points Andrew. Indeed, it is difficult to review a tool as complex as a load testing tool in a month or less. Though I may miss reporting on the experience of using a tool after the initial learning curve, the out of the box experience is an important one that will determine whether someone will continue to use the tool in the long term. I'm also looking for ways to publish reports from more experienced users of the tools.

2003-December-9

Phil Aston, the maintainer of The Grinder, sent me his feedback on the review.

Generally fair, but I felt it could have accentuated the positive a little more.

I'll let you accentuate a few things here. I said in the review that I didn't think a server-class machine would be able to run more than 100 test scripts at the same time, to which Phil responded -

I frequently run many hundreds of threads on reasonably spec'd PCs. In my opinion, the fact the machine has a single connection to the network is usually more of a barrier to scalability than the CPU.

I talked about the fact that The Grinder doesn't pose as one of the common web browsers. Phil pointed out that it's possible to add a User-Agent header in the scripts, and that in the next beta release the TCPProxy program will record this header automatically.

I mentioned that the Console doesn't help with distributing the properties file or the test script to the agent machines, and Phil responded -

 This is the feature that is preventing me from marking 3.0 as non-beta.

Actually, since the 2.x release never did this, I don't consider it a barrier to production release by itself. I think it's more important that the Console be able to start up the Agent processes automatically (after some one-time configuration on the agent machines).

It's a shame you had such problems with Multicast. Its one of those things that when it works it works. I know there are cases where a unicast channel would be appropriate and I'm considering building this as an option.

I complained about the way the tests in the scripts have to be numbered, and Phil said -

The tests require a unique number, but it doesn't need to be sequential. The number can't be generated automatically because otherwise the scripts would need to create all the tests "up-front" to ensure that the same number was generated for the same test. It would also prevent you from configuring different test scripts for different worker processes.

This makes me realize that a data-driven approach would make the test scripts much easier to maintain, and would allow a script to dynamically generate the test numbers. If you're running two more more different scripts, they'd have to cooperate in order to generate unique test numbers, or perhaps instead The Grinder could identify different scripts by name instead. The next step up from data-driven testing is a keyword-driven front end that could make it easier for non-Python programmers to write test scripts.

Regarding a sample properties file that's compatible with The Grinder 3, Phil says -

There's one in the examples directory in the binary distribution. There's no example in the documentation because all properties have reasonable defaults.

I wrote "It would be nice if stopping the tests also stopped the statistics gathering," which prompted this response -

In practice, it's best to stop the statistics gathering first. Reset the processes when you're ready for your next run. The console model is the result of much "production" use during benchmark sessions. I have um'd and ar'd about linking the process controls and the statistics controls many times in the past, but have not yet found an elegant way to do this.

Oh, and well done on FAQ #1!

2003-December-8

Feedback from Jeff Bay on the review of The Grinder -

That is one of the best things I've read that you've written. Very detailed information, well presented. I definitely saw many of the same problems with the 2.x version that you point out in the 3.x version. I wouldn't suggest anybody try to use Grinder without programming knowledge at all. I would much rather have control over my load testing product and the ability to enhance it myself than use LoadRunner or other similar tool, however, I wouldn't expect a non-programmer to get any benefits from Grinder. The customization is too hard.

2003-December-2

Posted a review of another load test tool, The Grinder. Phew! Reviewing load test tools is brutal work.

Also, for those who haven't been curious enough to ask yet, I added an explanation of the logo design to the Background page.

2003-November-19

Muli Ben-Yehuda, the lead developer of the syscalltrack tool mentioned in Technology Bulletin: System Call Hijacking Tools, gave me his feedback on my comments about syscalltrack.

Syscalltrack is able to avoid calling the system call by means of returning an error value. I am not sure that returning '0' (success) without actually calling the system call is supported, but it could be fairly easily. As for modifying the system call parameters, there's no technical reason why it can't be done, but nobody has done it yet.

Muli said that syscalltrack can indeed modify a system call's return value -
[The error_code] is what you want the system call to return. The kernel always returns 0 for success and a negative error value for error, which glibc then translates into 0 for success and -1 for errors, and sets errno appropriately.

Muli explains why I had such a hard time with my memory allocation test -

Glibc uses two methods for allocating virtual memory, first, by mmaping an anonymous file, and the second by brk(). Which it uses depends on how much memory you request and the state of the heap at the time. Getting malloc to fail by injecting a fault in the underlying system calls will then depend on the state of the heap, which is hard to predict. You picked a tough test case :-)

Muli says he's actually using vanilla kernels; the packaged distributions may have trouble running syscalltrack because of the patches that the vendors apply. Regarding the need to patch newer kernels so they can run syscalltrack, he said he is working on integrating the module into the core kernel so it's easier to set up. For links to similar tools, besides the link on the Subterfugue page I mentioned, see also http://syscalltrack.sourceforge.net/relevant.html. One last point -

I would appreciate it if you could also mention one point about syscalltrack specifically - that it's free software, the code is available, and that we're always looking for volunteers to improve it. :-)

Done. :-)

2003-October-31

Posted the Screen Capture Tools Survey, covering 39 screen capture tools.

2003-October-8

It turns out that my e-mail actually did get through to Mike Coleman, the original author of Subterfugue (Technology Bulletin: System Call Hijacking Tools). Mike gives some further background on the tool, including a hint that it might be more actively maintained than I thought.

The name was meant as a play on subterfuge and fugue, where fugue can be taken in the musical sense (separate parts working together) or in the psychological sense (from m-w.com: "a disturbed state of consciousness in which the one affected seems to perform acts in full awareness but upon recovery cannot recollect the deeds")

I might also add that the project is not really active right now. It's being somewhat maintained by some kind souls, apparently coordinated by the Debian maintainer.

There have been a number of such projects in Unix history going back over time (see some of the links on the Subtergue site, for example). This technique is sometimes called "system call interposition" in the literature.

2003-October-1

Posted Technology Bulletin: System Call Hijacking Tools, introducing an additional review format. Started to use a web log format to post news, feedback, and new features.

2003-September

For issue 7, I take a black box test tool through its paces, and report on data comparator tools that serve as oracles for our test suites. Features:
I recently attend the Fall Software Test Automation Conference in Boston, where I saw that the interest in freeware test tools is really exploding. The subject came up in many of the talks. Hopefully you'll pick up some knowledge in these pages that will make you more marketable in this environment.

You'll see my shiny new ISSN (International Standard Serial Number) at the top of this issue. Open Testware Reviews is now registered with the US Library of Congress.

On the radar for future issues - I have my eyes on The Grinder and FIT (Framework for Integration Test), plus one more mentioned below.

Robert Sabourin commented on the Holodeck review in the August 2003 issue -

I learned a lot - everything I read was relevant. I especially liked the reference to the Linux tool syscalltrack in similar tools - you anticipated my question about what to do on Linux systems.

I hoped that the "free Holodeck sw included with a book" would be a fully supported freeware product - not a limited function / time expired demo! Oh well.

None of the released Holodeck demos are time-limited as of yet. But I was disappointed at the limited functionality, since the book cover didn't say it wasn't the full tool. I've had a chance to glance at Whittaker's new book, How to Break Software Security, and it looks like the CD it includes has a slightly newer version of Holodeck and perhaps some other updates as well.

Another reader asked for a review of syscalltrack. I'll see if I can work that in while still maintaining some variety in the types of tools covered.

2003-August

Time for for issue number 6, celebrating half a year in publication! Read on to find a tool that is very likely to be able to crash your Windows applications, plus a survey of tools that automatically design tests. Plus, there's a nice bit of reader feedback about last month's JUnit review.
I just got back from the O'Reilly Open Source Convention in Portland, and it was really amazing seeing so many of the big names in the Open Source world there. At the last minute, I heard that my lightning talk about free test tools was accepted, so I gave my five minute pitch to raise awareness of the rich set of freeware test tools out there. I got mostly blank stares in response, so I know I still have much evangelizing to do! The nice thing is that I ran into several people who publically admit that they like testing, and even one other person whose primary role is as a tester.

For more tidbits from the convention, including an unanticipated brush with fame, stay tuned for the August-September 2003 issue of the Tejas Software Consulting Newsletter.

In my JUnit review, I said, "At the time I started the review, there were 47 open bugs in the bug database, and 48 open feature requests. There hasn't been any appreciable progress on the bug backlog since the past JUnit release." Pat McGee responded:
Is this a problem? I haven't looked at the bug database, but I use it and don't have any problems with it. Are the reported bugs things that matter? I believe that if JUnit never had another update, we'd be OK. Not fine, but OK.
You caught me trying to get away with a neutral position. I don't understand the tool well enough to know how serious most of the reported issues are, though I don't recall getting bitten by any serious bugs. So I just reported the facts. Pat also responded to my remark that JUnit is not well-suited for data-driven testing:
The stated goal of JUnit is to support unit tests of methods. If you're testing a single method, then you most likely shouldn't be doing data-driven testing or lots of test cases. Conversely, if you're doing data-driven testing or lots of test cases, then you're not doing unit testing, and you shouldn't be using JUnit.
Point taken. I hesitate to conclude that you should never use a data-driven approach for unit testing. But perhaps it's rare enough at the unit level that we shouldn't expect our tools to handle it yet. And more insightful comments from Pat:
I don't understand the point of the example [test cases in the appendix]. All of these tests will fail. How does that explain anything to the reader?
If you run the test, you will indeed see each subtest fail. If they passed, it wouldn't  be nearly as interesting. My natural inclination when taking a test harness through its paces is to verify that when something goes wrong it actually gets reported. This is similar to the practice in test-driven development of running the test once before developing the feature that corresponds to the test, to make sure the test will at least catch the most catastrophic type of failure. Granted, this wasn't a good real-world example of a test case.

Here's Lisa Crispin's response to the JUnit review:

I've had trouble with the classpath too! But I'm REALLY a Java newbie. I'm amazed at the interest in test tools these days among developers. Not just JUnit, but also FIT, FitNesse and others. What they really seem to like is the visual red bar/green bar. Green makes them so happy. Makes me happy too!

I didn't see Mike Clark's JUnit Primer mentioned but it's a good one too.

Alan Richardson wrote about JUnit and other things.:

JUnit- This was an interesting overview of JUnit from a different perspective. I can identify with your difficulties using it as I also had difficulties starting to use it because although I know how to program in Java I'm not that experienced with it.

This kind of article is useful to testers as we have to know about tools that we can recommend to developers and we have to understand the tool to some extent in order to answer their immediate questions about it.

The InstallWatch article was excellent. I have used InstallRite when beta testing shareware programs and the authors find the reports useful. And I'd like to encourage you to do more reviews on tools which support the test process that other people might not be aware of.
So of course, I asked Alan what tools he wasn't aware of, and he said he wasn't aware of them. :-) But I did manage to coax a nice list of suggestions from him, including: screen capture tools, data generation tools, file comparison tools, synchronization tools (file & database), database front ends, mock objects, system emulators, and CD-ROM drive emulators. He also mentioned specific tools like the Microsoft Windows Application Compatibility Toolkit, GUITAR, FIT, TestMaker, The Grinder, and AutoIt. I'll be exploring this list as I plan the features for the next issue. Which ones would you most like to hear about?

Alan added that he likes to hear about "tools that don't promote themselves as test tools." The best example that I've seen in that category is a transistor radio. But most radios aren't freeware.

2003-July

Time for issue number 5! Don't be alarmed by the shift in publication date - I'm jumping the month ahead to match the practices used by other publications, so it'll match the month right after when Open Testware Reviews is published.

Each issue has taken a lot of work, but this time, I had to learn a new programming language to be able to do a proper review of JUnit. So we won't have a survey this month, but the survey will be back next month. Note that this month's review is the first time I've granted "Production" maturity status to a tool I've reviewed.
I had an interesting trip to USENIX 2003 down in San Antonio recently. You can follow the link to get a sneak preview of my trip report. I'm headed out to the O'Reilly Open Source Convention in Portland next week - I'll be sure to let you know how it went.

Our latest subscriber is the first that we have from Austria. Welcome! It's wonderful to see such an international response.

Linda Hayes responded to my mention of Worksoft's commercial tool, Certify, in the May issue:

Certify does not use the keyword driven approach, we are table-data driven. You might also reference that I discuss this in the Automated Testing Handbook.

She also suggested that "test framework" would be a better description for this class of tools than the "keyword-driven" term I chose. I think "test framework" is a bit too generic, in fact, that term came up in this month's review, and it doesn't refer to a keyword-driven or data-driven test tool. 

Rich Paschburg commented:

You mention keyword based testing in your excellent review of GUI tools. I believe the ABT Toolset (Test Frame) is now called Unified TestPro. Contact Ed Kit's company, at http://www.sdtcorp.com/ for information. We used Unified Test Pro at my previous job.

Actually, Rich, you've pointed out a third commercial tool for test frameworks/keyword-driven tools. Logigear sells the ABT Toolset, and it is not associated with Unified TestPro. I'm pretty sure there's a fourth one, too, if I can just remember who the vendor is.

Speaking of terminology and Logigear, Hans Buwalda tells me that the ABT Toolset actually does not implement the Test Frame methodology that he wrote about in his book, but instead implements "action words" and some more refined concepts that he has developed since the book was written. Hans agrees with Linda on the more generic term "test frameworks," though I'm searching for a qualifier to make it more clear. And one more point before I lay the issue to rest for this month - I discovered that the book Just Enough Software Test Automation has extensive coverage of this sort of test tool, that should help you learn about the technology whether you're looking for a free or a commercial tool.

In response to the ALLPAIRS review in February, Elisabeth Hendrickson offers an explanation of the difference between the all-pairs approach and orthogonal arrays:

All pairs simply requires that each possible pair of values occurs at least once.  By contrast, because orthogonal arrays are constructed from latin squares (where each value occurs once in each column and each row), they ensure that each value occurs an equal number of times.  So the chief difference between the two test reduction methods is that it is possible for a set of test data to meet the all pairs condition without meeting the orthogonal arrays condition, though orthogonal arrays will give you all pairs.

Your feedback brightens my day whenever it comes in! Please keep the comments coming, whether positive, negative, or just to ask for clarification. And don't forget to keep my up to date on what your preferences are for future reviews. You can reach me at faught@tejasconsulting.com.

2003-May

Welcome to issue 4! I've got lots of good freeware stuff to show you as usual. This time, I review a binary-only freeware tool for the first time. It's this sort of tool that caused me not to go as far as calling this "Open Source Testware Reviews." But even though you can't look under the hood, you can't beat the price for InstallWatch.
I have published my first StickyMinds column - Testware for Free: Adventures of a Freeware Explorer. I would have given you advance notice so you could have added your comments when the column was featured on the site, but it went from vague article idea to published column in less than a week when I heard they needed to fill the spot quickly.

OpenSTA (reviewed March 2003) version 1.4.2 was released on May 1, 2003. I haven't had a chance to take it for a spin yet, except for using it as a test bed for InstallWatch, but I have seen some activity in the bug database, which is encouraging.

To help prepare for guest reviewers, and also to help readers understand the reviews, I have expanded the Background Information page for Open Testware Reviews. Speaking of which, if you're interested in contributing a review either in the format you've seen here or something less comprehensive, let me know.

Check the updated Travels section of the Tejas Software Consulting home page for upcoming events I'm attending, all of which have some bearing on freeware. This includes two full-day conference tutorials coming up. Let me know if I should look for you at any of the events.

Last time around, a reader asked me to let everyone know what features to look forward to in future issues. So I'll take a guess, but don't quote me on it. :-) My feedback so far indicates that we need more coverage of test design tools, so I'll look into writing a survey of that category. For the in-depth review, I definitely want to get JUnit covered soon. I also had a request to review Holodeck, which seems to be going commercial on us, but I'll investigate whether a Holodeck Lite version is still available as freeware.

One reader commented on the "Sclc Metrics Tool" feature from April -

I think this article is clear - provides much useful information - quite self contained. It made me think - "I wonder what other code metrics tools are available?"

Thanks. The metrics tools survey is still ahead, though not a high priority at the moment because few people have requested it.

A reader suggested that this service might be more valuable to him if he could get reviews of the software he was most interested in at the time. Of course, not every review will be useful to everyone. But the easiest way to get what you want is to ask! I like nothing better than writing something that I know someone wants to read. Outside the scope of Open Testware Reviews, I can also handle special projects to review particular tools on your time frame and with your unique requirements in mind. I can also produce surveys that will help you get started on a tool evaluation, or do a head-to-head comparison of a few tools. Contact me for pricing.

Please take the time to send me your feedback on this issue and the service in general. I recently revamped the storefront page for Open Testware Reviews - I'd love to hear whether you like it better. And don't forget to send your friends there if you like this service!

2003-April

I'm picking up steam now with issue number 3 of Open Testware Reviews. I haven't even scratched the surface yet on all the freeware tools that are out there.
I've talked to a few subscribers about online forums for subscribers to discuss freeware tools. Opinions are split regarding whether to use a public forum, or a private forum for subscribers only. I can create a private forum if there's a strong interest, but otherwise I can recommend the public free-testing-tools list. I was not involved in setting up this list, but because of my frequent complaints about the low signal-to-noise ratio, the founder has given me administrator privileges and free reign to make improvements to the list.

I encourage you to join the 100+ members of this list if you'd like to have discussions about free test tools. I don't mind if you discuss Open Testware Reviews and quote parts of the features within the fair use copyright guidelines.

I've done two reviews now that have some interesting things in common (ALLPAIRS and sclc). Both are smallish scripts with only one developer, and neither has much project infrastructure. For both of them, I had to rewrite parts of the review several times. Why? Because I reported the difficulties I had with the tools as soon as I found them. Then the pesky authors started fixing some of the problems!  :-)  Contrast this with OpenSTA, which has formal bug tracking and more than one developer. There were several other people reporting OpenSTA bugs, though none in the same timeframe that I did. The reponse to my bug reports for OpenSTA has been much slower. So perhaps it's easier to get attention when you're a lone voice, and when your issues aren't drowning in a large backlog of problems. I'll be watching the results of the next review with much interest.

I don't mind the rewrites - it's a pleasant surprise that I've been able to have such an immediate impact on the software that I reviewed. I encourage you to interact with the maintainers when you use freeware tools. They'll be happy to know that someone is using their creation.

A few people have asked for advance notice of the topics for upcoming issues. I have to admit that I've been focusing on getting issues out to you as soon as I can, before I start making decisions about future topics. I'll see if I can do some advance planning the next time around. You might have noticed that I'm taking a tour of the testingfaqs.org categories, plus adding the occasional new category.

I was asked if readers can give suggestions about tools to review. You each hopefully received a survey form after you subscribed, asking for your opinions about topics to cover. I encourage you to continue to let me know where your interests lie. I welcome any guidance you have in choosing among the hundreds of available tools to review. I'll give you another copy of the survey upon request, or just email your ideas.

I've implemented some small improvements based on your feedback so far. So keep your comments coming! Please send your feedback on this or previous issues to faught@tejasconsulting.com.

2003-March

I hope you enjoy issue number 2 of Open Testware Reviews. Both of this issue's features, linked below, were specifically requested by subscribers. 
The ALLPAIRS tool, reviewed last month, is now at version 1.2.1. This latest release includes a bug fix (not one of the bugs that I mentioned).

I just received my copy of Free Software, Free Society as a part of my Associate Membership in the Free Software Foundation. There have been several books about free software in general in the last few years. I thought I'd share a list of the ones that I have on my shelf.

Actually, Free as in Freedom is more about storytelling than the business of free software. I love reading the folklore and history of the hacker culture, which includes the free software movement. But I'll save that book list for another time. As I was surfing around to get publication details about these books, I realized that there are plenty of recent books about free software that I need to add to my reading list. Here are the ones I tripped across, including two more in the folklore category that I snuck in. Let me know if you have a favorite book about free software that I missed.

Alan Richardson responded to the review of the ALLPAIRS tool in the February 2003 issue:

There is a statement in the documentation section "There's no formal definition of the user interface," which I didn't understand. I have briefly used the tool and know that it is command line driven so I wasn't really sure what this meant.

What I meant to say was that I didn't see man page-style documentation describing the command line options for the tool. The command line interface and the data file format is specified by example, though, and it wasn't hard to figure out how it works.

Another subscriber asked me to elaborate on the difference between the all-pairs technique and orthogonal arrays. I can't claim to be an expert on this topic, and I'm surprised that in the references I've seen so far, the experts haven't given a clear head-to-head comparison. I recommend that you find a good reference on Robust Design and the Taguchi Method. The way I understand it, orthogonal arrays allow you to zero in on which test parameter is causing a failure, while for the all-pairs method, you just know which test cases failed.

I received a question about the 1-5 scale for "Maturity" and "Project activity", and which end of the scale is good. See the Background Information page for details. I'll also reproduce the scales here.

Maturity -
  1. Planning - Working on requirements/architecture/design.
  2. Construction - Coding is underway.
  3. Alpha - Some features implemented, but users are likely to encounter failures.
  4. Beta - Most of the functionality is working well, ready for real-world testing.
  5. Production - Suitable for a broad end-user audience.
Project activity -
  1. Orphaned - No one is maintaining the tool, no visible user community.
  2. Inactive - No changes to the tool or documentation in two years or more. Few users.
  3. Stable - Changes are uncommon, but it's not hard to find people who understand the code.
  4. Active - New releases are made one to four times a year. Active user community.
  5. Very active - New releases more than four times a year. Active design and development efforts.
Clearly, the higher the maturity, the better, unless you're looking for a project where you can influence the design early in the project. Project activity may be best at "Stable" for a Production-quality tool, though you'd want a higher activity level if you want to see regular bug fixes or new features getting added. "Very active" probably isn't good unless the tool isn't past the Alpha stage yet.

Please send your feedback on this or the previous issue to me at faught@tejasconsulting.com.

2003-February

Welcome to the first issue of Open Testware Reviews! With your support, there will be many more issues to come. This issue has two features that you can access via the links below. In future issues, this section will include news items about freeware test tools, focusing on tools and categories that have been covered in previous issues. Stay tuned!

I'll also include some background information about the world of free and open source software. This time around, I'll point you to  the items on my web site that are relevant to freeware. I need your feedback here! There's no reason for you to only hear my opinions. Would you like to describe your own experience with any of the tools covered here? Do you disagree with anything in the feature articles? Do you have anything else to add? Please write to me at faught@tejasconsulting.com and put "Open Testware" in the subject.

  Back to the Open Testware Reviews Subscribers Page