Open Testware Reviews

The Grinder

Copyright 2003 by Tejas Software Consulting - All rights reserved.

Contents

Overview -- Maturity -- Project activity -- Platforms -- Support -- Documentation -- Installation -- Implementation -- Performance -- Similar tools -- Limitations -- Observations -- Appendix A: sample grinder.properties -- Appendix B: httpscript.py -- Appendix C: httpscript_tests.py

Overview

Reviewed: 2003-December-2
Version reviewed: 3.0-beta14, 2003-November-29
Maintainer: Philip Aston
URL: http://grinder.sourceforge.net/
Testingfaqs.org category: Load and Performance Tools
License: BSD, plus GNU LGPL 2.1, and Apache Software License 1.1 for external components
User interface: Java-based GUI (optional), command line (required for some interactions)

The Grinder is a load test tool used primarily for web load testing, but Jython programmers can expand it to exercise other protocols as well. It includes a proxy-based capture tool, a GUI test controller called the Console, and a Grinder agent that runs the worker processes that generate the load. Translations are available for English, French, German, and Spanish. Programming skills are recommended but not absolutely required.

I reviewed a beta release of the Grinder 3 line rather than the outdated Grinder 2.x. Here's a screen shot of the Console.

screen shot of The Grinder

Maturity

3 - Alpha (on a scale of 1-5)

Problems with running tests from more than one machine, holes in the documentation, and unfriendly exceptions motivated me to drop the maturity down a notch from the self-declared Beta rating on SourceForge.

Project activity

5 - Very Active (on a scale of 1-5)

Fourteen beta releases in the past year. The user community doesn't appear to be huge, but it's there.

Platforms

The Grinder is written in Java, and like most Java programs, there is no documentation on platform requirements, except that you'll need Java 2 Standard Edition 1.3.1 or later. I ran The Grinder successfully on Red Hat Linux 7.3 and 9.0, Windows XP, Windows 2000, and MacOS 10.1.

Support

There are three mailing lists for The Grinder: grinder-use, grinder-announce, and grinder-development. All see a decent amount of traffic. The grinder-use list is most appropriate for asking questions about using the tool. Tool users should also subscribe to grinder-announce, because announcements are not forwarded to grinder-use as you might expect. The maintainer, Phil Aston, was responsive to my questions.

There are SourceForge trackers for bugs and feature requests. When I checked, there were only four open bugs, and only 26 total bugs opened since July 2001. This is a low bug count for a moderately complex tool with a decent user following. Source code is available, and there is public read-only access to the version control system. There is a change log in the distribution that goes back to version 0.8.

Bug fixes are not issued for the 2.x version of The Grinder, only for the beta version 3, which can be a problem if you don't want to use a beta version of the tool.

I didn't find anyone offering commercial support for The Grinder.

Documentation

The Grinder ships with a User Manual under the docs directory in the distribution - point a web browser to "docs/index.html." Do not use the documentation under the "User Manual" link - that is for version 2.8.6. Go to "The Grinder 3" instead. Both versions of the manual are also available on the tool's home page. There is no online help built into the application.

The book Professional Java 2 Enterprise Edition with BEA WebLogic Server is what inspired the original version of The Grinder. It includes a chapter about the tool, and the tool is reportedly used in examples in other parts of the book. The newer book J2EE Performance Testing includes significant documentation and case studies for The Grinder.

The documentation for The Grinder that is easily accessible left me frustrated. It took me several hours before I felt like I was somewhat in command of the tool. I have the books on order; perhaps that will help. You should probably plan to buy the books, at least the newer one, before diving in to this tool in earnest.

The "External References" page in the documentation mentions a handful of online articles about The Grinder. Also, the paper "An Overview of Load Test Tools" compares OpenSTA, DieselTest, TestMaker, The Grinder, LoadSim, JMeter, LoadRunner, and Rubis. The paper isn't very well-written, but you might be able to get a few nuggets of information from it. The paper includes an appendix with a long list of additional load test tools, both commercial and freeware.

Installation

The Grinder installs from a 2 1/2 megabyte zip file. You can drop the contents anywhere in your filesystem.

You have to include lib/grinder.jar in your CLASSPATH environment variable or with a -classpath option on the Java command line. I was frustrated trying to run The Grinder from a Cygwin/Windows shell until I realized that the Java interpreter wanted me to use a DOS-compatible pathname in the CLASSPATH instead of a Unix-style name. Here are three examples of setting the CLASSPATH to point to where I unpacked The Grinder:
The source files are available in a 3 megabyte zip file and can be installed into the same directory as the binary distribution. There are a few files at the top directory level that are the same between the two distributions. Note that if you install the source and not the binaries, you will not have some of the ancillary files, such as the examples directory, the User Manual, and the TODO file. There is a CHANGES file in one package that is not quite the same as the ChangeLog in the other.

Implementation

The Grinder is implemented in Java, with 240 Java source files totaling about 35,000 non-comment lines of code. Also included are 40 test files (4200 lines of non-comment code), implemented in Java and using JUnit. Seventeen Jython example files in are included in the binary distribution but not the source distribution. Class definitions are heavily commented; comments in implementation files are sparse.

Performance

Java introduces some sluggishness in program startup sometimes. I didn't have a chance to thoroughly analyze the resource requirements for each test process and thread. I did notice that reducing the sampling interval in the Console seemed to increase the rate that the tests were executing.

I'd guess that a server-class machine would struggle to run more than 100 parallel test scripts.

Similar tools

There are several freeware tools that perform a similar function at a similar scope as The Grinder. The article "An Overview of Load Test Tools," while it doesn't give much detail about each tool, provides a nice overview of the landscape. Most of these tools focus on web load testing. There are many more load test tools out there that perform a more specific function, usually targeting a specific technology, and they usually have a command line interface. I'll publish a survey of these tools when readers tell me that this is something they'd like to see.

There are also several commercial tool option in this space. The ones I hear about most often are Mercury LoadRunner, RadView's WebLoad, and Segue's SilkPerformer. Commercial load test tools are likely to have more bundled features than the freeware options, especially system monitoring capabilities. The challenge with the top-tier commercial is that they can get very expensive if you want to simulate a large number of users. A tool may cost well over US$100,000 for a license to simulate 5,000 users. Tools from smaller vendors are likely to be much cheaper.

The benefit of freeware is that the only limits to the size of the load they can generate is the amount of hardware you have to generate the load. Simpler tools are also not as resource-greedy as the more feature-rich tools. There is easily a 10-to-1 ratio between the RAM requirements per simulated user for the most complex load test tool and the simplest.

Limitations

The Grinder doesn't include some useful features that often are available in commercial tools, such as:
Not having much experience with Java programs, I was frustrated that I had to type more than a simple program name to start one of the Grinder applications. To start the Console, for example, I have to set the CLASSPATH and then type "java net.grinder.Console." Granted, I could have used a script to simplify this.

The connection between the Console and the Agent processes is weak. The Console doesn't seem to know how many Agent processes are expected to respond when it broadcasts the start signal, so it doesn't report a problem when one of them misses the signal. (The tradeoff is that the Console doesn't have to be reconfigured when you add an agent machine.) The Agent processes themselves must be started manually on each agent machine. If you accidentally click the Start button on the Console instead of the Reset button, you'll have to start them all again by hand.

To configure each Agent the same way, it's recommended that you store the properties file and the script file on a file server that each Agent accesses. The Console doesn't help with distributing these files, nor with collecting the detailed log files on each agent machine. I couldn't share the properties file between Windows and Unix-style filesystems when it included a full path to the test script. The Console doesn't use the properties file, so it must be configured separately. If the Console indicates that a test had an error, it doesn't tell you which agent machine has the log file with the error message.

I was not able to get the Console to reliably communicate with the Agent programs on remote machines. This feature uses TCP multicasting, which is a technology I'm not familiar with. Multicast reportedly works with no additional configuration required on most systems, but it failed to work for me more often than it worked. I tried several different operating system combinations and none worked better than the other, except for one Red Hat 7.3 system that refused to set up the multicast connection at all (another 7.3 installation worked better). Running the Agent on the same machine as the Console was much more reliable, but it wasn't practical running as many as 50 concurrent tests - the Console became temporarily unresponsive because of the resources consumed by the worker processes.

I saw a number of exceptions in the various Grinder components. It seems that dumping out raw exceptions with stack traces is the normal reporting method for errors encountered by The Grinder. I only saw few pop-up error dialogs. There were places where the tool was a bit fragile, like when a CGI script returned a set-cookie header using an invalid syntax. This was a fatal error for the test script. On one hand, it's good that the error was caught, but on the other hand, this brought all testing to a halt until I patched the CGI script, which was already working okay when I used real browsers. I also saw a few "java.io.IOException: Stream closed" exceptions from an Agent process scroll by during a test run. I'd rather that those exceptions had been logged by the Console as internal errors so they were more visible.

I saw several other anomalies -
There was at least one case where the tool didn't do enough checking - if you misspell an entry in the grinder.properties file, the property will be ignored.

I reported a few minor bugs on Source Forge: 851910 - exception on invalid communication on Console port, 852217 - sample interval jumps after I set it, and 852298 - null pointer exception when multicast fails.

Observations

I chose to review the beta release of The Grinder 3 instead of the stable version 2 line because the maintainer is no longer fixing bugs in version 2.x and the addition of Jython-based scripting makes the tool more versatile (though perhaps a bit harder to configure). There are three components of The Grinder - the TCPProxy, the Console, and the Grinder Agent. The Grinder is used primarily for web testing, but with some effort you should be able to configure it to test across other protocols as well. It can also run JUnit tests.

I invoked "java net.grinder.TCPProxy -httpPlugin -console" to record a web browsing session. This tool uses the common web capture technique of recording a session as it passes through a proxy. I set up my browser to use it as my proxy, cleared my cache, and browsed a few web pages. I stopped the proxy, and then had a script file plus another file that the first script includes - httpscript.py and httpscript_test.py (see Appendices B and C). I didn't have to learn Jython to be able to capture the Jython scripts. However, as with any capture mechanism, further editing is required to create a realistic test, for example, if you want to verify any of the data on the retrieved web pages or modify the "think time." As with other such tools, I frequently forgot to turn off the proxy in my browser when I stopped the proxy, so I had to continually reconfigure the browser if I wanted to surf the web.

The Console is optional, but you'd really want to use it if you're marshaling more than one test machine to generate the load. The Console sends multicast messages out to start, reset, and stop the test, hoping that there are Agents out there somewhere that will hear them. The Console also listens on a port to receive performance data. As mentioned in the limitations section, the Console doesn't do nearly all the work needed to get the remote agent machines going. And because the multicast didn't work reliably for me at all, I implemented a workaround. I was inspired to try this workaround when I noticed that one Console could control an Agent while the Agent was reporting to a different Console on remote system. I set the grinder.receiveConsoleSignals property to false, and left the grinder.reportToConsole set to true (usually both are set to true or false, depending on whether you're using the Console or not). That way, I would start and stop the tests manually (I had to start the Agents manually anyway), but I'd still get the Console's consolidated reporting.

The core component is the Grinder Agent, which spawns the worker processes that execute the test scripts. You can run them entirely from the command line, or you can tell them to work with the Console on a local or remote system. You can run more than one Agent with different test scripts on the same system, and you can run Agents on more than one system, with either the same or different scripts. If you do use the Console with two or more different test scripts, you need to make sure to use different test numbers in each script so two different tests aren't reported on the Console as if they were the same. "Test" here means a single transaction, like one web page hit. A test script contains one or more tests. The Grinder Agent is configured via the grinder.properties file (see Appendix A).

The Grinder's strengths are the fact that it uses an existing scripting language for the test scripts, and its portability to any system that has a Java virtual machine implemented. Weaknesses include: an ineffectual Console that seems to require expert diagnosis for multicast problems, user-unfriendly error reporting, and spotty documentation. The Grinder lives in a fairly crowded space among freeware tools, and that space is likely to expand before it consolidates. So zero in on the tool that works best for you, and do what you can to bring it closer to production quality.

Appendix A: sample grinder.properties

grinder.processes=1
grinder.threads=5
grinder.runs=0
grinder.whydoesntthiscauseanerror=1
grinder.receiveConsoleSignals=false
#grinder.reportToConsole=false
grinder.initialSleepTime=10
grinder.script=/home/dfaught/grindertest/httpscript.py
grinder.consoleAddress=192.168.1.181

Appendix B: httpscript.py

#
# The Grinder version 3.0-beta14
#
# Script recorded by the TCPProxy at Dec 1, 2003 10:09:26 PM
#

from httpscript_tests import *

class TestRunner:
    def __call__(self):

        tests[0].GET('http://testingfaqs.org:80/')

        grinder.sleep(70)
        tests[1].GET('http://testingfaqs.org:80/favicon.ico')

        grinder.sleep(5347)
        tests[2].GET('http://testingfaqs.org:80/boneyard.html')

        grinder.sleep(4136)
        tests[3].GET('http://testingfaqs.org:80/faqchanges.html')

        grinder.sleep(131)
        tests[4].GET('http://www.google.com:80/logos/Logo_25gry.gif')

        grinder.sleep(15472)
        tests[5].GET('http://testingfaqs.org:80/t-load.html')

Appendix C: httpscript_tests.py

#
# The Grinder version 3.0-beta14
#
# HTTP tests recorded by the TCPProxy at Dec 1, 2003 10:09:26 PM
#

from HTTPClient import NVPair
from net.grinder.plugin.http import HTTPRequest
from net.grinder.script import Test

tests = {}


headers0 = ( NVPair('Accept', 'text/xml,application/xml,application/xhtml+xml,te
xt/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,
*/*;q=0.1'),
            NVPair('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
            NVPair('Accept-Encoding', 'gzip,deflate'),
            NVPair('Accept-Language', 'en-us,en;q=0.5'), )
request0 = HTTPRequest(headers = headers0)
tests[0] = Test(0, 'GET ').wrap(request0)


headers1 = ( NVPair('Accept', 'video/x-mng,image/png,image/jpeg,image/gif;q=0.2,
*/*;q=0.1'),
            NVPair('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
            NVPair('Accept-Encoding', 'gzip,deflate'),
            NVPair('Accept-Language', 'en-us,en;q=0.5'), )
request1 = HTTPRequest(headers = headers1)
tests[1] = Test(1, 'GET favicon.ico').wrap(request1)

headers2 = ( NVPair('Accept', 'text/xml,application/xml,application/xhtml+xml,te
xt/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,
*/*;q=0.1'),
            NVPair('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
            NVPair('Accept-Encoding', 'gzip,deflate'),
            NVPair('Accept-Language', 'en-us,en;q=0.5'),
            NVPair('Referer', 'http://testingfaqs.org/'), )
request2 = HTTPRequest(headers = headers2)
tests[2] = Test(2, 'GET boneyard.html').wrap(request2)


headers3 = ( NVPair('Accept', 'text/xml,application/xml,application/xhtml+xml,te
xt/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,
*/*;q=0.1'),
            NVPair('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
            NVPair('Accept-Encoding', 'gzip,deflate'),
            NVPair('Accept-Language', 'en-us,en;q=0.5'),
            NVPair('Referer', 'http://testingfaqs.org/boneyard.html'), )
request3 = HTTPRequest(headers = headers3)
tests[3] = Test(3, 'GET faqchanges.html').wrap(request3)


headers4 = ( NVPair('Accept', 'video/x-mng,image/png,image/jpeg,image/gif;q=0.2,
*/*;q=0.1'),
            NVPair('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
            NVPair('Accept-Encoding', 'gzip,deflate'),
            NVPair('Accept-Language', 'en-us,en;q=0.5'),
            NVPair('Referer', 'http://testingfaqs.org/'), )
request4 = HTTPRequest(headers = headers4)
tests[4] = Test(4, 'GET Logo_25gry.gif').wrap(request4)


headers5 = ( NVPair('Accept', 'text/xml,application/xml,application/xhtml+xml,te
xt/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,
*/*;q=0.1'),
            NVPair('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
            NVPair('Accept-Encoding', 'gzip,deflate'),
            NVPair('Accept-Language', 'en-us,en;q=0.5'),
            NVPair('Referer', 'http://testingfaqs.org/faqchanges.html'), )
request5 = HTTPRequest(headers = headers5)
tests[5] = Test(5, 'GET t-load.html').wrap(request5)