Making
a mockery of your objects
2005-July-25
I've been aware of the concept of mock objects for a while, but hadn't been motivated to try them out until I had the opportunity to unit test some test libraries I was writing in Perl. So again I'll subject you to my preference for Perl, but hopefully many of the concepts will apply to other languages as well. A mock object is used to replace a real object used by the code under test. You're not replacing the code you're testing, just an object that the code uses that's making the testing difficult, either code in some other part of your project that the code under test depends on, or perhaps an object provided by a standard library.
I'm now a fan of mock objects within the context of unit testing. Here's an example of how I've used them. I was taking a crufty one-off script that I wrote that automates a test for a sort of web service, and converting it to a module that could be used for more than one test. My module uses Perl's LWP::UserAgent module to emulate the behavior of a web browser. For a proper unit test of this test code, it should not be necessary to set up a real web server for the library to access, or even to have it access a network at all. A feasible way to accomplish this is to mock the LWP::UserAgent class (to be precise, we'll be mocking the class outright here, not just one object). I'm going to replace the entire class with something of my own creation, with lots of help from Perl's Test::MockObject module. Now just to be clear, I'm testing a library I'm writing that will be used in programs that test a web service. This library uses LWP::UserAgent, which is someone I'm hoping has already been well-tested, and that's what I'll be stubbing out with a mock class.
I wrote a unit test script for the library. Here are the important modules I invoke at the top of my test script:
Before loading the module under test, here's how I set up my mock class:
I create a new mock, and then tell it to masquerade as the
LWP::UserAgent class. The LWP::UserAgent module will not be loaded at
all when the module under test asks for it. Then I call fake_new to
establish a do-nothing constructor, and I mock the one other method
that
the module under test uses - the "request" method. For now, this method
will do nothing at all except return 0. This suffices for my first
several tests, which just verify that the module compiles and properly
reports some error conditions, without ever calling the request method
that normally would try to contact a web server.
Then I need to see how my module handles a "not found" error from the web server. This is where I need to create an HTTP::Response object, which is what the LWP::UserAgent request method returns. So I create a response that indicates a 404 error, and then I tell the mocked request method to always return this object without ever looking at what URL it's asked to retrieve:
My test script then invokes my test library and verifies that it throws
an exception after getting the 404 error. I only use this 404 response
for one test before modifying the request method yet again. Here I
create
a response object indicating success, but fake up a message from the
web service that indicates that the transaction ended with an error.
Again we're looking to get an exception from my test
module:
It's very nice to be able to precisely set up this scenario with only
three lines of code and no real web site anywhere. As I delve more
deeply into the code I'm unit testing, I have to make the scenarios a
bit more complex so my mocked class appears just barely functional
enough to fool the module I'm testing and let it continue on to the
code I'm targeting with each test. I'll give one more code example,
leaving out the parts where I created two more response objects:
The next time the module under test calls the request method, it'll get
the $okResponse object I created. It will then call the request method
again, and get the $quitResponse object, indicating a successful end of
the transaction. I can string together any number of sequential return
values for the request method, or write my own method that implements
more complex semantics and plug it in to serve as the request method.
Another feature of Test::MockObject is that it records all of the calls to its mocked methods. I used this feature to verify that the data that the test module sends up to the server (or would have sent if there really were a server) is correct.
Now I should explain what kind of code works well with mock objects. First, it's far easier to unit test a module rather than a command-line program. So even if your code will only ever be part of a single command-line or GUI program, if you want to unit test it, it's best to put most of the code in a module (or library, etc., if that's what it's called in the language you're using), and then write a small wrapper on top to serve as the user interface. This has been the conventional wisdom for black box testing for ages, and it applies to unit testing even moreso. And also, it's much easier to unit test object-oriented (OO) code than non-OO code. It suffices for the purposes of mocking if you have non-OO code that uses the OO interface for the module you want to mock. My earlier frustrating experiences in stubbing out function calls and built-in commands in a massive non-OO program through the user interface are documented in my article "Diving in Test-First."
Additional reading: for more on mock objects in Perl see "A Test::MockObject Illustrated Example" by chromatic, and for an introduction using Java see "Mock Objects" by Dave Thomas and Andy Hunt.
Stay tuned for a listing of mock object libraries for various languages.
Whaddya mean, ports? We've got no ports here!
2005-June-15
I recently needed to test a feature in an application that worked something like this: the application will try to listen on network port 1000, and if that port is already in use, it will try ports 1001-1099 in sequence until it succeeds. So of course, I wondered what happens if all ports are already in use. I was happy to find that writing a Perl script that can "squat" on a given port and thus make it unavailable to other programs on the same computer was easy to do.
Here's my "portsquat" script:
You feed it a port on the command line, and optionally a second port to indicate a range of ports that need to be opened. A fraction of a second later, the script indicates that the ports are open. In the example above, I would run "portsquat 1000 1099". It then does nothing until you're done with your test and you kill portsquat, which immediately causes the ports to be released again. If it can't open a port, it'll just issue a warning and continue trying to open other ports in the range.
I ran the script successfully on Windows with ActiveState Perl and Cygwin Perl, as well as Mac OS X. I got a nice crash to report on one platform, and a missing error message on another. So the next time you test an application that acts as a network server, try to deprive it of its ports and see how it reacts.
If anyone wants to convert this to their scripting language of choice, I'd be glad to post those here too.
All in Good Time
2005-June-3
I sometimes do informal performance testing where I need to have a stopwatch. On a project not too long ago when I needed to test how long a device could run off of a capacitor after I removed the battery, I followed the time-honored process of googling for the first free program I could find that seemed to work. One of the first options I tried was TimeLeft, which worked well enough for what I needed.
TimeLeft
is a program for Microsoft Windows that features a stopwatch, a
countdown timer, a clock, and who knows what else I haven't taken the
time to explore. It's available in a freeware version from NesterSoft,
based in Canada, and there is also a commercial version and additional
add-ons available. I use the stopwatch feature pretty much exclusively.
Someone programmed an amazing array of chrome into it, with multiple
skins and animations to choose from. I find the start, pause, and reset
buttons to be annoyingly small and hard to hit when I'm in a hurry. I
also haven't gotten used to the fact that I have to reset the timer
every time before I start another timing, or else it'll just start
counting from where it left off before.
A few days ago when I needed to follow up on a hunch about performance problems for a workload I had set up to exercise a peer to peer program, I knocked the dust off of TimeLeft. It worked fine on the system where I had it installed, but I was too lazy to install it on all the systems I was using. And I didn't have the option of using it on Mac OS. On Mac OS I experimented with the stopwatch feature in RememberMe, which was awkward because the stopwatch is bolted onto an appointment calendar that I can't get rid of, and it has a strange habit of minimizing itself to the dock after I start it.
So in some cases I resorted to using the second hand on my watch and trying to remember where it had started from. One advantage my watch had over a software stopwatch - I could initiate an action in the program under test and start timing at essentially the same time. With TimeLeft, the mouse can only do one thing at a time, so there's a small delay before I can get both the application and the stopwatch going, which makes the results likely to be off by a few seconds.
I've been using an older version of TimeLeft but just upgraded to version 3.06. It didn't take long to find a few minor bugs in the new version. But once you get used it, it does a decent job of ticking off the seconds. Almost as good as a real stopwatch, which I do need to acquire eventually.
My Monkey
2005-May-12
I've been experimenting with monkey testing through a Windows GUI lately, that is, using a test script that clicks randomly on the screen. I've been surprised at how many bugs it's found in the application I'm testing. I thought I'd write about the technique I used to develop my monkey and the free tools that made it possible.
I started with the script in the article "PerlMonkeyTest.pl - monkey testing for any Windows window." The article gives a good proof of concept for using Perl and its Win32::GuiTest module for a monkey test. But it needed a lot of work to be a robust test.
I gave the monkey some training. The sample script is a dumb monkey. It does put the target application in the foreground, but then it clicks all over the screen, often clicking outside the application. Even if the app you're testing is maximized, it still might click on the Windows Taskbar, and it might resize or minimize the application. I had my monkey check the actual screen geometry of the application, and to prevent it from getting partially obscured offscreen or preempted by the Taskbar, I keep the monkey off the title bar, the window border, and the resize widget on the lower right corner. Maybe some day I'll make it smart enough to safely handle resizing and moving the window.
I found that you can't do much in Windows when you press more than one mouse button at a time. The original monkey script did this frequently, especially when I added the ability to click the middle button, so it wasn't accomplishing much. So whenever my script decides to push a mouse button, it releases any button that's already down. It still can do drag operations, though, because it handles button down, move, and button up events separately.
I tweaked the random decisions a bit, so that the events aren't so overwhelmed by button actions as compared to mouse movements. And I increased the likelihood that a mouse movement would be less than 10 pixels away, so it's more likely to activate items on the menu, though it still only rarely finds the menu bar.
Once I had restricted the mouse movements to the parent window, I had to deal with all the dialogs that can pop up. First, I ignore dialogs that aren't modal, i.e., if I can send the focus back to parent window, I just continue testing the parent. If a child window is set to stay on top and it happens to overlap the parent window on the screen, then the monkey will click on the part that overlaps, and possibly resize it as well. If a modal dialog pops up, then I find a way to dismiss it so I can get back to testing the parent, either by looking for a button or by typing Alt-F4. In a few cases, I wait for transient dialogs to go away on their own, because otherwise I'd be canceling some interesting activity the app is doing, and sending Alt-F4 to a transient dialog could end up killing the parent. So the monkey has to have a bit of smarts about what to do with each dialog, though there are only a few different categories of actions it takes.
There are many things I could do to train the monkey further, for example, by having it send random keyboard input, and perhaps activating menu items much more frequently. Also, it needs to do better at detecting when the app exits normally because the monkey found File->Exit on the menu.
For long unattended test runs, I found it's important to take a video capture of the screen. It makes for a far more effective bug report when I can explain what the monkey did right before an error. It's very difficult to determine the state of the application just using a log file of mouse coordinates. I have yet to find a free video capture tool that works well with long test runs and doesn't use a lot of CPU power when recording, so I'm experimenting with commercial recording tools. Because all but the smartest of monkey tests are not good at detecting failures, especially minor failures, I've found a few additional bugs by watching the monkey in action in real time.
It's not easy to find a recent version of the Win32::GuiTest module, especially if you don't have the proprietary compiler required to build from source. I found recent pre-built packages for ActiveState Perl 5.8 in the Win32::GuiTest project on SourceForge.
2005-March-9
Here's an update on the Mantis bug tracking tool. I reviewed Mantis 0.18.2 in May 2004. The most recent version is 0.19.2, which the project team I'm working with now is using for their bug tracking. I have not encountered any notable reliability problems in my recent use of Mantis. Of course, I'm not actually trying to break it. :-) I haven't really needed the documentation, mostly because I'm not the one maintaining the server, so I don't have anything new to report there. We did have a bit of trouble with email notifications when we first upgraded from 0.18.x, but our server admin was able to get it working in short order. I have not tried the CSV export function recently. We're not taxing any of the more advanced features very heavily.
One of the reasons I pushed the team to upgrade Mantis to 0.19.2 was because it was difficult to manage priorities. I couldn't filter on a particular priority, and the priority column on the View Issues page was always blank. After the upgrade, I can now filter on priority and I get icons in the Priority column. I can filter on a date range for the date of the last update, but not on any of the other date fields. There's an interesting "My View" page that shows the first 10 bugs found for 6 types of common queries. The "Assigned to Me (Unresolved)" section would be useful if it also showed resolved bugs, since most of the bugs in my queue are resolved and waiting for testing.
While it is a bit more flexible, I still find the filter mechanisms a bit clunky. The filter setting is global across my account, so if I filter on resolved bugs in one window and on recently modified bugs in another window, when I update the first window it takes on the filter setting of the second one. I frequently need several different views of the database as I look for other bugs that are relevant to one I'm investigating.
I still find the basic bug reporting screen very basic, and the advanced screen too cluttered. But after reading Joel Spolsky's compelling appeal for making the bug tracking database design simple, I'm content with simply including the relevant information in the Description and Comment fields rather than insisting on a custom field for each type of information.
The default workflow doesn't make much sense in a few places, and it doesn't look like the workflow can be changed via the web-based user interface. For example, a bug is considered resolved when the programmer marks it fixed. This means that anyone without manager access privileges must go through the process of reopening the bug before adding additional comments. There is often still much to discuss after the programmer fix declares the bug fixed, so this is annoying.
So Mantis is still not the ideal bug tracking tool in several places. But it remains much easier on the eyes than Bugzilla, and it's serving our fairly simple needs adequately.
2005-March-2
Henry Wasserman has released Samie 2.0, which addresses some of the concerns I mentioned in the Browser-Based Testing Survey. He says he has removed the requirement to manually call WaitForDocumentComplete after each page retrieval, and he added a PrintAllObjects routine.
Henry has also released Slingshot, which is a GUI development environment for Samie. He asks for an $11 payment before you can download it, so I initially categorized it as commercial software, but in fact the code is licensed under the GPL. Henry takes advantage of the fact that the author of free software (free as in the Free Software Foundation's idea of "freedom") is not required to distribute the software free of charge.
2005-January-20
A quick followup on the g4u disk imaging tool that I mentioned a few days ago. I used it today to back up an installation of Windows 2000 I did under VMware. I know I can simply save a copy of the VMware virtual disk, and I ended up doing that too, but I wanted something that I could easily copy to a set of CDs and possibly restore to a native filesystem on another machine.
Getting g4u running was very easy with VMware. I downloaded a 3-megabyte ISO file, attached it to VMware as a virtual CD-ROM device, and booted the virtual machine from the virtual CD-ROM. The CD image boots you into NetBSD.. Note that that doesn't mean you have to have a NetBSD system to use g4u. G4u is certainly easier than navigating a Unix command line, but not as user-friendly as Ghost. My previous knowledge of Unix filesystems helped me to figure out the disk naming conventions.
The hardest part of running g4u was setting up an ftp server. I wanted to transfer the image to the Windows XP host machine that was running VMware, but I didn't have an ftp server configured on that machine. I downloaded the Cygwin build of proftpd, but quickly saw out that it would take at least a couple of hours to figure out how to configure it. I had a working SSH daemon for Cygwin that can be used for file transfer, but g4u only supports ftp. So I booted up a nearby Linux server and was relieved to find out that ftp was already configured there.
G4u transfers the entire contents of a disk, even the parts that aren't written yet. In my case, it traversed the entire length of a 4-gigabyte disk. The saving grace is that it compresses the data, which greatly reduces the size of the empty blocks, as long as they still contain all 0's. I didn't time the process, but it was a large fraction of the total time that it took to install Windows in the first place. The disk, with about 900 megabytes of 4 gigabytes used, resulted in a 329 megabyte backup image, which will fit easily on a CD. I'm not sure if the main bottleneck was in the file transfer across a 100 megabit network, or in the cpu cycles dedicated to compression.
This tool certainly has some limitations, but it's worth further exploration. It did work on the first try, which I liked.
2005-January-7
Henry Wasserman, the author of Samie (featured in the Browser-Based Testing Survey), pointed out an interesting concern with my sample Travelocity script. He judged that running it all the way to clicking "Buy Now" posed too much of a risk that it will actually buy tickets on his behalf. I'm fairly sure that there are a few screens still to go before the transaction is committed, but I'm not certain enough to offer to pay him back for any tickets he buys by accident.
When I ran the script, I knew that I had always used a browser other than Internet Explorer when I used Travelocity. So I knew I wouldn't be logged in when the script ran IE. But maybe someone else who is an avid IE user will see different results. So this brings up the general issue of having a consistent test environment. A tool that simulates a browser probably doesn't save cookies across sessions. But a browser-based test tool will be affected by many things in the user's configuration and environment, including the cookies that keep the user logged in for a period of time.
I'd like to see the browser-based testing libraries export some cookie functions that make it easy to erase all of the cookies for a particular site. Henry says he's looking into that.
2005-January-6
Jonathan Kohl offered this feedback on the Browser-Based Testing Survey:
Note that Watir indeed has gone through its first release, and in fact also its second release. The current version at this time is 1.0.2. But because the library, unit tests, and examples are still actively under development, you might still want to get the latest source using CVS or the tarball method.
2005-January-6
Andy Hunt mentioned the g4u disk imaging tool on his blog. This is the first open source disk imaging tool that I've heard of (the commercial options I'm aware of are Norton Ghost and ImageCast).
2005-January-4
Bret Pettichord sheds some light on my confusion about COM, OLE, etc. in the Browser-Based Testing Survey:
Thanks, Bret. I'll stop trying to make much sense of it now. Bret recommends the book "Understanding ActiveX and OLE" by David Chappell as a reference.
Bret also reminded me that there's a cheat sheet for beginning Ruby users in the wtr/scripting101/doc directory in the Watir sources. The cheat sheet a nice way to get started for someone who already knows another programming language. Also, Ruby ships with the entire text of the first edition of the book Programming Ruby. I frequently referred to my printed copy of the second edition of Programming Ruby when I was writing my first Ruby script.
For those who are exploring Samie, Henry Wasserman recommends the site Picking Up Perl.
Go to the blotter for: 2003 / 2004
2005-July-25
I've been aware of the concept of mock objects for a while, but hadn't been motivated to try them out until I had the opportunity to unit test some test libraries I was writing in Perl. So again I'll subject you to my preference for Perl, but hopefully many of the concepts will apply to other languages as well. A mock object is used to replace a real object used by the code under test. You're not replacing the code you're testing, just an object that the code uses that's making the testing difficult, either code in some other part of your project that the code under test depends on, or perhaps an object provided by a standard library.
I'm now a fan of mock objects within the context of unit testing. Here's an example of how I've used them. I was taking a crufty one-off script that I wrote that automates a test for a sort of web service, and converting it to a module that could be used for more than one test. My module uses Perl's LWP::UserAgent module to emulate the behavior of a web browser. For a proper unit test of this test code, it should not be necessary to set up a real web server for the library to access, or even to have it access a network at all. A feasible way to accomplish this is to mock the LWP::UserAgent class (to be precise, we'll be mocking the class outright here, not just one object). I'm going to replace the entire class with something of my own creation, with lots of help from Perl's Test::MockObject module. Now just to be clear, I'm testing a library I'm writing that will be used in programs that test a web service. This library uses LWP::UserAgent, which is someone I'm hoping has already been well-tested, and that's what I'll be stubbing out with a mock class.
I wrote a unit test script for the library. Here are the important modules I invoke at the top of my test script:
use strict;The "strict" and "warnings" pragmas are something I use in almost all of my scripts, and here I indicate that all warnings should be fatal errors. The Test::More module is the most popular unit test framework for Perl, which incidentally is not an "xUnit" module derived from the jUnit library that has inspired unit test libraries for dozens of languages. Next you'll find that I asked for the Test::MockObject module, which will help our test but not drive it directly. And I'll need HTTP::Response to use as part of my mock class as you'll see in a moment.
use warnings FATAL => "all";
use Test::More 'no_plan';
use Test::MockObject;
use HTTP::Response;
Before loading the module under test, here's how I set up my mock class:
my $mockUserAgent = Test::MockObject->new();
$mockUserAgent->fake_module ('LWP::UserAgent');
$mockUserAgent->fake_new('LWP::UserAgent');
$mockUserAgent->set_false('request');
Then I need to see how my module handles a "not found" error from the web server. This is where I need to create an HTTP::Response object, which is what the LWP::UserAgent request method returns. So I create a response that indicates a 404 error, and then I tell the mocked request method to always return this object without ever looking at what URL it's asked to retrieve:
my $notFoundResponse = HTTP::Response->new(404);
$mockUserAgent->set_always('request', $notFoundResponse);
my $endResponse = HTTP::Response->new(200);
$endResponse->content(... XML code here indicating the end of the transaction ...);
$mockUserAgent->set_always('request', $endResponse);
$mockUserAgent->set_series('request', $okResponse, $quitResponse);
Another feature of Test::MockObject is that it records all of the calls to its mocked methods. I used this feature to verify that the data that the test module sends up to the server (or would have sent if there really were a server) is correct.
Now I should explain what kind of code works well with mock objects. First, it's far easier to unit test a module rather than a command-line program. So even if your code will only ever be part of a single command-line or GUI program, if you want to unit test it, it's best to put most of the code in a module (or library, etc., if that's what it's called in the language you're using), and then write a small wrapper on top to serve as the user interface. This has been the conventional wisdom for black box testing for ages, and it applies to unit testing even moreso. And also, it's much easier to unit test object-oriented (OO) code than non-OO code. It suffices for the purposes of mocking if you have non-OO code that uses the OO interface for the module you want to mock. My earlier frustrating experiences in stubbing out function calls and built-in commands in a massive non-OO program through the user interface are documented in my article "Diving in Test-First."
Additional reading: for more on mock objects in Perl see "A Test::MockObject Illustrated Example" by chromatic, and for an introduction using Java see "Mock Objects" by Dave Thomas and Andy Hunt.
Stay tuned for a listing of mock object libraries for various languages.
Whaddya mean, ports? We've got no ports here!
2005-June-15
I recently needed to test a feature in an application that worked something like this: the application will try to listen on network port 1000, and if that port is already in use, it will try ports 1001-1099 in sequence until it succeeds. So of course, I wondered what happens if all ports are already in use. I was happy to find that writing a Perl script that can "squat" on a given port and thus make it unavailable to other programs on the same computer was easy to do.
Here's my "portsquat" script:
#!/usr/bin/perl
use strict;
use warnings;
use IO::Socket::INET;
sub usageError {
die "usage: $0 firstPort [lastPort]\n";
}
my $startPort = shift @ARGV || &usageError;
my $lastPort = shift @ARGV || $startPort;
my @servers;
foreach my $port ($startPort..$lastPort) {
my $server = IO::Socket::INET->new(
LocalPort => $port,
Type => SOCK_STREAM);
if (! $server) {
warn "can't listen on port $port: $!\n";
}
push(@servers, $server);
}
print "done opening ports $startPort-$lastPort, sleeping\n";
sleep 10000000;
use strict;
use warnings;
use IO::Socket::INET;
sub usageError {
die "usage: $0 firstPort [lastPort]\n";
}
my $startPort = shift @ARGV || &usageError;
my $lastPort = shift @ARGV || $startPort;
my @servers;
foreach my $port ($startPort..$lastPort) {
my $server = IO::Socket::INET->new(
LocalPort => $port,
Type => SOCK_STREAM);
if (! $server) {
warn "can't listen on port $port: $!\n";
}
push(@servers, $server);
}
print "done opening ports $startPort-$lastPort, sleeping\n";
sleep 10000000;
You feed it a port on the command line, and optionally a second port to indicate a range of ports that need to be opened. A fraction of a second later, the script indicates that the ports are open. In the example above, I would run "portsquat 1000 1099". It then does nothing until you're done with your test and you kill portsquat, which immediately causes the ports to be released again. If it can't open a port, it'll just issue a warning and continue trying to open other ports in the range.
I ran the script successfully on Windows with ActiveState Perl and Cygwin Perl, as well as Mac OS X. I got a nice crash to report on one platform, and a missing error message on another. So the next time you test an application that acts as a network server, try to deprive it of its ports and see how it reacts.
If anyone wants to convert this to their scripting language of choice, I'd be glad to post those here too.
All in Good Time
2005-June-3
I sometimes do informal performance testing where I need to have a stopwatch. On a project not too long ago when I needed to test how long a device could run off of a capacitor after I removed the battery, I followed the time-honored process of googling for the first free program I could find that seemed to work. One of the first options I tried was TimeLeft, which worked well enough for what I needed.
TimeLeft
is a program for Microsoft Windows that features a stopwatch, a
countdown timer, a clock, and who knows what else I haven't taken the
time to explore. It's available in a freeware version from NesterSoft,
based in Canada, and there is also a commercial version and additional
add-ons available. I use the stopwatch feature pretty much exclusively.
Someone programmed an amazing array of chrome into it, with multiple
skins and animations to choose from. I find the start, pause, and reset
buttons to be annoyingly small and hard to hit when I'm in a hurry. I
also haven't gotten used to the fact that I have to reset the timer
every time before I start another timing, or else it'll just start
counting from where it left off before.A few days ago when I needed to follow up on a hunch about performance problems for a workload I had set up to exercise a peer to peer program, I knocked the dust off of TimeLeft. It worked fine on the system where I had it installed, but I was too lazy to install it on all the systems I was using. And I didn't have the option of using it on Mac OS. On Mac OS I experimented with the stopwatch feature in RememberMe, which was awkward because the stopwatch is bolted onto an appointment calendar that I can't get rid of, and it has a strange habit of minimizing itself to the dock after I start it.
So in some cases I resorted to using the second hand on my watch and trying to remember where it had started from. One advantage my watch had over a software stopwatch - I could initiate an action in the program under test and start timing at essentially the same time. With TimeLeft, the mouse can only do one thing at a time, so there's a small delay before I can get both the application and the stopwatch going, which makes the results likely to be off by a few seconds.
I've been using an older version of TimeLeft but just upgraded to version 3.06. It didn't take long to find a few minor bugs in the new version. But once you get used it, it does a decent job of ticking off the seconds. Almost as good as a real stopwatch, which I do need to acquire eventually.
My Monkey
2005-May-12
I've been experimenting with monkey testing through a Windows GUI lately, that is, using a test script that clicks randomly on the screen. I've been surprised at how many bugs it's found in the application I'm testing. I thought I'd write about the technique I used to develop my monkey and the free tools that made it possible.
I started with the script in the article "PerlMonkeyTest.pl - monkey testing for any Windows window." The article gives a good proof of concept for using Perl and its Win32::GuiTest module for a monkey test. But it needed a lot of work to be a robust test.
I gave the monkey some training. The sample script is a dumb monkey. It does put the target application in the foreground, but then it clicks all over the screen, often clicking outside the application. Even if the app you're testing is maximized, it still might click on the Windows Taskbar, and it might resize or minimize the application. I had my monkey check the actual screen geometry of the application, and to prevent it from getting partially obscured offscreen or preempted by the Taskbar, I keep the monkey off the title bar, the window border, and the resize widget on the lower right corner. Maybe some day I'll make it smart enough to safely handle resizing and moving the window.
I found that you can't do much in Windows when you press more than one mouse button at a time. The original monkey script did this frequently, especially when I added the ability to click the middle button, so it wasn't accomplishing much. So whenever my script decides to push a mouse button, it releases any button that's already down. It still can do drag operations, though, because it handles button down, move, and button up events separately.
I tweaked the random decisions a bit, so that the events aren't so overwhelmed by button actions as compared to mouse movements. And I increased the likelihood that a mouse movement would be less than 10 pixels away, so it's more likely to activate items on the menu, though it still only rarely finds the menu bar.
Once I had restricted the mouse movements to the parent window, I had to deal with all the dialogs that can pop up. First, I ignore dialogs that aren't modal, i.e., if I can send the focus back to parent window, I just continue testing the parent. If a child window is set to stay on top and it happens to overlap the parent window on the screen, then the monkey will click on the part that overlaps, and possibly resize it as well. If a modal dialog pops up, then I find a way to dismiss it so I can get back to testing the parent, either by looking for a button or by typing Alt-F4. In a few cases, I wait for transient dialogs to go away on their own, because otherwise I'd be canceling some interesting activity the app is doing, and sending Alt-F4 to a transient dialog could end up killing the parent. So the monkey has to have a bit of smarts about what to do with each dialog, though there are only a few different categories of actions it takes.
There are many things I could do to train the monkey further, for example, by having it send random keyboard input, and perhaps activating menu items much more frequently. Also, it needs to do better at detecting when the app exits normally because the monkey found File->Exit on the menu.
For long unattended test runs, I found it's important to take a video capture of the screen. It makes for a far more effective bug report when I can explain what the monkey did right before an error. It's very difficult to determine the state of the application just using a log file of mouse coordinates. I have yet to find a free video capture tool that works well with long test runs and doesn't use a lot of CPU power when recording, so I'm experimenting with commercial recording tools. Because all but the smartest of monkey tests are not good at detecting failures, especially minor failures, I've found a few additional bugs by watching the monkey in action in real time.
It's not easy to find a recent version of the Win32::GuiTest module, especially if you don't have the proprietary compiler required to build from source. I found recent pre-built packages for ActiveState Perl 5.8 in the Win32::GuiTest project on SourceForge.
2005-March-9
Here's an update on the Mantis bug tracking tool. I reviewed Mantis 0.18.2 in May 2004. The most recent version is 0.19.2, which the project team I'm working with now is using for their bug tracking. I have not encountered any notable reliability problems in my recent use of Mantis. Of course, I'm not actually trying to break it. :-) I haven't really needed the documentation, mostly because I'm not the one maintaining the server, so I don't have anything new to report there. We did have a bit of trouble with email notifications when we first upgraded from 0.18.x, but our server admin was able to get it working in short order. I have not tried the CSV export function recently. We're not taxing any of the more advanced features very heavily.
One of the reasons I pushed the team to upgrade Mantis to 0.19.2 was because it was difficult to manage priorities. I couldn't filter on a particular priority, and the priority column on the View Issues page was always blank. After the upgrade, I can now filter on priority and I get icons in the Priority column. I can filter on a date range for the date of the last update, but not on any of the other date fields. There's an interesting "My View" page that shows the first 10 bugs found for 6 types of common queries. The "Assigned to Me (Unresolved)" section would be useful if it also showed resolved bugs, since most of the bugs in my queue are resolved and waiting for testing.
While it is a bit more flexible, I still find the filter mechanisms a bit clunky. The filter setting is global across my account, so if I filter on resolved bugs in one window and on recently modified bugs in another window, when I update the first window it takes on the filter setting of the second one. I frequently need several different views of the database as I look for other bugs that are relevant to one I'm investigating.
I still find the basic bug reporting screen very basic, and the advanced screen too cluttered. But after reading Joel Spolsky's compelling appeal for making the bug tracking database design simple, I'm content with simply including the relevant information in the Description and Comment fields rather than insisting on a custom field for each type of information.
The default workflow doesn't make much sense in a few places, and it doesn't look like the workflow can be changed via the web-based user interface. For example, a bug is considered resolved when the programmer marks it fixed. This means that anyone without manager access privileges must go through the process of reopening the bug before adding additional comments. There is often still much to discuss after the programmer fix declares the bug fixed, so this is annoying.
So Mantis is still not the ideal bug tracking tool in several places. But it remains much easier on the eyes than Bugzilla, and it's serving our fairly simple needs adequately.
2005-March-2
Henry Wasserman has released Samie 2.0, which addresses some of the concerns I mentioned in the Browser-Based Testing Survey. He says he has removed the requirement to manually call WaitForDocumentComplete after each page retrieval, and he added a PrintAllObjects routine.
Henry has also released Slingshot, which is a GUI development environment for Samie. He asks for an $11 payment before you can download it, so I initially categorized it as commercial software, but in fact the code is licensed under the GPL. Henry takes advantage of the fact that the author of free software (free as in the Free Software Foundation's idea of "freedom") is not required to distribute the software free of charge.
2005-January-20
A quick followup on the g4u disk imaging tool that I mentioned a few days ago. I used it today to back up an installation of Windows 2000 I did under VMware. I know I can simply save a copy of the VMware virtual disk, and I ended up doing that too, but I wanted something that I could easily copy to a set of CDs and possibly restore to a native filesystem on another machine.
Getting g4u running was very easy with VMware. I downloaded a 3-megabyte ISO file, attached it to VMware as a virtual CD-ROM device, and booted the virtual machine from the virtual CD-ROM. The CD image boots you into NetBSD.. Note that that doesn't mean you have to have a NetBSD system to use g4u. G4u is certainly easier than navigating a Unix command line, but not as user-friendly as Ghost. My previous knowledge of Unix filesystems helped me to figure out the disk naming conventions.
The hardest part of running g4u was setting up an ftp server. I wanted to transfer the image to the Windows XP host machine that was running VMware, but I didn't have an ftp server configured on that machine. I downloaded the Cygwin build of proftpd, but quickly saw out that it would take at least a couple of hours to figure out how to configure it. I had a working SSH daemon for Cygwin that can be used for file transfer, but g4u only supports ftp. So I booted up a nearby Linux server and was relieved to find out that ftp was already configured there.
G4u transfers the entire contents of a disk, even the parts that aren't written yet. In my case, it traversed the entire length of a 4-gigabyte disk. The saving grace is that it compresses the data, which greatly reduces the size of the empty blocks, as long as they still contain all 0's. I didn't time the process, but it was a large fraction of the total time that it took to install Windows in the first place. The disk, with about 900 megabytes of 4 gigabytes used, resulted in a 329 megabyte backup image, which will fit easily on a CD. I'm not sure if the main bottleneck was in the file transfer across a 100 megabit network, or in the cpu cycles dedicated to compression.
This tool certainly has some limitations, but it's worth further exploration. It did work on the first try, which I liked.
2005-January-7
Henry Wasserman, the author of Samie (featured in the Browser-Based Testing Survey), pointed out an interesting concern with my sample Travelocity script. He judged that running it all the way to clicking "Buy Now" posed too much of a risk that it will actually buy tickets on his behalf. I'm fairly sure that there are a few screens still to go before the transaction is committed, but I'm not certain enough to offer to pay him back for any tickets he buys by accident.
When I ran the script, I knew that I had always used a browser other than Internet Explorer when I used Travelocity. So I knew I wouldn't be logged in when the script ran IE. But maybe someone else who is an avid IE user will see different results. So this brings up the general issue of having a consistent test environment. A tool that simulates a browser probably doesn't save cookies across sessions. But a browser-based test tool will be affected by many things in the user's configuration and environment, including the cookies that keep the user logged in for a period of time.
I'd like to see the browser-based testing libraries export some cookie functions that make it easy to erase all of the cookies for a particular site. Henry says he's looking into that.
2005-January-6
Jonathan Kohl offered this feedback on the Browser-Based Testing Survey:
One
point I noticed (which will be moot when a release comes out), is that
you mentioned someone needs to use CVS to access the Watir code base.
You can actually download a tarball from the CVS
source tree on RubyForge. If
you click the "Download tarball" link on this page, you can get Watir
that way without CVS.
Note that Watir indeed has gone through its first release, and in fact also its second release. The current version at this time is 1.0.2. But because the library, unit tests, and examples are still actively under development, you might still want to get the latest source using CVS or the tarball method.
2005-January-6
Andy Hunt mentioned the g4u disk imaging tool on his blog. This is the first open source disk imaging tool that I've heard of (the commercial options I'm aware of are Norton Ghost and ImageCast).
2005-January-4
Bret Pettichord sheds some light on my confusion about COM, OLE, etc. in the Browser-Based Testing Survey:
Microsoft
has changed the specific denotations of COM, OLE and
ActiveX so many times that most people, including me, now treat them
as near synonyms. Currently, Microsoft's preferred name for the
interface technology we use with WATIR is "Automation." Not COM
Automation or OLE Automation, just "Automation." Colloqually,
programmers seem to mostly call this technology COM, although, as you
note, the library names mostly call it OLE. In any case, .Net is
something totally different.
Thanks, Bret. I'll stop trying to make much sense of it now. Bret recommends the book "Understanding ActiveX and OLE" by David Chappell as a reference.
Bret also reminded me that there's a cheat sheet for beginning Ruby users in the wtr/scripting101/doc directory in the Watir sources. The cheat sheet a nice way to get started for someone who already knows another programming language. Also, Ruby ships with the entire text of the first edition of the book Programming Ruby. I frequently referred to my printed copy of the second edition of Programming Ruby when I was writing my first Ruby script.
For those who are exploring Samie, Henry Wasserman recommends the site Picking Up Perl.