Open Testware Reviews

Browser-Based Testing Survey

Reviewed: 2004-December-31
Testingfaqs.org category: Test Implementation Tools, Unit Test Tools

There are a number of web testing tools that work by simulating the core functionality of a web browser, such as Canoo WebTest and HttpUnit. But another approach is to simply use an existing browser rather building a new one, and this browser-based testing concept is what I'd like to cover in this survey.

We'll cover eight different tools that can drive Internet Explorer through an API. We'll also see example scripts for two of them.

About Internet Explorer automation

Of course, you could do web testing using traditional GUI test automation, and some testers actually do this. But automated GUI tests have well-known maintenance problems. A better way to test is through an application programming interface (API). Internet Explorer (IE) has a programming interface that we can use for this purpose, and there are several open source tools that take advantage of it. One nice thing about this type of automation is that it doesn't take over your mouse and keyboard, so it's immune to some types of spurious failures that happen when the tester bumps the mouse at the wrong moment when running a GUI-based test.

I tried to put together a concise summary of the Microsoft Windows technologies that come into play that facilitate the IE API, but it just seems to be turtles, I mean acronyms, all the way down. You don't necessarily have to understand the underlying details, except to gain an appreciation of what the tools are doing for you and maybe to debug them when you have a problem. Here's what I can confidently report. The foundation is Windows OLE (object linking and embedding). While I thought that OLE had been superseded by the likes of COM, ActiveX, and .NET, I still see many OLE references in the tools I looked at. There are libraries such as Ruby's win32ole, Python's win32com, and Perl's win32::OLE that gives us access to the API that we need.

Specific to IE, you'll also run across the DOM (Document Object Model), which is a standard data representation for the contents of a web page, which is what the libraries in this survey use to map our test script's references to web page objects into something they can automate. You can often get direct access to the DOM for a page through these libraries.

We could write scripts that directly call the low-level API for IE. The testing libraries save us from the tedium that would be required to do that, however. These libraries have been written in several different languages The libraries are different enough that a good deal of learning is required to learn another one after you've become proficient with one of them, even if you know both languages well.

Sample scripts

I decided to try out a few of the libraries. Here are examples that implement the same test, first with Samie in Perl, then with Watir in Ruby. I started out writing the scripts as part of a keyword framework, but I found that it was much easier to proceed with trial and error to strip them down to the bare necessities with custom-programmed test scripts. Once you understand how to use these libraries, I recommend using them to implement keywords using a keyword-driven automation approach. Also note that I didn't program any verification into the scripts. Proper test cases would verify that the important elements of each page were present in addition to showing that the test can proceed to the next page in the sequence.

Of course, if you try these scripts a few months from now, there's a good chance that Travelocity will have changed enough to break the scripts.

Here is a script that automates the first part of a Travelocity transaction using Samie 1.2a and ActiveState Perl 5.8.4. Make sure that you are not logged in to Travelocity when you run the script, or you may run the risk of initiating a real transaction.

#!c:/Perl/bin/perl

use strict;
use warnings;
use Win32::SAM;

# open a browser window and go to Travelocity
StartIE();
Navigate("http://www.travelocity.com/");
WaitForDocumentComplete();

# ask for flights from Dallas to Portland, using the default dates and number of travelers
SetEditBox("FO_from", "DFW");
SetEditBox("FO_to", "PDX");
ClickFormButton("submitFO");

# Wait for the "We are searching for your flights" screen to go away
while (! isTextPresent("Select Flight for")) {
  sleep 1;
}
WaitForDocumentComplete();

# select the first outgoing and return flights offered, continue without adding a hotel
ClickFormButtonByLabel("Continue");
WaitForDocumentComplete();
ClickFormButtonByLabel("Select");
WaitForDocumentComplete();
ClickLink("Continue without Hotel");
WaitForDocumentComplete();

# indicate that we understand the fare rules and proceed to the login screen
ClickCheckBox("is_non_refund");
ClickFormImage("Buy Now");  # use at your own risk!
WaitForDocumentComplete();

This script below does the same thing using a version of Watir that I pulled from the cvs server in mid-December 2004 and Ruby version 1.8.2-14RC10. These is no packaged release for Watir yet, so you'll need to use cvs to pull the code from Rubyforge. Also, I had problems with segmentation faults in the latest stable release of Ruby, so I used a recent beta version instead.

#!c:/ruby-1.8.2-14RC10/bin/ruby -w

require 'watir'  # assumes watir.rb is in your current directory

# open a browser window and go to Travelocity
$ie = IE.new
$ie.goto('http://www.travelocity.com/')

# ask for flights from Dallas to Portland, using the default dates and number of travelers
# Watir automatically waits for the "We are searching for your flights" screen to go away
$ie.textField(:id, 'FO_from').set('DFW')
$ie.textField(:id, 'FO_to').set('PDX')
$ie.button(:name, 'submitFO').click

# select the first outgoing and return flights offered, continue without adding a hotel
$ie.button(:caption, 'Continue').click
$ie.button(:caption, 'Select').click
$ie.link(:text, 'Continue without Hotel').click

# indicate that we understand the fare rules and proceed to the login screen
$ie.checkBox(:name, 'is_non_refund').set
$ie.button(:alt, 'Buy Now').click  # use at your own risk!

The most noticeable difference between the two is that Watir does more of the synchronization automatically, waiting for IE to complete an operation before allowing the script to continue. I had to do this manually with Samie, which I learned the hard way when I got OLE errors after leaving out a WaitForDocumentComplete() call. Note that when Travelocity is conducting the search, the user sees a screen asking them to wait. The spinner on the browser is not moving during this time. Samie continued as soon as the spinner stopped, so I had to add code to check for text that is on the next page that automatically appears. Watir somehow knew that more was coming, so I didn't need any extra code to handle this. However, if I wanted to verify any of the contents of that intermediate page, it may be difficult to do with Watir.

One place where Samie did better than Watir was in the last line of the script. Watir did not support identifying an image button using its alt tag. I mentioned this on the Watir mailing list, and the next day Paul Rogers added code to handle this. Besides having to use a beta version of Ruby, I frequently got a bizarre "No such file to load -- ubygems" error from Ruby. I traced this to a $RUBYOPT environment variable that I had to delete in my Cygwin shell to fix the problem.

I should note that Henry Wasserman, the author of Samie, was very gracious in offering to fix any problems I found, but I couldn't convince him that it was a bad thing when I got internal errors when I didn't call WaitForDocumentComplete() every place I needed to.

Overall, I preferred Watir over Samie for a number of reasons. Most importantly, I could not get the Samie script to run reliably. While sometimes it works fine, other times it would simply hang in the middle. And several times I hit a fatal error - "Can't call method 'getAttribute' on an undefined value at c:/Perl/site/lib/Win32/SAM.pm line 1115." Samie is poorly documented, while there is a decent user manual for Watir which you can find in the doc directory in the cvs sources. I didn't find a way to print out a full DOM representation of a web page with Samie. When I tried to use Data::Dumper to print it in the Perl debugger, the debugger hung. So I ended up using Watir's showAllObjects method to help me write the Perl script.

I had a hard time figuring out which property to use to identify an object using both libraries. The names of the properties only loosely correlate with the tags that you see in the HTML source. Also, it can take some experimenting to figure out whether a button is part of a form or is just a simple image hyperlink, especially when the button is implemented using JavaScript.

For those of you who are chomping at the bit for me to finally admit that Ruby is better than Perl now that I've written my first Ruby script, I'll have to disappoint you. I found Ruby's design largely similar to Perl's. Perhaps if I have an opportunity to do more extensive work with Ruby I can draw a more informed conclusion.

Reference

Brian Marick wrote about using Microsoft Word's COM interface in "Bypassing the GUI," STQE magazine, September/October 2002.

About the matrix

Thanks to the WTR project for helping me round out my list. All of these are also now listed on their Wiki, and there tends to be good discussion of a broad range of tools on their mailing list.

The available tools focus on IE, though Selenium claims to work with several different browsers. Perhaps tools implemented in JavaScript are more likely to be portable across browsers. There is a small bit of buzz about exposing the API's of additional browsers like Mozilla to the various scripting languages, but there doesn't seem to be much progress in that direction.

At least a few of these projects have reported problems on Windows XP Service Pack 2, which changes IE's security model so that the tests can't do things they used to be able to do.

Tool	Test Case Language	Notes
Avignon	XML	Acceptance testing framework. Includes a new "Internet Explorer bridge" which likely plugs into the IE API. Built on top of JUnit and fixtures are written in Java. Seems to focus on HTTPUnit, which is not a browser-based technology. Some background in the Acceptance Testing HTML white paper.
IeUnit	JavaScript	xUnit test framework designed for general web testing. Includes a GUI test runner front end. No mailing list. First release October 2002.
Pamie	Python	Written by Rob Marchetti. Patterned after Samie. The author has an alpha version for VB as well. Active mailing list since July 2004.
Jiffie	Java	Used internally at Tapster Rock in the UK. Active mailing list.
PyJTF	Python	Unit test tool for testing JavaScript code running in IE. The tests are written in Python, not JavaScript. Not sure whether it's useful for general web testing. Project has been dormant for a year. There are three other JavaScript unit test tools listed in the Unit Test Tool Survey.
Samie	Perl	Written by Henry Wasserman. Also uses Win32::GUITEST. Mailing list is largely inactive, but the author is reachable. First release was February 2003.
Selenium	Table-based	Implemented in JavaScript. Works in several different browsers on Windows and Linux. Uses a FIT-like table interface and an object-based language reminiscent of Worksoft Certify and the latest release of Mercury QuickTest Pro, though it's not a complete keyword-driven framework. Requires extra work to avoid cross-site scripting security protections. Sponsored by ThoughtWorks.
WATIR	Ruby	Successor to the cLabs Internet Explorer Controller (which is confusing referred to as ClIEController, IeController, cliec, iec, or cIE). No official release yet - use cvs. Part of the Web Testing with Ruby (WTR) project. Active mailing list. Contributions from Paul Rogers, Bret Pettichord, Chris Morris, and others.