Open Testware Reviews

FitNesse

Contents

Overview

Maturity

Project activity

Platforms

Support

Documentation

Installation

Implementation

Performance

Similar tools

Limitations

• The run.sh script has a fatal error. To work around it, change "fitnesse.FitNesse" to "fitnesse.Fitnesse" in the script.
• Creating large test pages caused an exception and no response from the browser (as mentioned earlier)
  
• I couldn't get the virtual wiki feature to work using the instructions provided.
  
• As I've seen with other Java programs, faulty input is reported with Java exceptions and stack traces rather than a more user friendly error (I was told that this is a limitation of the FIT engine). For example, a test using the column fixture that's missing the header line yields a NullPointerException, and using alpha cell data when numeric data is expected yields a NumberFormatException.
• I couldn't create a with a name that has a number on the end because it's not recognized as a wiki word.
• Tests are silent if no expected result is indicated. It might be more robust to flag this when it happens.
  
• I saw a few mysterious bumps in the night. One was clearly my fault--I copied a fit.jar file from a previous release in trying to work around a problem, which caused FitNesse to misbehave. In another case, I found that my fitnesse.jar file didn't match the fitnesse.jar file from the original zip file, and the symptom was a single page of the documentation that wouldn't load. I haven't figured out what caused that yet.
  

Observations

I thought action fixtures were pretty simple compared to row and column fixtures, probably because I was already familiar with keyword-driven testing. RowFixture was hard to get at first because of the matching aspect. I had to try it on a few cases to make sure I understood it completely. ColumnFixture wasn't really confusing but it is a bit of a switch from keyword-driven testing because each row is a discrete interaction and verification in itself.

The most confusing part about sorting out fixtures is of the subtle difference between the way fields and methods are treated, and the way objects under test are instantiated.

I think the most interesting and possibly confusing part of the different fixtures is that although the tests (tables) themselves look very similar, the fixtures for them embody quite different semantics and do very different things under the covers when they execute.

User experiences

J. B. Rainsberger, Diaspar Software Services

After seeing the power of FIT tests and having been a long-time wiki user, FitNesse seemed to be an ideal tool for managing an agile project with a mobile customer. This frequent traveler commented how convenient the tool was for him: he could check the site daily for new tests, stories and any questions we had for him. He could answer them directly, add a few tests and let me know when something failed. FitNesse did an excellent job of compensating for the fact that the customer couldn't be in the room with the rest of the team.

I have seen other projects try to implement end-to-end tests using Mercury Interactive's WinRunner and other home-grown scripting tools, mostly failing under the mounting maintenance costs. The best customer testing solution I'd used before FIT was HTMLUnit, an open source web application testing tool. Now I use FitNesse and HTMLUnit in tandem: my FitNesse tests verify application logic without worrying about the gory details of the web interface, while my HTMLUnit tests verify the web interface in isolation. Using the two together has been more effective (and time-saving) compared to writing large numbers of customer tests using HTMLUnit alone. That, and my customers don't usually speak Java.

As with any tool, there are a few problems to deal with. Creating a new wiki with FitNesse seems to require manually extracting the starter wiki document set, something I would expect it to do automatically. My one complaint with FIT is the need to use global data to communicate between fixtures. Having written articles condemning the Singleton, it is difficult to allow myself to indulge in building them. On the other hand, using FIT has given me the chance to revisit my opinions, which has helped keep them from going stale. FitNesse's strengths vastly outweigh any of its weaknesses – at least so far.

Still, as Ward's FIT site states: FIT is about tests that people can read. Do not underestimate the power of this approach. Our customers are writing their own tests! It's fantastic – and FitNesse integrates two of the most powerful additions to the Agile project developer's toolkit: FIT and wiki. I could not imagine embarking on another project without FitNesse.

Mark Windholtz, ObjectWind.com

Early on, I started using FitNesse on an Open-Source project, Ecal, that I'm organizing for the XP-Cincinnati users group. The group meets monthly to program. FitNesse is excellent for communicating stories during the gap between meetings. The latest FitNesse version and all test pages are kept in CVS with the project code. The test pages are then implemented during the meetings.

Install and set up of FitNesse is simple. So the xp-cinci group decided to use the FitNesse framework as the basis for our own web development. Take a look at the source code of FitNesse and especially FIT. The code is clean and illustrates lots of good testing techniques.

For my clients, I've added FitNesse "smoke tests" as part of the automated build cycle of legacy applications. After build and deploy, FitNesse runs to check logon and logoff. A test like this checks configuration settings and database availability. After that, more tests can be incrementally added.

While FitNesse is fine for web user interface testing, it really shines at testing domain logic. Its excellent at testing a plain Java domain layer. That's where it makes the “logic visible.”

In the past, my teams have built customized acceptance test frameworks, either from the ground-up or based on JUnit. FIT now provides a generalized approach. Tests can be written for a wide variety of technologies by adding our own fixtures.

FitNesse combines a wiki for free-form project notes and a testing framework so that the communication can be precise. FitNesse opens up testing review to a wide audience. Managers, programmers, and the customer can review the deliverables early in the cycle. Other testing tools are limited to use and review only by testers.

FitNesse tests can be written before the code, and can be used to track progress as the iteration progresses. Other testing tools are limited to testing at the end.

FitNesse has great support for iterative development. You can point FitNesse at your class files, then change the tests in one window and change your code in your IDE, hopping back and forth in little steps.

There is a still a divide between writing tests in table formats and writing new fixtures in Java code. Testers are able to write test pages, but programmers are needed to write the fixtures.

FIT and FitNesse provide a standard open-source approach to testing. It will do for acceptance testing what JUnit has done for unit testing. I expect it to become the standard approach over the next few years.

Appendix A: FooTest page

The foo function checks to see if its two floating point arguments (a and x) are equal.
|myFixtures.foo|
|a|x|foo()|
|1|1|true|
|1|0|false|
|1|1.00001|false|
|1|0.99999999999999999|true|
----
ClassPath

Appendix B: ClassPath page

!path fitnesse.jar
!path Examples

Appendix C: Examples/myFixtures/foo.java fixture

package myFixtures;
import fit.ColumnFixture;

public class foo extends ColumnFixture
{
  public double a;
  public double x;
  public boolean foo()
  {
    return (a == x);
  }
}