Finding a better failure
When I teach my introductory testing class, I stress the importance of not stopping at the first symptom of a bug that you see. Testers should work to find the worst consequence of a bug that they can.
While playing online games with my kids this weekend, I found a good example. A game on Webkinz.com called Quizzy’s Blast is available for a one-hour period only a few times a week. When we play the game, we aim darts at a dartboard to determine the subject of a quiz question and the bonus points. We get 5 darts, and the maximum score is 50 points. After playing 5 darts, we have to wait until the game is available on a later date before we can play again.
This time, though, when we started the game, it appeared to be frozen. I clicked the “X” icon to quit the game, and got a dialog explaining that I hadn’t used all of my darts, and that I wouldn’t get another chance if I quit. If I were a tester on the project, I could have stopped here and reported a bug, or at least a production problem. But I smelled that there was something bigger. I clicked the button to return to the game, and it looked like it was working now. So we played the game, and after a few minutes we realized that the “Dartz Left” counter was stuck at 5, never decrementing when we used a dart. We played dozens of darts, racking up hundreds of points. Here’s a snapshot of the dartboard that’s never supposed to have more than 5 darts:
So what could have been reported as an inability to play the game (admittedly, a serious bug) is now an issue with the integrity of game, possibly granting far more points that it should.
What would you do next if you were testing this? The first thing that I thought of was taking the points up to 4 digits, to see if there might be a catastrophic failure. Alas, family obligations intervened and my experiment came to an end. But earlier I did trip across a digit overflow issue in another Webkinz game, Operation Gumball, a Mastermind clone. I got a 6-digit score in a tournament, and accidentally discovered that the score board couldn’t display all 6 digits:
I’d love to get a chance to explore what kind of testability features the Webkinz developers have built in to the code to facilitate testing this sort of thing without needing to master the game. Especially in this case, where a score of “111111″ in a proportional font might have actually fit, where “100000″ probably wouldn’t have.