Tejas Software Consulting Newsletter

v2 #6, December 2002/January 2003

Software is always the secret sauce.
This is from Todd Bradley, CEO of Palm, Inc., during a recent visit to Fort Worth. He was talking about whether it's the hardware or the software that sells systems, and as a software person, I really liked the answer. :-) And a counterpoint from a completely different point of view: "May your experiences with new software be dull, dull, dull." This is according to Boris Beizer in the little-known book The Frozen Keyboard, and you'll learn more about what that quote is all about in this issue's feature article.

Welcome to the outlet for my uncontrollable need to write. My audience is mostly software quality professionals, but hopefully there's something interesting for everyone else too. I've been eagerly waiting for this issue ever since I realized that I started my bi-monthly publication schedule out of phase compared to how most people would do it. I love the dissonance of a December/January issue!

If you're not yet a subscriber, browse on over to http://tejasconsulting.com/newsletter/ for a complimentary electronic subscription. Redistribution of this newsletter is encouraged, provided you forward it in its original format and in its entirety.

-Danny R. Faught
faught@tejasconsulting.com
http://tejasconsulting.com/
817-294-3998

Contents

Tejas Newswire

Tejas Software Consulting was a Bronze Brush Sponsor of the 2002 Cowtown Brushup, which facilitates the painting and minor repair of the homes of elderly, disabled and low-income citizens in Fort Worth. The T-shirts for the event mark the first time my logo has appeared in cotton. Click here for a few pictures from the event. By the way, my company also provides annual support to the Free Software Foundation and the Mary Kay Ash Charitable Foundation. How's that for balance?

I've been making progress on my free testing tools compilation CD, which I'm now calling the Open Testware Collection. I'm hoping to offer a beta version for sale at a big discount in the first quarter of 2003.

I have posted my notes on Being a Consultant to answer the many people who ask me "How do I get into consulting?" Also, I added a Job Hunting section to my "Software Resources for North Texas" page.

I've updated my resume so it's more timely and more appropriate for consulting work. Let me know if you know of an organization that needs help with its software quality.

Feedback on the October/November issue

See the newsletter archive at http://tejasconsulting.com/newsletter/ to figure out what we're talking about.

Jerry Weinberg commented on last issue's book review:

Thank you for that very generous review.

And speaking of bugs, I slapped myself on the head when you pointed out that I had no reference to any of Virginia's books in More Secrets.  I was sure I'd referenced them all.  <aaargh>

Jerry recommends Virginia Satir's book Making Contact, published in 1976, for the lay reader who doesn't have a family therapy background. It's out of print but used copies are readily available.

Robert Coutré commented on Satir's Communication Wheel:

Would a "super-reasonable" person/organization be like the Houyhnhnms in Swift's "Gulliver's Travels"? The Houyhnhnms, who looked like horses, demonstrated how horrible it would be if mankind really was a purely rational animal. The other race of creatures in the story were the extreme of dumb-emotional, the Yahoos. It's been 20 years since I read it, so that's about all I remember.

I have to admit that I haven't read that one yet - it sounds like I should!

Mark Wiley continues the thread on certification:

I'd be surprised if I could get certified. I've read some of the books, but I didn't memorize them. I have yet to find something in a testing book that was truly profound, but I will admit that they seemed to be basically correct, at a simplistic level. So much of the talk seems to be about definitions and terminology, and while I recognize the value of them, many of the concepts they attempt to define are much more complex then the definitions would suggest.

I think Alan Richardson hit it right on the head, especially this line, "And importantly - design software, program software and test, Test, TEST software."

My favorite books are the ones that talk more about implementation details than process. They're rare.

Andrew Gardner followed up on his question about general testing books:
I now have the book you recommended. One more thing; what would you recommend as a good first book on testing Internet/Web software? I'm sure that a lot of people will also be interested in that answer.

I'm glad you asked your questions in the order that you did. You need to first understand the basics of software testing before tackling a book about web testing. All of the fundamentals still apply to web testing, and the web testing books do not thoroughly address the basics, nor should they. With that said, there are some books that do a good job of explaining how to tackle web testing specifically.

All of these are reviewed on StickyMinds.com. Oh, and see "Load Testing for eConfidence" reviewed at http://tejasconsulting.com/newsletter/2001June.html for one more option.

Feature Article
Yes Virginia, We Do Want Our Software to Work

Egad! I spent part of my work day today trying to get the Mac OS X Mail application to work After I coaxed it into to downloading email over a wireless link, I encountered a dismally poor user interface for the StuffIt program when I tried unsuccessfully to extract a zip file attachment. I gave up and read the attachment on a Windows machine. When I tried to email the individual files by dragging them from WinZip to Mozilla, Mozilla failed with a mysterious error when it tried to send the attachments.

This sort of experience is part of a typical day for millions of computer users, novice and expert alike. Software professionals are generally well-equipped to isolate and work around bugs, not just for the software they develop, but also for the software that they use. We don't like to have to do it - we have enough hassles from the software we're developing, and like anyone else, we don't have time to deal with crappy software. But at least we have a fighting chance in getting it to work.

What do we expect the average computer user to do when they get frustrated by bugs? Several authors have tried to help them by publishing books on the subject targeted to a broad audience.

An early book that I stumbled across is The Frozen Keyboard: Living With Bad Software, published in 1986 and written by Boris Beizer. I'm not really sure what the average computer enthusiast was like in 1986 -- I was still playing with my Commodore 64 then. The book is more technical than the newer ones in the same genre. But in between the details about user interface design, I found some real gems, like the section on "Smelling an Impending Crash." There's a section on how not to break software, basically the polar opposite of what you'll read in James Whittaker's book that I reviewed in the article below. There's some great terminology that should have caught on more widely - see the section on "Shriekers, Loopers, Freezers, and Babblers." And I was delighted to find a section on "The Audio Monitor," which used to be a standard part of computer hardware, but nowadays we have to make do with a cheap AM radio. The wildly informal and irreverent writing style (not to mention the Shoe cartoons) kept me reading until the end, despite the fact that a large fraction of the book is outmoded.

A more recent book about software consumer advocacy is Bad Software: What to Do When Software Fails, by Cem Kaner and David Pels, published in 1998. The book explains to average computer users how to talk to technical support people, and if that doesn't resolve the issue, how to contact consumer protections agencies or even go to court. There's an appendix about the ill-conceived UCITA act, which is threatening again to infect my state in the coming year.

One more book I'll highlight is The Software Conspiracy: Why Software Companies Put Out Faulty Products, How They Can Hurt You, and What You Can Do About It, published in 1999 and written by Mark Minasi. This is a nicely-written book (except for some deja vu-inspiring repetition) that follows the same general tack as Bad Software, but with more on why software companies do what they do and less on the legal aspects. The basic theme is that software companies generally believe that consumers want them to focus on building more features rather than making software that works.

So how has the public responded to these resources? It seems that the predominant response has been to ignore them. The online booksellers show that people who bought these books also bought technical books, which implies that it's mostly technical people who bought them. I use the past tense there, because of the three books, only the last one I mentioned is still in print, and it seems to have been relegated to an electronic-only edition (all three books are readily available from used book suppliers, though). So it seems that the public isn't ready to be rallied into a movement to demand better quality and better customer service from software vendors. I'll have to admit, if I had trouble with my lawnmower, I probably wouldn't read a customer advocacy book about engine manufacturers, especially if the media were giving rosy reports about how well the newer lawnmowers work.

If software vendors are going to get the message about the customer demand for quality products (assuming we really do want working software more than we want new features), we'll have to have more software quality champions out there, spreading the message to a broad audience that software should work as well as lawnmowers and other consumer products. Here's a quick summary of the possible actions to take when encountering bad software. You can use it as a field guide.
  1. Do your homework.
    We have to isolate the problems we find with our software before we can do much to fix them. We should check the available documentation, the knowledge base on the vendor's web site, and independent sources of help on the Internet to verify that the software is misbehaving.

  2. Work with the vendor.
    If you can't resolve the problem with a reasonable amount of effort, you should contact the vendor's technical support department. Even if you do find a workaround, if it isn't documented, you still probably have a bug that needs to be reported to the vendor. You would be justified in asking for a full refund. However, you probably have a lot of time and perhaps data files invested in the software, so returning the product might be very disruptive to your work. So you would also be justified in asking for a patch to the program or its documentation to resolve the issue. You shouldn't have to buy a newer version of the program. Even a free update to a newer version might not be acceptable if it requires additional hardware resources that you don't have.

    Dealing with technical support people can be a challenge. Bad Software has some good advice to help you negotiate with them. And even before you have problems, you can let the vendor know that you want them to invest more in software quality and less in new features in order to win your loyalty.

  3. Work with the industry.
    If you don't get what you want from the vendor, there are a variety of actions that you can take. You can report the bug to an independent service like BugNet.com. You can contact the Better Business Bureau. They won't be surprised - complaints about computer-related companies are one of their largest categories of complaints. You can write up your experiences in the trade press, and lobby the magazines or alternative venues to provide more balanced reviews of software products. And to the best of your ability, be sure to consider software quality when you make purchase decisions, so we stop sending the message that we want to buy software that doesn't work.

  4. Work with the legal system.
    Few people will go as far as suing the software vendor (that's why we have so few legal precedents). But you can definitely make your voice heard with respect to the laws that affect software transactions. The most pressing movement that needs more voices in each state is the UCITA act that has been sent to state legislatures. See AFFECT web site for details about opposing this law that is a major setback for software consumer advocacy..
I looked at BugNet.com, by the way, at the recommendation of the consumer advocacy books. I tried to sign up for a trial subscription (the only way for new users to sign up), and the web form had several required demographics fields that requested details about my personal income and my organization that I wasn't comfortable sharing. There was no way to sign up without providing this information. So I decided to take my own advice, and I took an extra minute to tell BugNet about the feature that is preventing me from using their service.

Additional references - my earlier attempt to popularize the need for software quality in two pages or less,  "Position Statement - 'Technology and Your Business'" (pdf); my layman's introduction to UCITA - "Consumer protection efforts threatened by UCITA", and also "My observations on UCITA in Texas"; Cem Kaner's Bad Software web site.

Bonus Feature
Mini-Review - How to Break Software

A brief look at How to Break Software: A Practical Guide to Testing, James A. Whittaker, Addison-Wesley, 2002, ISBN 0-21-79619-8, 178 pages.

As a track chair at the STAR East conference earlier this year, I had the great pleasure of introducing James Whittaker before his "How to Break Software" presentation. Having seen him speak a few times before, I set up the audience to expect greatness, and James exceeded those expectations. He's a fabulous speaker -- he actually demonstrates tests that find bugs during his talk. He explained that the talk was a condensed version of his recently published book, How to Break Software. So I bought the book, and I thought I'd give you a quick report on it.

I've loved James' approach to testing since the first time I met him. The core of the book is organized into different "attacks."  Like the title suggests, his goal is to help you break software, and with his help, you'll be launching a relentless series of attacks that almost always flush out scores of bugs from any new software product. The attacks in the book are based on a "fault model" that is explained in the first chapter. He presents each of the attacks separately using a style borrowed from the patterns community, which I found to be a very useful way to capture the techniques. Note that the book seems to target beginning testers, but there are several casual references to testing techniques that new testers might not understand.

One big thing that seems to be missing from this approach is some assurance of getting good functional coverage of the product under test. And while the error-guessing approach does incorporate some amount of risk management, it isn't really tuned to finding the most important bugs first. As the first bug I file on a new product, I prefer to uncover a problem with a mainstream feature, even if minor, rather than a spectacular crash in a rare corner case. I'll admit that I also have a strong impulse to go for the fireworks, and I'll admit that I often give in to that impulse, but it's important to teach testers (and remind ourselves) what's most important.

The book describes two tools - Canned HEAT and Holodeck Lite. They are included (binaries only) with the book on a CD-ROM; they seem to be the same versions that are freely available on the web. I was disappointed with how well the tools worked - Canned HEAT has some odd user interface bugs, and most of Holodeck's features are disabled in the version that comes with the book.  However, I was able to get Canned HEAT working well enough to artificially inject faults into a real-world product that I'm testing, and it found several latent bugs. Moreover, it inspired me to recommend that my client implement a more sophisticated fault injection system that would be much more effective at finding robustness problems than any amount of black box stress testing.

The techniques in the book are not sufficient for a complete test of a system's functionality (few if any recent books have accomplished that), and I think this book doesn't go far enough to explain that fact. Reading the book is not nearly as exciting as listening to James give a live presentation, but the robustness testing patterns that he presents are a useful addition to the testing literature. The book provides a useful service by helping to pull the idea of fault injection out of the dusty realm of academia and into real-world usage.

Since this review was written, James has also co-authored How to Break Softwawre Security and How to Break Web Software. See also my review of Holodeck.

Copyright 2002, Danny R. Faught
#####


  Back to the newsletter index    Back to the home page