Note: please check your records. ASQ has announced that they are discontinuing
the asqnet.org mail forwarding service. Please replace any record you have
of my faught@asqnet.org address with faught@tejasconsulting.com.
Here are the answers to the questions that you had put up.
Thank you for letting us know that process is not all that simple as it is to spell :-)
-from Yogita Sahoo
...By the way, please realize that the results we get are always the outcome of the process we use. The process is there, whether or not it is recognized, conscious, or even intentional. The SEI has done some good in creating awareness of the need to pay attention to how we do things ('process"); but they've also in my opinion done harm in trying to usurp the term "process" to mean their "Process." In order to improve one's results, one must change one's process In order to change it, one first must recognize what it REALLY is. In many instances, what people presume to be their process is not what their REAL process is.
The SEI starts with a presumption of what everyone's process is and then directs them to impose the Process the SEI prescribes. Process improvement is not the same as process imposition. Also, it doesn't mean that one has to get burdened with busywork. Unfortunately, for every advocate the SEI wins, they probably create 10 people who resist because they think "process" automatically means busywork. Since the SEI has fostered that image intentionally, along with the notion that theirs is the only acceptable view, their net overall benefit may be negative, or at least far less than it could be. That's their REAL process.
-from Robin Goldsmith
...I view performance testing as a subset of behavioral testing, and often but not necessarily related to non-functional requirements (I often have a section called performance requirements). I would finish with asking/echo, if not performance, then what?Another data point that seems just as viable comes from the book that I reviewed in this month's feature, which suggests that "load testing" is an umbrella term that includes performance testing. Also, since I also have Software System Testing and Quality Assurance handy, I'll note that Boris Beizer used the term "load" in various ways as relevant to both stress testing and performance testing, though he didn't use the term "load testing." I think the jury is still out on this one.-from Jon Hagar
Keep your feedback coming! You can contact me at
faught@tejasconsulting.com.
For those of you in the north Texas area, take note that I will be speaking at the July 12 meeting of the Dallas/Forth Unix Users Group on the topic "Event-Driven Scripting." I've developed a handful of test harness scripts using an event-driven technique that allows you to take advantage of parallelism and make testing more efficient. It should be a fun talk. Also for local folks, note that I've put up a new web page, "Software Resources for North Texas," where I list organizations and other local resources that are useful for software people.
I will also be giving a track presentation at the Fall
Software Test Automation Conference, August 29-30, on the topic "Scripts
on My Tool Belt," where I'll take a tour of some of the scripting languages
I've used and how they've helped the testing process, sometimes even when
the tests aren't automated.
Book Review
The book Load Testing for eConfidence is an interesting little freebie
from Segue. It's self-published, with no ISBN, and weighs in at 142 pages.
My copy was printed October 2000, and the copyright is 1992-2000. The author,
Stefan Asböck, is mentioned on the title page but not on the cover,
which hints that the book was written as a marketing tool. You can order
a copy of Load Testing for eConfidence from Segue at no charge if
you're willing to hand over your contact information to Segue's sales force.
I'm already on just about every tool vendor mailing list there is, so I
figured it wouldn't hurt to request a copy. What I got was a nice surprise.
Load Testing for eConfidence
Before I go on, let me mention that Asböck's definition of load testing, which I'll adhere to here, is that load testing "should measure the stability, responsiveness, and throughput of the application." (p. 129) He emphasizes that load testing is conducted using a realistic workload, but he also considers stress testing to be a type of load testing.
Books that have substantial coverage of load testing are hard to find. Several books have a few paragraphs on the topic, but few devote even as much as a chapter to it. The earliest one that I'm aware of is Boris Beizer's Software System Testing and Quality Assurance, published in 1984. Chapter 8, "Background, Stress, and Performance Testing," gives fairly in-depth coverage of the state of the art in 1984. The book is out of print, but sometimes available at used book stores. (My favorite source for used books is abebooks.com, which has one copy available as I write this). Hung Nguyen's Testing Applications on the Web includes a short chapter on "Performance, Load, and Stress Tests." Steve Splaine and Stefan Jaskiel's The Web Testing Handbook has 108 pages that are relevant, divided into the chapters on "Performance," "Scalability," and "Reliability and Availability."
Load Testing for eConfidence starts by covering some networking and World Wide Web basics. This might be useful for managers who are trying to gain some understanding of load testing, but test engineers who will be implementing load testing should already have a thorough knowledge of these topics.
One thing that I like about the book is that it gives some low-level details about planning and conducting load tests. The author discusses several different workload distributions, and different ways to capture web traffic. He gives information on planning the network resources for the load test, and gives heuristics to help determine the bandwidth required to ensure that a network bottleneck doesn't affect the load test results. The book discusses the many different types of hardware resources on the test driver machines that should be analyzed to make sure that the driver machine doesn't become a bottleneck that alters the load test results. Again, he gives some very useful heuristics, though there's no data to back up his suggestions on what level of resources is sufficient.
There's a nice chapter on the details of emulating a web browser. The chapter on "Administering Load Tests" gives a whirlwind tour of load test design, monitoring, and results evaluation. The chapter gives some interesting information, but leaves me wishing for more thorough coverage. There's a brief chapter on "Load Testing Benefits and Drawbacks," and then a case study using Segue's SilkPerformer. There are some details in the case study that may not make sense to someone who isn't familiar with SilkPerformer. The book wraps up with an afterword from Jenny Jones, eSegue Director, plus a glossary and an index. The bibliography includes some sophisticated references on performance modeling and testing.
I was impressed that the contents of the book don't come across as an advertisement for Segue. Segue and their tools aren't mentioned until the case study near the end of the book. The text most likely reflects Segue's biases, for example, where it discusses browser emulation without mentioning that some tools utilize real web browsers instead of a browser emulation. Much of the information is relevant to environments that use a different vendor's load testing tool, though it's hard to tell which of the heuristic hardware guidelines are specific to SilkPerformer. Given its length, I can't really say that this is the first book-length work that's dedicated to load testing, but I do feel like the book serves a useful purpose.
There isn't enough overlap between this book and the chapters on load testing in The Web Testing Handbook to consider it a cheap replacement for them. One of the most interesting points from the Handbook that you won't find covered in Load Testing for eConfidence is the concept of web site abandonment. Splaine and Jaskiel suggest designing web site abandonment into your load tests in order to accurately simulate real-world conditions. This means that as the response time during the test increases, the chance of the simulated user abandoning the web site (i.e., aborting the test) should increase. This changes the results of your measurements - actually allowing you to initiate more transactions because some users are now inflicting a lower load on the system when they give up, but it introduces an additional metric to track. Another nice thing the Handbook includes is a large number of references to tools and other resources.
You can order the book Load Testing for eConfidence from Segue
Software's web page at http://www.segue.com/.
The link is currently under "econfidence guides" on the home page, though
it may move around as time goes on. Also available is Gain eConfidence:
The e-Business, Reliability Guide, published by Segue in 1999.
#####
| Back to the newsletter index | Back to the home page |