If you find yourself forwarding the newsletter to other people who are benefiting from it, encourage them to subscribe to be sure they get every issue.
If one looks at the standard pap that circulates for (generally high-priced) "wisdom" on risk, you'll find that what people claim are risks are really mainly incompetent software development practices. The top pick on practically everyone's list is "poorly defined requirements." First, it's not a risk, it's a certainty. Second, it's not a risk, it's merely ineptitude. The only way to break the cycle so we actually correct the problems is to stop treating them as risk and start treating them as our job, learning to do them well before we're willing to accept these high salaries we keep thinking we're entitled to for work as an industry that would be malpractice in many other fields.If you already have poor requirements, then I certainly agree that it's not a risk, but a problem that needs to be dealt with. Risks are things that might or might not cause a problem. If you don't have requirements yet, then "poorly defined requirements" is still not a risk, but a cause. You might formulate it like this: "The organization does not have the skills to create well-defined requirements [cause]. Poor requirements could cause us to create a product that doesn't meet customer needs [risk - probably with a high likelihood!], which would reduce our market penetration dramatically [result]."
My thanks to Robin for taking the time to share his thoughts. Your feedback on the items in this issue of the newsletter is highly encouraged.
One more thing to mention regarding risk management is that the March 2001 issue of the Risk Management Newsletter is now available. This is the newsletter of the Project Management Institute's RiskSIG, and I am the editor. This issue highlights the Symposium on Risk 2001, a free symposium coming up May 9-10 in Hampton, Virginia.
You can subscribe using the form on the home page at testingfaqs.org. This announce-only mailing list is separate from the Tejas Software Consulting News mailing list.
I have also sent an update on recently added items on StickyMinds to
the testing-paper-announce mailing list. StickyMinds will soon be launching
its own mailing list to send out information on newly added items. In the
meantime, you may wish to subscribe to testing-paper-announce at Yahoo
Groups to get pointers to new papers from StickyMinds and also directly
from several authors who write about testing.
Software Quality Notes
One overhead made it all so clear. I was attending Steve Splaine's "Web
Site/Application System Testing" tutorial at the Spring 2001 Software Test
Automation Conference. Right there on slide 22, he flashed up a wonderful
solution to my years of struggling with terms related to performance testing.
Performance Testing Terms - the Big Picture
Anyone who works in the software quality field figures out pretty quickly that the jargon flies fast and free, and everyone has their own unique definitions for these words. What makes it worse is that people don't realize that their definitions don't match, and huge miscommunications are often the result. When I wrote a paper on reliability testing a few years ago, I tried to find the standard terminology to use for the things I was describing. I couldn't find any consensus on this, so I just defined the terms that I chose and went from there.
Below is my synopsis of the terms and definitions the Steve introduced on that overhead. Steve pointed out to me that the distinguishing factor is the objective of the tests.
Load testing - Attempts to model the anticipated real world over a short period of time, with the expected number of users and average user interaction delay times. This is looking for the typical user experience.Stress & hot spot testing - For a short period of time, the site is hit with larger than expected loads, requiring extensive computations/data retrieval. Here you're looking for how the system breaks down under stress. A variation is "hot spot testing," where you focus the stress on a specific portion of the product, looking for a weak link.
Spike & bounce testing - Testing with a sudden growth in load over a very short period of time, looking to see if the system can respond to abrupt changes in the workload . A variation is to follow the spike with a bounce down to a very low load level, and then continue repeating the up and down pattern. This tests whether the system can recycle its resources properly.
Integrity testing - Combines functional testing with stress testing to ensure that functionality that worked under low volumes still works.
Endurance testing - A load or stress test that is run for an extended period of time, typically several days, with the purpose of detecting slow-to-appear defects. This is measuring the reliability of the system.
I've seen authors mention a few of these terms, and several other terms as well, but I've never seen such a complete picture as this. Some of the terms are new to me, and I'm glad to have words for the concepts now. For example, I've developed "hot spot" tests before, but I couldn't figure out what to call them until now. I've instinctively built spikes and bounces into system test scenarios, and I wish I had put more conscious thought into it - I would have paid as much attention to the bounces as I did the spikes.
The inevitable controversy with definitions can't be avoided, though. For example, many people associate load testing with a very heavy load, rather than a load that is representative of average usage. Integrity testing sounds a lot like testing with a background load, and many people will have other preferences for alternatives for some of these terms.
Did you catch what may be the largest controversy of all? I talked with Steve about whether all of these terms are really related to "performance testing." He indicated that this might not be the right word, but none of the alternatives we considered were any better. In fact, some of the important performance testing concepts that I'm familiar with, including single-user time-to-solution performance measurement, aren't on the list at all. Rational's Load Testing FAQ lists performance testing at the same scope as load testing. Is "performance testing" the right wrapper for these terms? Let me know what you think.
Because this tutorial was an abbreviated version of the training class that Steve's material came from, I checked his book, The Web Testing Handbook, for additional information. I found the term "smoke testing," and it's very interesting to see it applied in the context of performance testing - checking some basic performance characteristics before committing to an elaborate performance test. Steve had mentioned this during the training, but it wasn't included on the definitions slide. Upon further skimming, I found several other terms discussed, and several discussed in a different context than in the training. So I'll leave further analysis until after I've had time to properly absorb the book.
The Web Testing Handbook, Steve Splaine and Stefan P. Jaskiel, STQE Publishing, 2001, ISBN 0-9704363-0-0.
The full Web System Testing course is offered by SQE, and you'll find condensed versions at future SQE conferences. Steve Splaine can be reached at steve@splaine.net.
| Back to the newsletter index | Back to the home page |