Critical Testing Processes Plan, Prepare, Perform, Perfect

by Black, Rex

• ISBN13: 9780201748680
• ISBN10: 0201748681
• Edition: 1st
• Format: Paperback
• Copyright: 2003-07-29
• Publisher: Addison-Wesley Professional
• More Book Details
• Purchase Benefits

Summary

bull; A must-read for software testers from a noted software testing guru bull; Examples, specifics, and a running case study bring the content to life bull; Separates software test processes into three categories: routing, highly-visible, and mission-critical

Author Biography

Rex Black is president and principal consultant of Rex Black Consulting Services, Inc., a testing and quality-assistance firm.

Table of Contents

Foreword	p. xvii
Acknowledgments	p. xix
Preface	p. xxi
Introduction	p. xxv
Book Scope	p. xxv
Who Should Read This Book?	p. xxvii
Demystifying Processes	p. xxviii
Types and Relationships of Critical Testing Processes	p. xxix
Critical Testing Processes in Context	p. xxx
Sumatra, a Hypothetical Project	p. xxxiii
Terminology	p. xxxiii
On Using This Book	p. xxxiv
Recognize Good Processes	p. xxxiv
Handle Challenges	p. xxxv
Implement Improvements	p. xxxv
To Your Testing Success!	p. xxxvi
Plan	p. 1
Start with the Big Picture: Put the Test Role in the Broader Context	p. 3
A Testing Process	p. 8
Understanding Operational and Organizational Context	p. 11
Introducing the Sumatra Project and the SpeedyWriter Test Team	p. 14
Testing within the System Development Lifecycle	p. 18
Organizing the Testers and the Testing	p. 24
Management Dimensions	p. 31
Beyond Testing Process Context	p. 33
Focus the Effort: Analyze Quality Risks	p. 35
A Quality Risk Analysis Process	p. 36
Jamal Introduces the Team to Quality Risk Analysis	p. 36
Recognize a Good Quality Risk Analysis Process	p. 44
Involve the Right Participants	p. 44
Employ the Right Technique	p. 48
Separate the Vital Few From the Trivial Many	p. 50
Find Potential Bugs before Testing	p. 52
Start at the Right Time	p. 54
Adapt to Change	p. 54
Handle Challenges	p. 56
Getting People to Get Real about Risk	p. 56
Quality Risks from Various Sources	p. 58
Who Drives the Quality Risk Analysis Bus?	p. 59
Eliminating the Appearance of Magic in the Quantification of Risk	p. 60
Dealing with Variation in Ratings	p. 61
Convincing Stakeholders to Invest Time and Accept Reality	p. 62
Implement Improvements	p. 63
Gaze into the Crystal Ball: Estimate the Work Ahead	p. 65
An Estimation Process	p. 67
Jamal Predicts the Future	p. 70
Using Rules of Thumb for Rough Estimation	p. 81
Effort, Duration, and Dependencies	p. 83
Planning	p. 83
Staffing	p. 86
Test Development	p. 87
Test Environment Configuration	p. 91
Test Execution	p. 92
Moving on to Dollars and Cents	p. 95
It's Not What It Costs, It's What It Saves: Budget and Return on Investment	p. 97
Analyzing Return on Investment for Testing	p. 98
Bugs That Get Fixed--or Prevented	p. 99
Bugs That Don't Get Fixed--but Are Known	p. 106
Tests That Mitigate Risks	p. 107
Guiding the Project	p. 110
A Final Note on Return on Investment Estimates	p. 112
Jamal Prepares the Bill--and the Justification	p. 113
Moving Forward, Then Looking Back	p. 120
From Estimate to Baseline: Obtain Commitment to Realistic, Actionable, Truthful Estimates	p. 121
Selling the Estimate	p. 121
Jamal Makes His Case	p. 126
Recognize a Good Estimation Process	p. 129
Understand the Factors that Affect the Estimate	p. 129
Paint a Realistic Picture of the Project's Future	p. 132
Assign Responsibility for Action	p. 134
Predict with Honesty	p. 135
Estimate Quickly	p. 137
Handle Challenges	p. 137
What Kind of Investment Is Testing, Really?	p. 138
It Might Look Like Magic--But It's Only an Estimate	p. 144
The Nine-Month Rule	p. 145
Be Careful with Nonstakeholders	p. 147
When Selling Fails	p. 148
Safety- or Mission-Critical Systems	p. 149
Implement Improvements	p. 150
Gain and Communicate Insights: Plan the Test Effort	p. 153
A Test Planning Process	p. 154
Jamal Has a Plan	p. 155
Beyond the Process: Key Considerations for the Test Plan	p. 170
Selecting Strategies for Testing	p. 171
A Special Case: Regression Risk Management Strategies	p. 176
Understanding Test Environment Execution Resources	p. 181
Managing Test Systems and Environment Configurations	p. 183
Test Subproject Risks	p. 185
Getting Everyone On Board	p. 187
From Suggestion to Commitment: Assemble Stakeholder Support for Good Test Plans	p. 189
Jamal Makes His Pitch	p. 189
Recognize a Good Test Planning Process	p. 194
Establish Clear Criteria for Phases	p. 194
Establish Consensus, Common Expectations, and Commitment	p. 197
Complete at a Reasonable Time	p. 199
Promote Reasonable Flexibility and Creativity	p. 200
Produce Appropriate Documentation	p. 202
Provide Opportunities to Catch Errors	p. 204
Exploit Synergies with the Project, Development, Build, and Integration Plans	p. 205
Handle Challenges	p. 207
Transcending the Template	p. 207
Planning for Outsourcing	p. 208
When Key Players Don't Support the Test Plan	p. 209
Obeying the Law and Pleasing Regulators	p. 213
Implement Improvements	p. 214
Prepare	p. 217
Bring on the Great Testers: The How and the Who of Hiring	p. 219
A Team Building Process	p. 220
Variations on the Team Building Process	p. 222
The Sumatra Test Team Grows	p. 223
Identifying and Managing the Critical Test Team Skills	p. 232
Regarding Test Technicians	p. 235
Test Team Staffing Variations: Temporary Assignment, Rotation, Farm Team, and Backwater	p. 238
Education, Training, Certification, and the Professionalization of Testing	p. 241
Attitude Matters Too	p. 244
Beyond the Obvious Interview Questions	p. 245
From Adding People to Growing Skills	p. 248
Grow Excellent Test Teams: Skills, Attitudes, and Career Paths	p. 249
Jamal and Lin-Tsu Discuss Skills Growth	p. 249
Using Skills Assessment as a Career Growth and Team Management Tool	p. 252
Recognize a Good Team Building Process	p. 254
Operate on a Win-Win Philosophy	p. 254
Accurately Define the Job	p. 256
Provide the Employee with a Long-Term Career Path	p. 257
Give Employees Appropriate and Fair Salaries	p. 262
Retain Employees with Sufficient Education, Experience, and Skill	p. 263
Allow Appropriate Specialization	p. 264
Handle Challenges	p. 266
What Do Testers Need to Know--and What Can You Get?	p. 266
White Elephants	p. 268
Using Contractors and Consultants	p. 269
Implement Improvements	p. 271
Archimedes' Bathtub: Design and Implement Test Systems	p. 273
A Test System Design and Implementation Process	p. 275
Emma Creates Stress on the Job	p. 275
Three Crucial Considerations	p. 286
Test Oracles	p. 286
Appropriate Documentation	p. 288
Combinatorial Explosions	p. 293
Shifting to the Management Picture	p. 296
Fill the Bathtub: Test System Coverage and Quality	p. 297
A Brief Survey of Test Coverage Analysis Techniques	p. 297
Jamal Assesses Coverage	p. 301
Recognize a Good Test System Design and Implementation Process	p. 305
Provides Capable Tools for Assessment of Quality	p. 305
Efficiently Implements, Uses, Reuses, and Maintains the Test System	p. 307
Selects the Right Techniques	p. 311
Prevents Bugs	p. 314
Handle Challenges	p. 314
How Many Conditions in a Test Case?	p. 314
Retesting Third-Party Components	p. 316
Vaguely Defined Requirements, Usage Profiles, and Environments	p. 317
Balancing Coverage, Schedule, and Budget	p. 319
Implement Improvements	p. 320
Perform	p. 323
An Essential Handoff: Manage Test Releases	p. 325
A Test Release Process	p. 326
The Big Build Hits the Test Lab	p. 327
Recognize a Good Test Release Process	p. 335
Provide Installable, Improved, and Stable Test Releases	p. 335
Produce Predictable, Timely, and Regular Test Releases	p. 337
Use Defined, Customer-Like Install and Uninstall Processes	p. 340
Give the Test Release a Name That It Knows	p. 342
Build from Carefully Selected, Coordinated, and Documented Content	p. 344
Handle Challenges	p. 347
Who Owns the Test Release Processes?	p. 347
Complexity of the System Under Test--and Beyond	p. 348
Incomplete and Unrealistic System and Acceptance Test Releases	p. 351
Implement Improvements	p. 352
Assess Quality: Execute Test Cases	p. 355
A Test Execution Process	p. 355
Test Tracking Worksheets	p. 357
The Attack on the Big Build	p. 360
Recognize a Good Test Execution Process	p. 368
Find the Scary Stuff First	p. 368
Produce, Gather, and Disseminate Valuable Information	p. 370
Correctly Interpret Test Results	p. 372
Exhibit the Proper Attitudes	p. 375
Proceed Efficiently	p. 377
Handle Challenges	p. 379
Dealing with Extended Shifts and Outsourced Testing	p. 379
Accommodating Holidays, Cultures, and Vacations	p. 381
Capturing the History on the Ground	p. 382
Implement Improvements	p. 382
Perfect	p. 385
Where Quality Falls Short: Report Bugs	p. 387
A Bug Reporting Process	p. 388
A Big Bug in the Big Build	p. 392
Beyond the Failure Description	p. 398
Recognize a Good Bug Reporting Process	p. 402
Effectively Communicate Useful Information	p. 402
Describe One Symptom per Report and File One Report per Symptom	p. 405
Draw a Clear Boundary Between Testing and Debugging	p. 406
Handle Challenges	p. 408
Separating Bugs, Features, and "So What"	p. 408
Handling Bugs That Got Fixed by Accident and Irreproducible Symptoms	p. 410
Keeping Noise Out of the Signal	p. 412
Building Trust and Credibility with Programmers	p. 413
Selecting the Right Bug Tracking Tool	p. 414
Implement Improvements	p. 416
Illuminate the Fourth Element: Report the Test Results	p. 419
A Test Results Reporting Process	p. 420
Jamal Defines a Dashboard--and Reports on the Big Build's Test Results	p. 422
Recognize a Good Test Results Reporting Process	p. 440
Deliver Useful, Credible, and Timely Information	p. 441
Relate Test Status to Project Progress--and Influence Future Progress	p. 444
Show Trends and Destinations in Quality over Time	p. 447
Use Appropriate Mechanisms and Documents	p. 448
Tune the Communication to Each Listener	p. 450
Handle Challenges	p. 451
Handling the Presentation	p. 452
The Bearer of Bad News	p. 453
An Ever-Changing Complex Story	p. 457
Maintaining a Passion for Quality While Giving a Dispassionate Presentation	p. 458
Implement Improvements	p. 459
Leverage Opportunities to Learn: Manage Change and Its Effects on Testing	p. 461
A Change Management Process	p. 462
Jamal Makes a Case--and Accommodates	p. 464
An Interconnected Process	p. 474
Recognize a Good Change Management Process	p. 475
Select the Right Changes in the Right Order	p. 476
Balance Considerations of Features, Schedule, Budget, and Quality	p. 477
Handle Challenges	p. 478
The Complex Nature of Change Effects	p. 478
The Ripple Effects of Change	p. 480
Becoming a Roadblock--or Becoming Perceived as a Roadblock	p. 481
Implement Improvements	p. 482
Return to the Big Picture: Perfect the Testing Process	p. 485
Recognize a Good Test Process	p. 485
Provide Valuable, Economical Services	p. 485
Pervade the Project	p. 487
Use a Mature Process	p. 489
Apply the Appropriate Level of Formality and Standardization	p. 490
Employ Appropriate Strategies	p. 496
Successfully Introduce Tools	p. 498
Enjoy Continuous Learning and Inspiration	p. 500
Handle Challenges	p. 501
How Can We Measure What We Can't Define?	p. 501
Immature Development or Maintenance Processes	p. 502
Unrealistic Expectations	p. 503
Relationships	p. 505
"Addicted to Test"	p. 506
Incremental Process Improvement	p. 508
Fine-Tune the Big Processes	p. 508
Identify and Remove the Big Process Obstacles	p. 510
Implement Improvements	p. 512
Jamal Brown Looks Back--and Looks Forward	p. 514
Conclusion	p. 517
Glossary	p. 519
Bibliography	p. 535
Index	p. 545
Table of Contents provided by Ingram. All Rights Reserved.

Excerpts

I've spent most of my twenty years in the software and hardware business inthe arena of testing. For the first few years as a test practitioner, I struggled tokeep my head above water. Ultimately, I mastered some basic tools and techniques. As I learned more about testing, I started to notice certain common themes.Some of these themes had to do with eventsgood and badthat happenedover and over again on software, hardware, and system projects. For some of theseevents, I found that some teams could create order in their projects. These teamshandled these common events better than the teams that bounced from one crisis tothe next, reacting constantly, immersed in chaos. The successful teams had goodprocesses. Some of these successful project teams implemented writtenprocesses, while others accumulated "institutional knowledge" in theirwiseand sometimes prematurely grayheads. While I have nothing againsta shared company culture, it's hard to pass along the processes you'velearned unless you write them down-whether formally or informally, as checklists.This book takes the informal road. I describe twelve specific test processes, usingchecklists.Each process is critical to test team success. I describe theseprocesses in chronological order. First we plan the test activities. Next we prepareto test. After that, we perform the tests. Finally, we perfect the system under testand the testing activities themselves. Many other books have covered thetopics of preparing and performing tests in great detail. My experience is that, astesters, we generally do a good job in these areas. So, instead of rehashing what wealready know, I focus on opportunities for improvement. I devote eleven of theseventeen chapters to the topics of planning and perfecting. By far, these are theareas where we as testers have the most difficulty. This is especially true forcomplex and critical projects. Where will this book take you? During theearly colonization of the American continent in the 1540s, Francisco Vasquez deCoronado searched the deserts of present-day Arizona and New Mexico for the SevenCities of C'bola, including El Dorado, a city whose streets were supposedly pavedwith gold. Juan Ponce De Leon searched for a Fountain of Youth. In 1911, one of thefirst management consultants, Frederick Winslow Taylor, wrote a book calledThePrinciples of Scientific Management. Taylor espoused the idea of the one bestwaythe perfect processfor each activity on an assembly line or in anyother industrial enterprise. But none of these three men found streets of gold, lifewithout death, or perfect processes. This book isn't about Quixoticquests. There are no streets of gold that will make us effortlessly rich. Wecan't side-step our human limitations. I don't have infallibleprocesses. As Frederick Brooks wrote in theMythical Man-Month,SecondEdition, we don't have any silver bullets to kill our system projectmonsters, including the ones that live in quality and testing. That said, I havefound many ways for testers to deliver valuable information and services to theproject team, and each of these ways has its strong points and its weaknesses. The processes in this book might differ from what you're doing now. In somecases you'll decide, based on the success of your current processes, thatyou're doing a fine job already. In some cases, though, you may want toimplement improvements. I'll discuss specific ways to do that, but two themesapply to process change throughout the book. First, only change what'sbroken when changing it will help. Process change for its own sake, or processchange to perfect an already-good process, often doesn't help the test team orthe organization. Indeed, such efforts c