Software Test Automation

by Fewster, Mark; Fewster, Mark; Graham, Dorothy

• ISBN13: 9780201331400
• ISBN10: 0201331403
• Edition: 1st
• Format: Paperback
• Copyright: 1999-06-28
• Publisher: Addison-Wesley Professional
• More Book Details
• Purchase Benefits

Summary

This book describes how to build and implement an automated testing regime for software development. It presents a detailed account of the principles of automated testing, practical techniques for designing a good automated testing regime, and advice on choosing and applying off-the-shelf testing tools to specific needs. This sound and practical introduction to automated testing comes from two authors well known for their seminars, consultancy and training in the field.

Author Biography

Dorothy Graham and Mark Fewster are the principal consultant partners of Grove Consultants which provides consultancy and training in software testing, test automation, and inspection. Mark Fewster developed the test automation design techniques which are the primary subject of this book. He has been refining and applying his ideas through consultancy with a wide variety of clients since 1991. Dorothy Graham is the originator and co-author of the CAST Report (Computer Aided Software Testing tools) published by Cambridge Market Intelligence, and the coauthor of Software Inspection published by Addison-Wesley in 1993. Both authors are popular and sought-after speakers at international conferences and workshops on software testing.

Table of Contents

Foreword	xii
Preface	xiii
Part 1: Techniques for Automating Test Execution
Test automation context	3	(23)
Introduction	3	(1)
Testing and test automation are different	4	(2)
The V-model	6	(1)
Tool support for life-cycle testing	7	(2)
The promise of test automation	9	(1)
Common problems of test automation	10	(3)
Test activities	13	(4)
Automate test design?	17	(5)
The limitations of automating software testing	22	(4)
Summary	25	(1)
Capture replay is not test automation	26	(39)
An example application: Scribble	26	(8)
The manual test process: what is to be automated?	34	(8)
Automating test execution: inputs	42	(6)
Automating test result comparison	48	(8)
The next steps in evolving test automation	56	(6)
Conclusion: automated is not automatic	62	(3)
Summary	63	(2)
Scripting techniques	65	(36)
Introduction	65	(10)
Scripting techniques	75	(17)
Script pre-processing	92	(9)
Summary	97	(4)
Automated comparison	101	(42)
Verification, comparison, and automation	101	(4)
What do comparators do?	105	(2)
Dynamic comparison	107	(1)
Post-execution comparison	108	(6)
Simple comparison	114	(1)
Complex comparison	115	(4)
Test sensitivity	119	(3)
Comparing different types of outcome	122	(8)
Comparison filters	130	(10)
Comparison guidelines	140	(3)
Summary	142	(1)
Testware architecture	143	(33)
What is testware architecture?	143	(1)
Key issues to be resolved	144	(5)
An approach	149	(25)
Might this be overkill?	174	(2)
Summary	174	(2)
Automating pre- and post-processing	176	(15)
What are pre -and post-processing?	176	(3)
Pre- and post-processing	179	(4)
What should happen after test case execution?	183	(1)
Implementation issues	184	(7)
Summary	190	(1)
Building maintainable tests	191	(12)
Problems in maintaining automated tests	191	(1)
Attributes of test maintenance	192	(7)
The conspiracy	199	(1)
Strategy and tactics	200	(3)
Summary	202	(1)
Metrics	203	(26)
Why measure testing and test automation?	203	(4)
What can we measure?	207	(2)
Objectives for testing and test automation	209	(2)
Attributes of software testing	211	(8)
Attributes of test automation	219	(6)
Which is the best test automation regime?	225	(1)
Should I really measure all these?	226	(3)
Summary	227	(2)
Other issues	229	(19)
Which tests should be automated (first)?	229	(3)
Selecting which tests to run when	232	(2)
Order of test execution	234	(2)
Test status	236	(7)
Designing software for (automated) testability	243	(1)
Synchronization	243	(1)
Monitoring progress of automated tests	244	(2)
Tailoring your own regime around your tools	246	(2)
Summary	247	(1)
Choosing a tool to automate testing	248	(34)
Introduction to Chapters 10 and 11	248	(1)
Where to start in selecting tools: your requirements, not the tool market	249	(1)
The tool selection project	250	(1)
The tool selection team	250	(2)
Identifying your requirements	252	(8)
Identifying your constraints	260	(7)
Build or buy?	267	(1)
Identifying what is available on the market	268	(3)
Evaluating the shortlisted candidate tools	271	(8)
Making the decision	279	(3)
Summary	280	(2)
Implementing tools within the organization	282	(25)
What could go wrong?	282	(1)
Importance of managing the implementation process	283	(2)
Roles in the implementation/change process	285	(2)
Management commitment	287	(1)
Preparation	288	(2)
Pilot project	290	(3)
Planned phased installation or roll-out	293	(1)
Special problems in implementing testing tools	294	(2)
People issues	296	(3)
Conclusion	299	(8)
Summary	300	(2)
Part 2: Test Automation Case Studies and Guest Chapters
Introduction: case studies and guest chapters	302	(5)
Racal-Redac case history	307	(19)
Mark Fewster
Keiron Marsden
Introduction	307	(1)
Background	307	(2)
Solutions	309	(1)
Integration test automation	310	(1)
System test automation	311	(4)
The results achieved	315	(4)
Summary of the case history up to 1991	319	(1)
What happened next?	320	(6)
The evolution of an automated software test system	326	(16)
Marnie Hutcheson
Introduction	326	(1)
Background	326	(1)
Gremlin 1	327	(3)
Gremlin 2.0: a step beyond capture replay	330	(2)
Finding the real problem	332	(4)
Lessons learned	336	(5)
Summary	341	(1)
Experiences with test automation	342	(9)
Clive Bates
Background	342	(1)
Planning, preparation, and eventual success	343	(3)
Benefits of test automation	346	(1)
Lessons learned	347	(2)
The way forward	349	(1)
Summary	349	(2)
Automating system testing in a VMS environment	351	(17)
Peter Oakley
Background	351	(1)
The first attempt at automation	352	(2)
New tool selection and evaluation	354	(2)
Implementation of V-Test	356	(9)
Conclusion	365	(3)
Automated testing of an electronic stock exchange	368	(11)
Paul Herzlich
Background	368	(1)
The system and testing	369	(2)
Test automation requirements	371	(1)
Test tool selection	372	(3)
Implementation	375	(1)
Maturity and maintenance	376	(1)
Our results	377	(2)
Insurance quotation systems tested automatically every month	379	(12)
Simon Mills
Background: the UK insurance industry	379	(1)
The brief, or how I became involved	380	(1)
Why automation?	381	(1)
Our testing strategy	382	(1)
Selecting a test automation tool	383	(1)
Some decisions about our test automation plans	384	(2)
The test plan	386	(2)
Some additional issues we encountered	388	(1)
A telling tale: tester versus automator	388	(1)
Summary	389	(2)
Three generations of test automation at ISS	391	(19)
Steve Allott
Introduction	391	(1)
The software under test	391	(1)
First generation	392	(3)
Second generation	395	(5)
Third generation	400	(6)
Three generations: a summary	406	(4)
Test automation failures: lessons to be learned	410	(8)
Stale Amland
Introduction	410	(1)
The projects	410	(2)
Problems	412	(3)
Recommendations	415	(2)
Pilot project	417	(1)
Epilogue	417	(1)
An unexpected application of test automation	418	(13)
Bob Bartlett
Susan Windsor
Introduction and background	418	(2)
Helping the bank make its product selection	420	(4)
Doing the testing	424	(2)
Automated testing	426	(2)
The results	428	(3)
Implementing test automation in an Independent Test Unit	431	(15)
Lloyd Roden
Introduction and background	431	(1)
The evaluation process	432	(2)
The implementation phase	434	(1)
The deployment of the tool	435	(2)
How QARun has been used	437	(5)
Problems we have experienced	442	(2)
The benefits achieved in two years	444	(1)
Conclusion	445	(1)
Testing with Action Words	446	(19)
Hans Buwalda
Introduction	446	(4)
Test clusters	450	(4)
The navigation	454	(3)
The test development life cycle	457	(4)
Applicability for other types of tests	461	(1)
Templates: meta-clusters	462	(2)
Summary	464	(1)
Regression testing at ABN AMRO Bank Development International	465	(9)
Iris Pinkster
Background	465	(2)
Problems with conventional testing	467	(2)
Pilot project using TestFrame	469	(1)
Regression test project	470	(2)
Spin-offs	472	(1)
Future	473	(1)
Summary	473	(1)
Business Object Scenarios: a fifth-generation approach to automated testing	474	(8)
Graham Dwyer
Graham Freeburn
Introduction	474	(1)
The five generations of testware development	474	(2)
RadSTAR	476	(1)
Window-centric Scenario Libraries	477	(1)
Business Object Scenarios	477	(2)
Mixing Business Object Scenarios with existing tests	479	(1)
Reuse versus repeatability	479	(1)
Conclusion	480	(2)
A test automation journey	482	(11)
Jim Thomson
Introduction	482	(1)
First steps	482	(4)
An off-the-shell automated test foundation: RadSTAR	486	(1)
How we have implemented automated testing using RadSTAR	487	(4)
Payback	491	(2)
Extracts from The Automated Testing Handbook	493	(24)
Linda Hayes
Introduction to this chapter	493	(1)
Introduction to the Handbook	493	(7)
Fundamentals of test automation	500	(4)
Test process and people	504	(2)
Test execution: analyzing results	506	(2)
Test metrics	508	(6)
More information about the Handbook	514	(3)
Building maintainable GUI tests	517	(20)
Chip Groder
Introduction and background	517	(2)
Cost drivers	519	(3)
Test planning and design	522	(5)
Well-behaved test cases	527	(1)
Encapsulated test set-up	528	(6)
Putting it all together	534	(1)
Summary	535	(2)
Test automation experience at Microsoft	537	(16)
Angela Smale
History	537	(4)
Batch files	541	(1)
Capture/playback tools	542	(1)
Scripting language	543	(1)
Cosmetic dialog box testing	544	(2)
Help testing tool	546	(1)
Tools to randomize test execution	547	(2)
What should I automate first?	549	(2)
My top ten list for a successful test automation strategy	551	(2)
Appendix	553	(2)
References	555	(2)
Glossary	557	(6)
Index	563