Software Testing Concepts and Operations

by Mili, Ali; Tchier, Fairouz

• ISBN13: 9781118662878
• ISBN10: 1118662873
• Edition: 1st
• Format: Hardcover
• Copyright: 2015-06-15
• Publisher: Wiley

Summary

Explores and identifies the main issues, concepts, principles and evolution of software testing, including software quality engineering and testing concepts, test data generation, test deployment analysis, and software test management

This book examines the principles, concepts, and processes that are fundamental to the software testing function. This book is divided into five broad parts. Part I introduces software testing in the broader context of software engineering and explores the qualities that testing aims to achieve or ascertain, as well as the lifecycle of software testing. Part II covers mathematical foundations of software testing, which include software specification, program correctness and verification, concepts of software dependability, and a software testing taxonomy. Part III discusses test data generation, specifically, functional criteria and structural criteria. Test oracle design, test driver design, and test outcome analysis is covered in Part IV. Finally, Part V surveys managerial aspects of software testing, including software metrics, software testing tools, and software product line testing.

• Presents software testing, not as an isolated technique, but as part of an integrated discipline of software verification and validation
• Proposes program testing and program correctness verification within the same mathematical model, making it possible to deploy the two techniques in concert, by virtue of the law of diminishing returns
• Defines the concept of a software fault, and the related concept of relative correctness, and shows how relative correctness can be used to characterize monotonic fault removal
• Presents the activity of software testing as a goal oriented activity, and explores how the conduct of the test depends on the selected goal
• Covers all phases of the software testing lifecycle, including test data generation, test oracle design, test driver design, and test outcome analysis

Software Testing: Concepts and Operations is a great resource for software quality and software engineering students because it presents them with fundamentals that help them to prepare for their ever evolving discipline.

Author Biography

ALI MILI is Professor at New Jersey Institute of Technology, USA. His research is focused on software engineering, including technical and organizational aspects, and on software engineering education.

FAIROUZ TCHIER is Associate Professor at King Saud University, Saudi Arabia. Her main areas of research are discrete mathematics, theoretical computer science, software engineering, and fuzzy theory.

Table of Contents

Preface xiv

Part I Introduction to Software Testing 1

1 Software Engineering: A Discipline Like No Other 3

1.1 A Young, Restless Discipline 3

1.2 An Industry Under Stress 5

1.3 Large, Complex Products 5

1.4 Expensive Products 7

1.5 Absence of Reuse Practice 9

1.6 Fault-Prone Designs 9

1.7 Paradoxical Economics 10

1.7.1 A Labor-Intensive Industry 10

1.7.2 Absence of Automation 11

1.7.3 Limited Quality Control 11

1.7.4 Unbalanced Lifecycle Costs 12

1.7.5 Unbalanced Maintenance Costs 12

1.8 Chapter Summary 13

1.9 Bibliographic Notes 13

2 Software Quality Attributes 14

2.1 Functional Attributes 15

2.1.1 Boolean Attributes 15

2.1.2 Statistical Attributes 15

2.2 Operational Attributes 17

2.3 Usability Attributes 18

2.4 Business Attributes 19

2.5 Structural Attributes 20

2.6 Chapter Summary 21

2.7 Exercises 21

2.8 Bibliographic Notes 22

3 A Software Testing Lifecycle 23

3.1 A Software Engineering Lifecycle 23

3.2 A Software Testing Lifecycle 27

3.3 The V-Model of Software Testing 32

3.4 Chapter Summary 33

3.5 Bibliographic Notes 34

Part II Foundations of Software Testing 35

4 Software Specifications 37

4.1 Principles of Sound Specification 38

4.1.1 A Discipline of Specification 38

4.2 Relational Mathematics 39

4.2.1 Sets and Relations 39

4.2.2 Operations on Relations 39

4.2.3 Properties of Relations 41

4.3 Simple Input Output Programs 42

4.3.1 Representing Specifications 42

4.3.2 Ordering Specifications 46

4.3.3 Specification Generation 48

4.3.4 Specification Validation 53

4.4 Reliability Versus Safety 60

4.5 State-based Systems 61

4.5.1 A Relational Model 62

4.5.2 Axiomatic Representation 64

4.5.3 Specification Validation 70

4.6 Chapter Summary 72

4.7 Exercises 72

4.8 Problems 76

4.9 Bibliographic Notes 78

5 Program Correctness and Verification 79

5.1 Correctness: A Definition 80

5.2 Correctness: Propositions 83

5.2.1 Correctness and Refinement 83

5.2.2 Set Theoretic Characterizations 85

5.2.3 Illustrations 86

5.3 Verification 88

5.3.1 Sample Formulas 89

5.3.2 An Inference System 91

5.3.3 Illustrative Examples 94

5.4 Chapter Summary 98

5.5 Exercises 99

5.6 Problems 100

5.7 Bibliographic Notes 100

6 Failures, Errors, and Faults 101

6.1 Failure, Error, and Fault 101

6.2 Faults and Relative Correctness 103

6.2.1 Fault, an Evasive Concept 103

6.2.2 Relative Correctness 104

6.3 Contingent Faults and Definite Faults 107

6.3.1 Contingent Faults 107

6.3.2 Monotonic Fault Removal 109

6.3.3 A Framework for Monotonic Fault Removal 114

6.3.4 Definite Faults 114

6.4 Fault Management 116

6.4.1 Lines of Defense 116

6.4.2 Hybrid Validation 118

6.5 Chapter Summary 121

6.6 Exercises 122

6.7 Problems 123

6.8 Bibliographic Notes 124

7 A Software Testing Taxonomy 125

7.1 The Trouble with Hyphenated Testing 125

7.2 A Classification Scheme 126

7.2.1 Primary Attributes 127

7.2.2 Secondary Attributes 131

7.3 Testing Taxonomy 136

7.3.1 Unit-Level Testing 136

7.3.2 System-Level Testing 138

7.4 Exercises 139

7.5 Bibliographic Notes 140

Part III Test Data Generation 141

8 Test Generation Concepts 143

8.1 Test Generation and Target Attributes 143

8.2 Test Outcomes 146

8.3 Test Generation Requirements 148

8.4 Test Generation Criteria 152

8.5 Empirical Adequacy Assessment 155

8.6 Chapter Summary 160

8.7 Exercises 161

8.8 Bibliographic Notes 162

8.9 Appendix: Mutation Program 163

9 Functional Criteria 165

9.1 Domain Partitioning 165

9.2 Test Data Generation from Tabular Expressions 171

9.3 Test Generation for State Based Systems 176

9.4 Random Test Data Generation 184

9.5 Tourism as a Metaphor for Test Data Selection 188

9.6 Chapter Summary 190

9.7 Exercises 190

9.8 Bibliographic Notes 192

10 Structural Criteria 193

10.1 Paths and Path Conditions 194

10.1.1 Execution Paths 194

10.1.2 Path Functions 196

10.1.3 Path Conditions 201

10.2 Control Flow Coverage 202

10.2.1 Statement Coverage 202

10.2.2 Branch Coverage 204

10.2.3 Condition Coverage 207

10.2.4 Path Coverage 209

10.3 Data Flow Coverage 214

10.3.1 Definitions and Uses 214

10.3.2 Test Generation Criteria 217

10.3.3 A Hierarchy of Criteria 220

10.4 Fault-Based Test Generation 220

10.4.1 Sensitizing Faults 221

10.4.2 Selecting Input Data for Fault Sensitization 225

10.4.3 Selecting Input Data for Error Propagation 227

10.5 Chapter Summary 228

10.6 Exercises 229

10.7 Bibliographic Notes 232

Part IV Test Deployment and Analysis 233

11 Test Oracle Design 235

11.1 Dilemmas of Oracle Design 235

11.2 From Specifications to Oracles 238

11.3 Oracles for State-Based Products 242

11.3.1 From Axioms to Oracles 243

11.3.2 From Rules to Oracles 244

11.4 Chapter Summary 250

11.5 Exercises 251

12 Test Driver Design 253

12.1 Selecting a Specification 253

12.2 Selecting a Process 255

12.3 Selecting a Specification Model 257

12.3.1 Random Test Generation 257

12.3.2 Pre-Generated Test Data 263

12.3.3 Faults and Fault Detection 266

12.4 Testing by Symbolic Execution 269

12.5 Chapter Summary 274

12.6 Exercises 275

12.7 Bibliographic Notes 279

13 Test Outcome Analysis 280

13.1 Logical Claims 281

13.1.1 Concrete Testing 281

13.1.2 Symbolic Testing 282

13.1.3 Concolic Testing 283

13.2 Stochastic Claims: Fault Density 284

13.3 Stochastic Claims: Failure Probability 287

13.3.1 Faults are Not Created Equal 287

13.3.2 Defining/Quantifying Reliability 289

13.3.3 Modeling Software Reliability 291

13.3.4 Certification Testing 294

13.3.5 Reliability Estimation and Reliability Improvement 295

13.3.6 Reliability Standards 299

13.3.7 Reliability as an Economic Function 300

13.4 Chapter Summary 307

13.5 Exercises 308

13.6 Problems 310

13.7 Bibliographic Notes 310

Part V Management of Software Testing 311

14 Metrics for Software Testing 313

14.1 Fault Proneness 314

14.1.1 Cyclomatic Complexity 315

14.1.2 Volume 316

14.2 Fault Detectability 317

14.3 Error Detectability 320

14.4 Error Maskability 323

14.5 Failure Avoidance 324

14.6 Failure Tolerance 326

14.7 An Illustrative Example 327

14.7.1 Cyclomatic Complexity 327

14.7.2 Volume 328

14.7.3 State Redundancy 328

14.7.4 Functional Redundancy 328

14.7.5 Non-injectivity 329

14.7.6 Non-determinacy 329

14.7.7 Summary 330

14.8 Chapter Summary 330

14.9 Exercises 331

14.10 Bibliographic Notes 332

15 Software Testing Tools 333

15.1 A Classification Scheme 333

15.2 Scripting Tools 334

15.2.1 CppTest 334

15.2.2 SilkTest 335

15.3 Record-and-Replay Tools 336

15.3.1 TestComplete 336

15.3.2 Selenium IDE 337

15.4 Performance-Testing Tools 338

15.4.1 LoadRunner 338

15.4.2 Grinder 339

15.4.3 QF-Test 340

15.4.4 Appvance PerformanceCloud 340

15.4.5 JMeter 341

15.5 Oracle Design Tools 342

15.5.1 JUnit 342

15.5.2 TestNG 343

15.6 Exception Discovery 343

15.6.1 Rational Purify 343

15.6.2 Astree 344

15.7 Collaborative Tools 345

15.7.1 FitNesse 345

15.8 Chapter Summary 345

16 Testing Product Lines 347

16.1 PLE: A Streamlined Reuse Model 347

16.2 Testing Issues 351

16.3 Testing Approaches 353

16.4 Illustration 354

16.4.1 Domain Analysis 354

16.4.2 Domain Modeling 356

16.4.3 A Reference Architecture 359

16.4.4 Domain Implementation 360

16.4.5 Testing at Domain Engineering 365

16.4.6 Testing at Application Engineering 369

16.5 Chapter Summary 372

16.6 Exercises 372

16.7 Problems 372

16.8 Bibliographic References 373

Bibliography 374

Index 377