Basics of Software Engineering Experimentation

by Juristo, Natalia; Moreno, Ana M.

• ISBN13: 9780792379904
• ISBN10: 079237990X
• Format: Hardcover
• Copyright: 2000-12-01
• Publisher: Kluwer Academic Pub

Summary

Basics of Software Engineering Experimentation is a practical guide to experimentation in a field which has long been underpinned by suppositions, assumptions, speculations and beliefs. It demonstrates to software engineers how Experimental Design and Analysis can be used to validate their beliefs and ideas. The book does not assume its readers have an in-depth knowledge of mathematics, specifying the conceptual essence of the techniques to use in the design and analysis of experiments and keeping the mathematical calculations clear and simple. Basics of Software Engineering Experimentation is practically oriented and is specially written for software engineers, all the examples being based on real and fictitious software engineering experiments.

Table of Contents

List of Figures	xi
List of Tables	xiii
Foreword	xix
Acknowledgements	xxi
PART I: INTRODUCTION TO EXPERIMENTATION
Introduction
Pre-Scientific Status of Software Engineering	3	(3)
Why Don't We Experiment In Se?	6	(4)
Kinds of Empirical Studies	10	(2)
Amplitude of Experimental Studies	12	(5)
Goals of This Book	17	(1)
Who Does This Book Target?	18	(1)
Objectives To Be Achieved By the Reader of This Book	19	(1)
Organisation of the Book	20	(3)
Why Experiment? The Role of Experimentation In Scientific and Technological Research
Introduction	23	(1)
Research and Experimentation	23	(3)
The Social Aspect in Software Engineering	26	(1)
The Experimentation/Learning Cycle	27	(6)
Scientific Method	33	(2)
Why Do Experiments Need To Be Replicated?	35	(5)
Empirical Knowledge Versus Theoretical Knowledge	40	(5)
How To Experiment?
Introduction	45	(1)
Searching for Relationships Among Variables	45	(2)
Strategy of Stepwise Refinement	47	(2)
Phases of Experimentation	49	(2)
Role of Statistics In Experimentation	51	(6)
PART II: DESIGNING EXPERIMENTS
Basic Notions of Experimental Design
Introduction	57	(1)
Experimental Design Terminology	57	(8)
The Software Project As An Experiment	65	(5)
Response Variables In Se Experimentation	70	(10)
Suggested Exercises	80	(3)
Experimental Design
Introduction	83	(1)
Experimental Design	83	(2)
One-Factor Designs	85	(5)
How To Avoid Variations of No Interest To The Experiment: Block Designs	90	(7)
Experiments With Multiple Sources of Desired Variation: Factorial Designs	97	(5)
What To Do When Factorial Alternatives Are Not Comparable: Nested Designs	102	(1)
How To Reduce The Amount of Experiments: Fractional Designs	103	(1)
Experiments With Several Desired and Undesired Variations: Factorial Block Designs	104	(9)
Importance of Experimental Design and Steps	113	(3)
Specific Considerations for Experimental Designs In Software Engineering	116	(3)
Suggested Exercises	119	(6)
PART III: ANALYSING THE EXPERIMENTAL DATA
Basic Notions of Data Analysis
Introduction	125	(1)
Experimental Results As A Sample of A Population	126	(2)
Statistical Hypotheses and Decision Making	128	(4)
Data Analysis for Large Samples	132	(5)
Data Analysis for Small Samples	137	(10)
Readers' Guide To Part III	147	(4)
Suggested Exercises	151	(2)
Which Is The Better of Two Alternatives? Analysis of One-Factor Designs With Two Alternatives
Introduction	153	(1)
Statistical Significance of the Difference Between Two Alternatives Using Historical Data	153	(7)
Significance of the Difference Between Two Alternatives When No Historical Data Are Available	160	(3)
Analysis for Paired Comparison Designs	163	(2)
One-Factor Analysis With Two Alternatives In Real Se Experiments	165	(8)
Suggested Exercises	173	(2)
Which of K Alternatives Is the Best? Analysis for One-Factor Designs and K Alternatives
Introduction	175	(1)
Identification of the Mathematical model	176	(3)
Validation of the Basic Model That Relates the Experimental Variables	179	(7)
Calculations the Factor- and Error-induced Variation in the Response Variable	186	(3)
Calculating the Statistical Significance of the Factor-Induced Variation	189	(6)
Recommendations Or Conclusions of the Analysis	195	(4)
Analysis of One Factor With K Alternatives In Real Se Experiments	199	(2)
Suggested Exercises	201	(2)
Experiments With Undesired Variations: Analysis for Block Designs
Introduction	203	(1)
Analysis for Designs With A Single Blocking Variable	203	(13)
Analysis for Designs With Two Blocking Variables	216	(3)
Analysis for Two Blocking Variable Designs and Replication	219	(1)
Analysis for Designs With More Than Two Blocking Variables	220	(7)
Analysis When There Are Missing Data In Block Designs	227	(2)
Analysis for Incomplete Block Designs	229	(3)
Suggested Exercises	232	(3)
Best Alternatives for More Than One Variable: Analysis for Factorial Designs
Introduction	235	(1)
Analysis of General Factorial Designs	236	(10)
Analysis for Factorial Designs With Two Alternatives Per Factor	246	(23)
Analysis for Factorial Designs Without Replication	269	(11)
Handling Unbalanced Data	280	(6)
Analysis of Factorial Designs In Real Se Experiments	286	(3)
Suggested Exercises	289	(4)
Experiments With Incomparable Factor Alternatives: Analysis for Nested Designs
Introduction	293	(1)
Identification of the Mathematical Model	294	(1)
Validation of the Model	294	(1)
Calculation of the Variation In the Response Variable Due To Factors and Error	295	(1)
Statistical Significance of the Variation In The Response Variable	296	(1)
Suggested Exercises	297	(2)
Fewer Experiments: Analysis for Fractional Factorial Designs
Introduction	299	(1)
Choosing the Experiments In A 2k-p Fractional Factorial Design	300	(5)
Analysis for 2k-p Designs	305	(5)
Suggested Exercises	310	(3)
Several Desired and Undesired Variations: Analysis for Factorial Block Designs
Introduction	313	(1)
Identification of the Mathematical Model	314	(2)
Calculation of Response Variable Variability	316	(1)
Statistical Significance of the Variation In the Response Variable	317	(3)
Analysis of Factorial Block Designs In Real Se Experiments	320	(1)
Suggested Exercises	321	(2)
Non-Parametric Analysis Methods
Introduction	323	(1)
Non-Parametric Methods Applicable To Independent Samples	324	(4)
Non-Parametric Methods Applicable To Related Samples	328	(2)
Non-Parametric Analysis In Real Se Experiments	330	(4)
Suggested Exercises	334	(3)
How Many Times Should An Experiment Be Replicated?
Introduction	337	(1)
Importance of the Number of Replications In Experimentation	338	(1)
The Value of the Means of the Alternatives To Be Used To Reject H0 Is Known	338	(3)
The Value of the Difference Between Two Means of the Alternatives To Be Used To Reject H0 Is Known	341	(1)
The Percentage Value To Be Exceeded By the Standard Deviation To Be Used To Reject H0 Is Known	342	(1)
The Difference Between the Means of the Alternatives To Be Used To Reject H0 Is Known for More Than One Factor	343	(2)
Suggested Exercises	345	(4)
PART IV: CONCLUSIONS
Some Recommendations On Experimenting
Introduction	349	(1)
Precautions To Be Taken Into Account In Se Experiments	349	(5)
A Guide To Documenting Experimentation	354	(5)
REFERENCES	359
ANNEXES
Some Software Project Variables	367
Some Useful Latin Squares and How They Are Used To Build Greco-Latin and Hyper-Greco-Latin Squares	379
Statistical Tables	385