Methods Matter Improving Causal Inference in Educational and Social Science Research

by Murnane, Richard J.; Willett, John B.

• ISBN13: 9780199753864
• ISBN10: 0199753865
• Format: Hardcover
• Copyright: 2010-09-17
• Publisher: Oxford University Press

Summary

Educational policy-makers around the world constantly make decisions about how to use scarce resources to improve the education of children. Unfortunately, their decisions are rarely informed by evidence on the consequences of these initiatives in other settings. Nor are decisions typically accompanied by well-formulated plans to evaluate their causal impacts. As a result, knowledge about what works in different situations has been very slow to accumulate. Over the last several decades, advances in research methodology, administrative record keeping, and statistical software have dramatically increased the potential for researchers to conduct compelling evaluations of the causal impacts of educational interventions, and the number of well-designed studies is growing. Written in clear, concise prose,Methods Matter: Improving Causal Inference in Educational and Social Science Researchoffers essential guidance for those who evaluate educational policies. Using numerous examples of high-quality studies that have evaluated the causal impacts of important educational interventions, the authors go beyond the simple presentation of new analytical methods to discuss the controversies surrounding each study, and provide heuristic explanations that are also broadly accessible. Murnane and Willett offer strong methodological insights on causal inference, while also examining the consequences of a wide variety of educational policies implemented in the U.S. and abroad. Representing a unique contribution to the literature surrounding educational research, this landmark text will be invaluable for students and researchers in education and public policy, as well as those interested in social science.

Author Biography

Richard J. Murnane, Juliana W. and William Foss Thompson Professor of Education and Society at Harvard University, is an economist who focuses his research on the relationships between education and the economy, teacher labor markets, the determinants of children's achievement, and strategies for making schools more effective.

John B. Willett, Charles William Eliot Professor of Education at Harvard University, is a quantitative methodologist who has devoted his career to improving the research design and data-analytic methods used in education and the social sciences, with a particular emphasis on the design of longitudinal research and the analysis of longitudinal data .

Table of Contents

Preface	p. xi
The Challenge for Educational Research	p. 3
The Long Quest	p. 3
The Quest Is Worldwide	p. 9
What This Book Is About	p. 10
What to Read Next	p. 12
The Importance of Theory	p. 14
What Is Theory?	p. 15
Theory in Education	p. 19
Voucher Theory	p. 21
What Kind of Theories?	p. 24
What to Read Next	p. 24
Designing Research to Address Causal Questions	p. 26
Conditions to Strive for in All Research	p. 27
Making Causal Inferences	p. 29
Past Approaches to Answering Causal Questions in Education	p. 31
The Key Challenge of Causal Research	p. 33
What to Read Next	p. 39
Investigator-Designed Randomized Experiments	p. 40
Conducting Randomized Experiments	p. 41
The Potential Outcomes Framework	p. 41
An Example of a Two-Group Experiment	p. 45
Analyzing Data from Randomized Experiments	p. 48
The Better Your Research Design, the Simpler Your Data Analysis	p. 48
Bias and Precision in the Estimation of Experimental Effects	p. 52
What to Read Next	p. 60
Challenges in Designing, Implementing, and Learning from Randomized Experiments	p. 61
Critical Decisions in the Design of Experiments	p. 62
Defining the Treatment	p. 64
Defining the Population from Which Participants Will Be Sampled	p. 66
Deciding Which Outcomes to Measure	p. 67
Deciding How Long to Track Participants	p. 68
Threats to the Validity of Randomized Experiments	p. 69
Contamination of the Treatment-Control Contrast	p. 70
Cross-overs	p. 70
Attrition from the Sample	p. 71
Participation in an Experiment Itself Affects Participants' Behavior	p. 73
Gaining Support for Conducting Randomized Experiments: Examples from India	p. 74
Evaluating an Innovative Input Approach	p. 75
Evaluating an Innovative Incentive Policy	p. 79
What to Read Next	p. 81
Statistical Power and Sample Size	p. 82
Statistical Power	p. 83
Reviewing the Process of Statistical Inference	p. 83
Defining Statistical Power	p. 92
Factors Affecting Statistical Power	p. 96
The Strengths and Limitations of Parametric Tests	p. 101
The Benefits of Covariates	p. 102
The Reliability of the Outcome Measure Matters	p. 103
The Choice Between One-Tailed and Two-Tailed Tests	p. 105
What to Read Next	p. 106
Experimental Research When Participants Are Clustered Within Intact Groups	p. 107
Random-Intercepts Multilevel Model to Estimate Effect Size When Intact Groups Are Randomized to Experimental Conditions	p. 110
Statistical Power When Intact Groups of Participants Are Randomized to Experimental Conditions	p. 120
Statistical Power of the Cluster-Randomized Design and Intraclass Correlation	p. 122
Fixed-Effects Multilevel Models to Estimate Effects Size When Intact Groups of Participants Are Randomized to Experimental Conditions	p. 128
Specifying a Fixed-Effects Multilevel Model	p. 128
Choosing Between Random-and Fixed-Effects Specifications	p. 131
What to Read Next	p. 134
Using Natural Experiments to Provide "Arguably Exogenous" Treatment Variability	p. 135
Natural- and Investigator-Designed Experiments: Similarities and Differences	p. 136
Two Examples of Natural Experiments	p. 137
The Vietnam-Era Draft Lottery	p. 137
The Impact of an Offer of Financial Aid for College	p. 141
Sources of Natural Experiments	p. 145
Choosing the Width of the Analytic Window	p. 150
Threats to Validity in Natural Experiments with a Discontinuity Design	p. 152
Accounting for the Relationship Between the Outcome and the Forcing Variable in a Discontinuity Design	p. 153
Actions by Participants Can Undermine Exogenous Assignment to Experimental Conditions in a Natural Experiment with a Discontinuity Design	p. 163
What to Read Next	p. 164
Estimating Causal Effects Using a Regression-Discontinuity Approach	p. 165
Maimonides' Rule and the Impact of Class Size on Student Achievement	p. 166
A Simple First-Differences Analysis	p. 170
A Difference-in-Differences Analysis	p. 171
A Basic Regression-Discontinuity Analysis	p. 174
Choosing an Appropriate Bandwidth	p. 181
Generalizing the Relationship Between the Outcome and the Forcing Variable	p. 186
Specification Checks Using Pseudo-Outcomes and Pseudo-Cut-Offs	p. 192
Regression-Discontinuity Designs and Statistical Power	p. 195
Additional Threats to Validity in a Regression-Discontinuity Design	p. 197
What to Read Next	p. 202
Introducing Instrumental-Variables Estimation	p. 203
Introducing Instrumental-Variables Estimation	p. 204
Bias in the OLS Estimate of the Causal Effect of Education on Civic Engagement	p. 206
Instrumental-Variable Estimation	p. 215
Two Critical Assumptions That Underpin Instrumental-Variables Estimation	p. 223
Alternative Ways of Obtaining the Instrumental-Variables Estimate	p. 226
Obtaining an Instrumental-Variables Estimate by the Two-Stage Least-Squares Method	p. 227
Obtaining an Instrumental-Variables Estimate by Simultaneous-Equations Estimation	p. 233
Extensions of the Basic Instrumental-Variable Estimation Approach	p. 238
Incorporating Exogenous Covariates into Instrumental-Variable Estimation	p. 238
Incorporating Multiple Instruments into the First-Stage Model	p. 243
Examining the Impact of Interactions Between the Endogenous Question Predictor and Exogenous Covariates in the Second-Stage Model	p. 247
Choosing Appropriate Functional Forms for Outcome/Predictor Relationships in First- and Second-Stage Models	p. 251
Finding and Defending Instruments	p. 252
Proximity of Educational Institutions	p. 253
Institutional Rules and Personal Characteristics	p. 257
Deviations from Cohort Trends	p. 261
The Search Continues	p. 263
What to Read Next	p. 264
Using IVE to Recover the Treatment Effect in a Quasi-Experiment	p. 265
The Notion of a "Quasi-Experiment"	p. 267
Using IVE to Estimate the Causal Impact of a Treatment in a Quasi-Experiment	p. 269
Further Insight into the IVE (LATE) Estimate, in the Context of Quasi-Experimental Data	p. 274
Using IVE to Resolve "Fuzziness" in a Regression-Discontinuity Design	p. 280
What to Read Next	p. 285
Dealing with Bias in Treatment Effects Estimated from Nonexperimental Data	p. 286
Reducing Observed Bias by the Method of Stratification	p. 289
Stratifying on a Single Covariate	p. 289
Stratifying on Covariates	p. 299
Reducing Observed Bias by Direct Control for Covariates Using Regression Analysis	p. 304
Reducing Observed Bias Using a Propensity-Score Approach	p. 310
Estimation of the Treatment Effect by Stratifying on Propensity Scores	p. 316
Estimation of the Treatment Effect by Matching on Propensity Scores	p. 321
Estimation of the Treatment Effect by Weighting by the Inverse of the Propensity Scores	p. 324
A Return to the Substantive Question	p. 328
What to Read Next	p. 331
Methodological Lessons from the Long Quest	p. 332
Be Clear About Your Theory of Action	p. 333
Learn About Culture, Rules, and Institutions in the Research Setting	p. 335
Understand the Counterfactual	p. 337
Always Worry About Selection Bias	p. 338
Use Multiple Outcome Measures	p. 340
Be on the Lookout for Longer-Term Effects	p. 341
Develop a Plan for Examining Impacts on Subgroups	p. 342
Interpret Your Research Results Correctly	p. 344
Pay Attention to Anomalous Results	p. 346
Recognize That Good Research Always Raises New Questions	p. 348
What to Read Next	p. 349
Substantive Lessons and New Questions	p. 350
Lower the Cost of School Enrollment	p. 352
Reduce Commuting Time	p. 352
Reduce Out-of-Pocket Educational Costs	p. 353
Reduce Opportunity Costs	p. 354
Change Children's Daily Experiences in School	p. 355
More Books?	p. 355
Smaller Classes?	p. 356
Better Teaching?	p. 357
Improve Incentives	p. 359
Improve Incentives for Teachers	p. 359
Improve Incentives for Students	p. 361
Create More Schooling Options for Poor Children	p. 364
New Private-School Options	p. 365
New Public-School Options	p. 367
Summing Up	p. 367
Final Words	p. 368
References	p. 369
Index	p. 381
Table of Contents provided by Ingram. All Rights Reserved.