Social Experiments in Practice: The What, Why, When, Where, and How of Experimental Design and Analysis New Directions for Evaluation, Number 152

by Peck, Laura R.

• ISBN13: 9781119348887
• ISBN10: 1119348889
• Edition: 1st
• Format: Paperback
• Copyright: 2017-01-10
• Publisher: Jossey-Bass

Summary

This issue considers social experiments in practice and how recent advances improve their value and potential applications. Although controversial, it is clear they are here to stay and are in fact increasing.

With their greater abundance, experimental evaluations have stretched to address more diverse policy questions, no longer simply providing a treatment–control contrast but adding multiarm, multistage, and multidimensional (factorial) designs and analytic extensions to expose more about what works best for whom. Social experiments are also putting programs under the microscope when they are most ready for testing, enhancing the policy value of their findings.

This volume provides new developments in all these areas from scholars instrumental to recent scientific advances. In some instances, established ideas are given new attention, connecting them to new opportunities to learn and inform policy. By all means, this issue aims to encourage stronger and more informative social experiments in the future.

This is the 152nd issue in the New Directions for Evaluation series from Jossey-Bass. It is an official publication of the American Evaluation Association.

Author Biography

Laura R. Peck is a principal scientist at Abt Associates Inc., Social and Economic Policy Division and director of the Research and Evaluation Expertise Center.

Table of Contents

EDITOR’S NOTES. SOCIAL EXPERIMENTS IN PRACTICE: INTRODUCTION, FRAMING, AND CONTEXT 9
Laura Peck, Issue Editor, and Naomi Goldstein

New approaches to designing and conducting experimental evaluations can improve their utility by enhancing the generalizability of their findings as well as allowing them to determine which specific aspects of a program or policy produce impacts.

1. On the “Why” of Social Experiments: Some Lessons on Overcoming Barriers from 45 Years of Social Experiments 19
Howard Rolston

The history of social policy experiments has demonstrated their utility and value and produced methods for overcoming practical barriers to their application.

2. On the “When” of Social Experiments: The Tension Between Program Refinement and Abandonment 33
Diana Epstein, Jacob Alex Klerman

This chapter considers how to proceed when a process analysis shows that a program does not satisfy the intermediate steps of its own logic model or when an impact analysis does not show clear evidence of impact: when to further refine the program model and when to abandon the program model.

3. On the “Where” of Social Experiments: The Nature and Extent of the Generalizability Problem 47
Stephen H. Bell, Elizabeth A. Stuart

This chapter describes research strategies for investigating how much nonrepresentative site selection biases affect findings produced by rigorous evaluations of social programs.

4. On the “Where” of Social Experiments: Selecting More Representative Samples to Inform Policy 61
Robert B. Olsen, Larry L. Orr

This chapter describes how social experiments can evaluate interventions or programs in samples that are more representative of the population of interest from a policy perspective.

5. On the “How” of Social Experiments: Using Implementation Research to Get Inside the Black Box 73
Lawrence M. Mead

Implementation research describes the actual character of evaluated programs, analyzes why local sites vary in performance, and helps identify additional programs that might be promising.

6. On the “How” of Social Experiments: Analytic Strategies for Getting Inside the Black Box 85
Laura R. Peck

Analysis of postrandom assignment (endogenous) events or experiences in experimental evaluation data is becoming increasingly widespread. This chapter highlights some analytic strategies for revealing mediators of program impacts.

7. On the “How” of Social Experiments: Experimental Designs for Getting Inside the Black Box 97
Stephen H. Bell, Laura R. Peck

This chapter discusses several experimental evaluation designs that randomize individuals—including multiarmed, multistage, and factorial designs—to permit estimating impacts for specific policy design features or program elements within a larger examination of an overall program.

8. Program and Policy Evaluations in Practice: Highlights from the Federal Perspective 109
Rebecca Maynard, Naomi Goldstein, Demetra Smith Nightingale

From the vantage point of federal agencies that commission a large share of domestic program evaluations, this chapter explores the conditions under which particular methodological strategies are and are not helpful in advancing the job of policy analysts, policymakers, and program administrators.

INDEX 137