Randomized Experiments for Planning and Evaluation A Practical Guide

by Robert F. Boruch

• ISBN13: 9780803935099
• ISBN10: 0803935099
• Format: Hardcover
• Copyright: 1997-01-23
• Publisher: Sage Publications

Summary

How can you tell if a new program has been effective or not? Is there any way to predict how a group of individuals would have fared without the new program or service? Packed with rich and varied examples, Robert F. Boruch untangles the complexities of randomized field experiments so that researchers can better evaluate the impact of new programs. Beginning with an exploration of how to judge whether or not a program worked (and for how much of the time), Boruch explores the context of experiments; the standards used to judge the ethical propriety of randomized experiments; methods for studying populations; and sampling methods, including pilot tests and arrangements for random allocation. He also covers baseline measures, including when to measure costs; missing data registry; and analyses of quality assurance, including analyses that have to be engineered to suit the departures from the design. Randomized Experiments for Planning and Evaluation is thoughtfully written, providing researchers with a reliable guide for constructing their own randomized field experiments. Randomized Experiments for Planning and Evaluation is for all professionals and practitioners in evaluation/research methods, statistics, psychology, sociology, political science, social work, and public health who need to stay on the cutting edge.

Table of Contents

1. Introduction	1	(18)
A General Problem	1	(1)
A Specific Example	2	(1)
Randomized Experiments for Planning and Evaluation	3	(1)
Distinctions	4	(1)
Illustrative Experiments from Different Fields	5	(5)
Elements of a Randomized Experiment	10	(9)
2. Experiments in the Context of Evaluation	19	(19)
A Common Scenario	19	(2)
Fundamental Questions That Underlie Evaluation Policy and Practice	21	(12)
Contemporary Interest in Randomized Field Experiments	33	(5)
3. Ethics, Law, and Randomized Experiments	38	(33)
Ethics and Federal Law	39	(12)
Ethical Propriety: Federal Judicial Center Guidelines	51	(2)
Statutory Protection of Privacy	53	(11)
The Law and Randomized Experiments	64	(2)
Scarce Resources and Lottery Allocation	66	(1)
Technical Approaches to Meeting Ethical Standards	67	(2)
Empirical Approaches to Understanding Fairness in Randomized Field Tests	69	(2)
4. Population, Power, and Pipeline	71	(25)
Units and Target Population	72	(5)
Statistical Power	77	(11)
Pipeline Study	88	(8)
5. Randomization Plans and Processes	96	(20)
The Randomization Procedure and Sequence	96	(3)
Subversion of Random Assignment	99	(1)
Integrity in the Random Assignment Procedure	100	(6)
Pilot Tests, Run-In Stages, and Precedent	106	(2)
Random Numbers and Randomization Algorithms	108	(2)
Arrangements for Random Allocation	110	(6)
6. Identifying, Engaging, and Maintaining Contact With Target Individuals	116	(22)
Identification	116	(3)
Mandatory Engagement in the Experiment	119	(1)
Voluntary Engagement in the Experiment	120	(2)
Incentives and Disincentives	122	(9)
Maintaining Contact	131	(7)
7. Observation and Measurement	138	(26)
The Role of Theory	139	(1)
Response Variables	140	(8)
Treatment Variables: Programs, Projects, or Practices	148	(6)
Baseline Measures	154	(3)
Observations on the Context of the Study	157	(2)
Costs	159	(3)
Missing Data Registry	162	(2)
8. Operations	164	(31)
Elements of Operations	165	(3)
The Sponsor's Role	168	(3)
Engaging Organizations	171	(9)
Scouting, Selecting, and Terminating Sites	180	(4)
On-Site-Advisory Groups	184	(1)
Information Collection	184	(2)
Oversight and Coordination of Multisite Field Experiments	186	(2)
Training and Training Materials	188	(8)
Appendix	196
9. Analysis	195	(24)
Analyses of Study Quality	196	(5)
Core Analysis	201	(6)
Understanding Estimates of Treatment Differences	207	(6)
Assessing Generalizability and Concordance	213	(6)
10. Reporting	219	(17)
Audiences for the Reports	219	(4)
Timeliness of Reports	223	(1)
The Kinds and Contents of Reports	224	(7)
Proprieties	231	(5)
References	236	(15)
Author Index	251	(6)
Subject Index	257	(8)
About the Author	265