Pitfalls and Pratfalls: Null and Negative Findings in Evaluating Interventions New Directions for Evaluation, Number 110

by Hudley, Cynthia; Parker, Robert Nash

• ISBN13: 9780787988289
• ISBN10: 0787988286
• Edition: 1st
• Format: Paperback
• Copyright: 2006-09-11
• Publisher: Jossey-Bass

Summary

This volume examines the problem of null or negative evaluation findings, a topic rarely discussed in the literature but all too commonplace in the experience of evaluators. The Southern California Center of Excellence on Youth Violence Prevention, housed in the Robert Presley Center for Crime and Justice Studies at the University of California, Riverside, has taken up the challenge to discuss candidly evaluation efforts that can be described only as challenging. The individual chapters discuss a range of design, implementation, and analysis issues relevant not only to evaluation studies but also to interventions that can contribute to negative or null findings in the evaluation of an intervention program. These problems that are the realities of life for anyone who conducts prevention and intervention research are typically the stuff of research seminar comments and barroom digressions late in the evening at professional meetings. This issue brings those important lessons into the larger discussion that will influence prevention science and public policy. The contributors to this volume not only admit a set of problems and shortcomings but also attempt to draw general lessons, cautions, and advice for those who evaluate prevention and intervention efforts. This is the 110 th volume of New Directions for Evaluation, a quarterly journal published by Jossey-Bass.

Table of Contents

EDITORS' NOTES	1	(4)
1. Qualitative Lessons from a Community-Based Violence Prevention Project with Null Findings	5	(14)
Kirk R. Williams, Sabrina Arredondo Mattson
This chapter discusses the difficulty of intervening to change attitudes and behavior in the face of peer group and community norms that support risky adolescent behavior.
2. Masked Intervention Effects: Analytic Methods for Addressing Low Dosage of Intervention	19	(14)
John E. Lochman, Caroline Boxmeyer, Nicole Powell, David L. Roth, Michael Windle
This chapter critically compares multiple analysis strategies for examining intervention effects when rates of participation in the program vary widely.
3. Evaluation of a Teen Dating Violence Social Marketing Campaign: Lessons Learned When the Null Hypothesis Was Accepted	33	(12)
Emily F. Rothman, Michele R. Decker, and Jay G. Silverman
Problems inherent in the evaluation of mass media campaigns are critically discussed and alternative strategies for evaluation are compared.
4. How Much of a Good Thing Is Too Much? Explaining the Failure of a Well-Designed, Well-Executed Intervention in Juvenile Hall for Hard-to-Place Delinquents	45	(14)
Robert Nash Parker, Emily K. Asencio, Deborah Plechner
Political and institutional policies and practices that can influence program implementation and evaluation are critically analyzed.
5. What Works (and What Does Not) in Youth Violence Prevention: Rethinking the Questions and Finding New Answers	59	(14)
Nancy G. Guerra, Paul Boxer, Clayton R. Cook
Evaluation results from the Metropolitan Area Child Study are used to examine the unique and unexpected impact of local contexts on intervention programming and evaluation findings.
6. Who Is Watching the Watchers? The Challenge of Observing Peer Interactions on Elementary School Playgrounds	73	(14)
Cynthia Hudley
This chapter describes issues in collecting observational data and synthesizing the results of an evaluation study that employed a mixed-methods design.
7. Commentary on the Pitfalls and Pratfalls of Evaluation Research with Intervention and Prevention Programs	87	(10)
Karen L. Bierman
The final chapter highlights themes running throughout the chapters and suggests implications for evaluating prevention programs and evaluation overall.
INDEX	97