Evaluation in the Face of Uncertainty Anticipating Surprise and Responding to the Inevitable

by Morell, Jonathan A.

• ISBN13: 9781606238578
• ISBN10: 1606238574
• Edition: 1st
• Format: Paperback
• Copyright: 2010-08-12
• Publisher: The Guilford Press

Summary

Unexpected events during an evaluation all too often send evaluators into crisis mode. This insightful book provides a systematic framework for diagnosing, anticipating, accommodating, and reining in costs of evaluation "surprises." The result is evaluation that is better from a methodological point of view, and more responsive to stakeholders. Jonathan A. Morell identifies the types of surprises that arise at different stages of a program's life cycle and that may affect different aspects of the evaluation, from stakeholder relationships to data quality, methodology, funding, deadlines, information use, and program outcomes. His analysis draws on 18 concise cases from well-known researchers in a variety of evaluation settings. Morell offers guidelines for responding effectively to surprises and for determining the risks and benefits of potential solutions.

Author Biography

Jonathan A. Morell is a Senior Policy Analyst at the Vector Research Center, a division of TechTeam Government Solutions, and Editor of Evaluation and Program Planning. He is also active in the American Evaluation Association, where he has been instrumental in founding two of its topical interest groups—Systems, and Business and Industry—and is a recipient of the Association\u2019s Marcus Ingle Distinguished Service Award. His professional life has integrated his role as an evaluation practitioner with his theoretical interests. As a practitioner, he evaluates organizational change, R&D, and safety programs. He is also deeply involved in organizational change design. His theoretical interests include the nature and use of logic models, the role of Lean Six Sigma methodologies in evaluation, complex system behavior, and the nature of practical action.

Table of Contents

From Firefighting to Systematic Action	p. 1
Adding "Surprise" to the Mix	p. 1
Historical Roots: Evaluation, Planning, and System Behavior	p. 2
From Explaining Surprise to Dealing with It	p. 4
Development Path of This Book	p. 6
Guiding Principles	p. 8
How to Read This Book	p. 12
In Sum	p. 14
Structure of the Unexpected	p. 17
Where Does Surprise Come From?	p. 17
Beyond Simple Distinctions	p. 21
In Sum	p. 26
Placing Surprise in the Evaluation Landscape	p. 27
When Is the Probability of Surprise High?	p. 27
When Is Surprise Distruptive to Evaluation?	p. 35
In Sum	p. 44
Minimizing Foreseeable Surprise	p. 47
Theory: Using Explanatory Power and Simplified Relationships	p. 48
Exploiting Past Experience: Capitalizing on What We Already Know	p. 58
Limiting Time Frames to Minimize the Opportunity for Surprise	p. 63
In Sum	p. 67
Shifting from Advance Planning to Early Detection	p. 71
Leading Indicators	p. 72
System-Based Logic Modeling	p. 78
In Sum	p. 81
Agile Evaluation	p. 83
Data	p. 83
Agile Methodology	p. 87
Retooling Program Theory	p. 95
Agility and Stakeholder Needs	p. 98
In Sum	p. 98
How Much Is Too Much?: Appreciating Trade-Offs and Managing the Balance	p. 101
A Framework for Appreciating Design Trade-Offs	p. 103
Maximizing Choice, Minimizing Risk	p. 107
Evaluation Design	p. 112
In Sum	p. 113
Applying the Examples to Categories of Cases: The Life Cycle View	p. 115
"Unintended Consequences": Unity across Programs and Their Evaluations	p. 116
Interpreting Cases through a Life Cycle Perspective	p. 117
In Sum	p. 123
Applying the Examples to Categories of Cases: The Social/Organizational View	p. 125
Navigating through the Cases	p. 125
Placement of Cases on the Social/Organizational Map	p. 131
Categorizations Derived from the Data	p. 147
In Sum	p. 154
Lessons from Individual Cases: Tactics for Anticipating Surprise	p. 157
In Sum	p. 168
Lessons from Individual Cases: Responding to Surprise	p. 169
The Middle	p. 169
Leading Indicators and Agile Evaluation	p. 172
In Sum	p. 176
Unanticipated Program Outcomes	p. 177
Case Descriptions	p. 177
Applying the Cases to Unintended Program Outcomes	p. 181
Comparing the Cases	p. 183
Predicting the Need for Agile Evaluation	p. 187
In Sum	p. 191
Concluding Thoughts	p. 193
Cases
Grasping at Straws and Discovering a Different Program Theory: An Exercise in Reengineering Analysis Logic in a Child Care Evaluation Setting	p. 197
Shifting Sands in a Training Evaluation Context	p. 200
Evaluating Programs Aimed at Promoting Child Well-Being: The Case of Local Social Welfare Agencies in Jerusalem	p. 204
Assessing the Impact of Providing Laptop Computers to Students	p. 210
Quasi-Experimental Strategies When Randomization Fails: Propensity Score Matching and Sensitivity Analysis in Whole-School Reform	p. 214
Unexpected Changes in Program Delivery: The Perils of Overlooking Process Data When Evaluating HIV Prevention	p. 219
Evaluating Costs and Benefits of Consumer-Operated Services: Unexpected Resistance, Unanticipated Insights and Déjà Vu All Over Again	p. 224
Keep Up with the Program!: Adapting the Evaluation Focus to Align with a College Access Program's Changing Goals	p. 231
Assumptions about School Staff's Competencies and Likely Program Impacts	p. 235
Mixed Method Evaluation of a Support Project for Nonprofit Organizations	p. 241
Evaluating the Health Impacts of Central Heating	p. 244
Recruiting Target Audience: When All Else Fails, Use the Indirect Approach for Evaluating Substance Abuse Prevention	p. 249
Unintended Consequences of Changing Funder Requirements Midproject on Outcome Evaluation Design and Results in HIV Outreach Services	p. 253
Generating and Using Evaluation Feedback for Providing Countywide Family Support Services	p. 258
Trauma and Posttraumatic Stress Disorder among Female Clients in Methadone Maintenance Treatment in Israel: From Simple Assessment to Complex Intervention	p. 263
From Unintended to Undersirable Effects of Health Intervention: The Case of User Fees Abolition in Niger, West Africa	p. 270
Unintended Consequences and Adapting Evaluation: Katrina Aid Today National Case Management Consortium	p. 277
Evaluation of the Integrated Services Pilot Program from Western Australia	p. 281
References	p. 287
Index	p. 293
About the Author	p. 299
Contributors	p. 301
Table of Contents provided by Ingram. All Rights Reserved.