Evaluation in the Face of Uncertainty Anticipating Surprise and Responding to the Inevitable

by Morell, Jonathan A.

• ISBN13: 9781606238585
• ISBN10: 1606238582
• Edition: 1st
• Format: Hardcover
• Copyright: 2010-08-11
• Publisher: The Guilford Press

Summary

Unexpected events during an evaluation all too often send evaluators into crisis mode. This insightful book provides a systematic framework for diagnosing, anticipating, accommodating, and reining in costs of evaluation "surprises." The result is evaluation that is better from a methodological point of view, and more responsive to stakeholders. Jonathan A. Morell identifies the types of surprises that arise at different stages of a program's life cycle and that may affect different aspects of the evaluation, from stakeholder relationships to data quality, methodology, funding, deadlines, information use, and program outcomes. His analysis draws on 18 concise cases from well-known researchers in a variety of evaluation settings. Morell offers guidelines for responding effectively to surprises and for determining the risks and benefits of potential solutions.

Author Biography

Jonathan A. Morell is a Senior Policy Analyst at the Vector Research Center, a division of TechTeam Government Solutions, and Editor of Evaluation and Program Planning. He is also active in the American Evaluation Association, where he has been instrumental in founding two of its topical interest groups—Systems, and Business and Industry—and is a recipient of the Association\u2019s Marcus Ingle Distinguished Service Award. His professional life has integrated his role as an evaluation practitioner with his theoretical interests. As a practitioner, he evaluates organizational change, R&D, and safety programs. He is also deeply involved in organizational change design. His theoretical interests include the nature and use of logic models, the role of Lean Six Sigma methodologies in evaluation, complex system behavior, and the nature of practical action.

Table of Contents

1. From Firefighting to Systematic Action Adding \u201cSurprise\u201d to the Mix Historical Roots: Evaluation, Planning, and System Behavior From Explaining Surprise to Dealing with It Development Path of This Book Guiding Principles How to Read This Book In Sum 2. Structure of the Unexpected Where Does Surprise Come From? Beyond Simple Distinctions In Sum 3. Placing Surprise in the Evaluation Landscape When Is the Probability of Surprise High? When Is Surprise Disruptive to Evaluation? In Sum 4. Minimizing Foreseeable Surprise Theory: Using Explanatory Power and Simplified Relationships Exploiting Past Experience: Capitalizing on What We Already Know Limiting Time Frames to Minimize the Opportunity for Surprise In Sum 5. Shifting from Advance Planning to Early Detection Leading Indicators System-Based Logic Modeling In Sum 6. Agile Evaluation Data Agile Methodology Retooling Program Theory Agility and Stakeholder Needs In Sum 7. How Much Is Too Much?: Appreciating Trade-Offs and Managing the Balance A Framework for Appreciating Design Trade-Offs Maximizing Choice, Minimizing Risk Evaluation Design In Sum 8. Applying the Examples to Categories of Cases: The Life Cycle View \u201cUnintended Consequences\u201d: Unity across Programs and Their Evaluations Interpreting Cases through a Life Cycle Perspective In Sum t;"9. Applying the Examples to Categories of Cases: The Social/Organizational View Navigating through the Cases Placement of Cases on the Social/Organizational Map Categorizations Derived from the Data In Sum 10. Lessons from Individual Cases: Tactics for Anticipating Surprise In Sum 11. Lessons from Individual Cases: Responding to Surprise The Middle Leading Indicators and Agile Evaluation In Sum 12. Unanticipated Program Outcomes Case Descriptions Applying the Cases to Unintended Program Outcomes Comparing the Cases Predicting the Need for Agile Evaluation In Sum 13. Concluding Thoughts Cases Case 1. Grasping at Straws and Discovering a Different Program Theory: An Exercise in Reengineering Analysis Logic in a Child Care Evaluation Setting, Dennis P. Affholter Case 2. Shifting Sands in a Training Evaluation Context, James W. Altschuld and Phyllis M. Thomas Case 3. Evaluating Programs Aimed at Promoting Child Well-Being: The Case of Local Social Welfare Agencies in Jerusalem, Anat Zeira Case 4. Assessing the Impact of Providing Laptop Computers to Students, J. Dan Strahl, Deborah L. Lowther, and Steven M. Ross Case 5. Quasi-Experimental Strategies When Randomization Fails: Propensity Score Matching and Sensitivity Analysis in Whole-School Reform, Gary L. Bowen, Roderick A. Rose, and Shenyang Guo Case 6. Unexpected Changes in Program Delivery: The Perils of Overlooking Process Data When Evaluating HIV Prevention, Bryce D. Smith Case 7. Evaluating Costs and Benefits of Consumer-Operated Services: Unexpected Resistance, Unanticipated Insights, and D\u00e9j\u00e0 Vu All Over Again, Brian T. Yates Case 8. Keep Up with the Program!: Adapting the Evaluation Focus to Align with a College Transition Program\u2019s Changing Goals, Kristine L. Chadwick and Jennifer Conner Blatz Case 9. Assumptions about School Staff\u2019s Competencies and Likely Program Impacts, Laura Hassler Lang, Christine E. Johnson, and Shana Goldwyn Case 10. Mixed Method Evaluation of a Support Project for Nonprofit Organizations, Riki Savaya and Mark Waysman Case 11. Evaluating the Health Impacts of Central Heating, Jeremy Walker, Richard Mitchell, Stephen Platt, and Mark Petticrew Case 12. Recruiting Target Audience: When All Else Fails, Use the Indirect Approach for Evaluating Substance Abuse Prevention, Molly Engle Case 13. Unintended Consequences of Changing Funder Requirements Midproject on Outcome Evaluation Design and Results in HIV Outreach Services, Lena Lundgren, Therese Fitzgerald, and Deborah Chassler Case 14. Generating and Using Evaluation Feedback for Providing Countywide Family Support Services, Deborah L. Wasserman Case 15. Trauma and Posttraumatic Stress Disorder among Female Clients in Methadone Maintenance Treatment in Israel: From Simple Assessment to Complex Intervention Miriam Schiff and Shabtay Levit Case 16. From Unintended to Undesirable Effects of Health Intervention: The Case of User Fees Abolition in Niger, West Africa, Val\u00e9ry Ridde and Aissa Diarra Case 17. Unintended Consequences and Adapting Evaluation: Katrina Aid Today National Case Management Consortium, Amanda Janis and Kelly M. Stiefel Case 18. Evaluation of the Integrated Services Pilot Program from Western Australia, Peter Hancock, Trudi Cooper, and Susanne Therese Bahn