Evaluation in Social Work The Art and Science of Practice

by Unrau, Yvonne A.; Gabor, Peter A.; Grinnell, Richard M.

• ISBN13: 9780195308068
• ISBN10: 0195308069
• Edition: 4th
• Format: Paperback
• Copyright: 2006-09-14
• Publisher: Oxford University Press

Summary

This book was written for you-a graduate social work student-as an introduction to program evaluation. We selected and arranged its contents so it can be used in a beginning one-semester social work program evaluation course, a social work administrative course, or a program planning course. It is designed to prepare you to: Participate in evaluative activities within your social service organization, Become a beginning critical producer of the professional evaluative literature, Become a beginning consumer of the professional evaluative literature, Master more advanced evaluation courses and texts.

Table of Contents

About the Authors	xxvii
Part I Preparing for an Evaluation
Becoming an Accountable Practitioner	3	(24)
Evaluation and Accountability	4	(4)
The Council on Social Work Education	4	(2)
The National Association of Social Workers	6	(2)
Quality Improvement in Social Service Programs	8	(2)
Case-Level Evaluation	8	(1)
Program-Level Evaluation	9	(1)
Case-Level and Program-Level Data for Quality Improvement	9	(1)
Evaluation and the Profession	10	(9)
Contributing to Evidence-Based Practice	10	(1)
Collaborating With Program Stakeholders	11	(2)
Policymakers	13	(1)
General Public	14	(1)
Funders	15	(1)
Administrators	15	(1)
Practitioners	15	(1)
Clients	16	(1)
A Word About Collaboration Among Stakeholder Groups	16	(1)
Integrated Accountability With Service Delivery	17	(1)
Client-Centered Practice	18	(1)
Evaluation From a Person-in-Environment Perspective	19	(1)
Evaluation From a Program-in-Environment Perspective	20	(2)
Summing Up and Looking Ahead	22	(1)
Recap and Online Materials	23	(1)
Study Questions	23	(1)
References, Further Reading, and Resources	24	(3)
Approaches to Accountability	27	(26)
The External Project Approach	32	(3)
Characteristics of the External Project Approach	32	(1)
Externally Driven	32	(2)
Resistant Social Workers	34	(1)
Intrusiveness	34	(1)
Periodic or No Feedback to Social Workers	34	(1)
Large Recommended Changes	34	(1)
Not Practical in Applied Settings	35	(1)
Difficult to Incorporate in Practice Settings	35	(1)
The Internal Monitoring Approach	35	(7)
Characteristics of the Internal Monitoring Approach	36	(1)
Internally Driven	36	(1)
Cooperative Social Workers	37	(1)
Integrated	37	(1)
Ongoing Continuous Feedback	38	(1)
Minor Recommended Change	39	(1)
Easy to Incorporate in Practice Settings	39	(1)
Advantages of the Internal Monitoring Approach	39	(1)
Increased Understanding of Programs	39	(1)
Relevant Feedback	40	(1)
Timely Feedback	41	(1)
Self-Protection	41	(1)
Practitioner and Client Satisfaction	41	(1)
Professionalism	41	(1)
Fine-Tuning Programs	42	(2)
Summing Up and Looking Ahead	44	(6)
Recap and Online Materials	50	(1)
Study Questions	50	(1)
References, Further Reading, and Resources	51	(2)
Designing Client-Centered Programs	53	(34)
Social Service Agencies	53	(3)
Agency Mission Statements	54	(1)
Agency Goals	55	(1)
Requirements for Goals	55	(1)
Agency Objectives	56	(1)
Social Service Programs	56	(5)
An Agency Versus a Program	61	(1)
Program Designs	61	(8)
Program Goals	62	(2)
Unintended Program Results	64	(1)
Program Goals Versus Agency Goals	64	(2)
Types of Program Objectives	66	(1)
Knowledge-Based Objectives	66	(1)
Affect-Based Objectives	67	(1)
Behavioral-Based Objectives	67	(1)
Qualities of Program Objectives	67	(1)
Meaningful	67	(1)
Specific	68	(1)
Measurable	68	(1)
Directional	69	(1)
Program Versus Practice Objectives	69	(2)
Example 1: Bob's Self-Sufficiency	69	(1)
Example 2: Jane's Job Dissatisfaction	70	(1)
Program Activities	71	(1)
Program Logic Models	71	(11)
Summing Up and Looking Ahead	82	(1)
Recap and Online Materials	82	(1)
Study Questions	82	(2)
References, Further Reading, and Resources	84	(3)
Getting Ready for an Evaluation	87	(34)
Program Scope and Evaluation	88	(1)
Planning With Stakeholders	89	(16)
Asking Evaluation Questions	91	(3)
Mapping Concepts	94	(2)
Reviewing the Literature	96	(2)
Developing Schedules	98	(1)
Tasks	99	(1)
Roles	99	(2)
Timelines	101	(2)
Producing Documentation	103	(2)
Identifying Data Needs	105	(1)
Focusing Evaluation Efforts	106	(8)
Client Demographics	107	(2)
Service Statistics	109	(1)
Quality Standards	110	(1)
Feedback	111	(1)
Client Outcomes	112	(2)
Summing Up and Looking Ahead	114	(1)
Recap and Online Materials	115	(1)
Study Questions	115	(1)
References, Further Reading, and Resources	116	(5)
Part II Doing an Evaluation
Doing a Needs Assessment	121	(34)
What Are Needs Assessments?	122	(1)
Social Problems	122	(2)
Social Needs	124	(3)
Perceived Needs	126	(1)
Normative Needs	126	(1)
Relative Needs	127	(1)
Expressed Needs	127	(1)
Program Solutions	127	(1)
Steps in Doing a Needs Assessment	128	(22)
Step 1: Focusing the Problem	130	(1)
Step 2: Developing Needs Assessment Questions	130	(1)
Step 3: Identifying Targets for Intervention (Unit of Analysis)	131	(1)
Establishing Target Parameters	132	(1)
Sampling (Data Sources)	133	(1)
Step 4: Developing a Data Collection Plan	134	(1)
Reviewing Existing Reports	135	(1)
Secondary Data Analyses	136	(1)
Individual Interviews	137	(1)
Group Interviews	138	(3)
Telephone and Mail Surveys	141	(2)
Step 5: Analyzing and Displaying Data	143	(1)
Collecting Quantitative Data	143	(2)
Collecting Qualitative Data	145	(1)
Step 6: Disseminating and Communicating Findings	146	(4)
Summing Up and Looking Ahead	150	(2)
Recap and Online Materials	152	(1)
Study Questions	152	(1)
References, Further Reading, and Resources	153	(2)
Doing a Process Evaluation	155	(36)
Purposes of Process Evaluations	156	(3)
To Improve a Program's Operations	157	(1)
To Generate Knowledge for Our Profession	158	(1)
To Estimate Cost Efficiency	158	(1)
Steps in a Process Evaluation	159	(23)
Step 1: Deciding What Questions to Ask	159	(1)
What Is the Program's Background?	159	(1)
What Is the Program's Client Profile?	160	(1)
What Is the Program's Staff Profile?	160	(1)
What Is the Amount of Service Provided to Clients?	161	(1)
What Are the Program's Interventions and Activities?	162	(2)
What Administrative Supports Are in Place to Support Client Service Delivery?	164	(7)
How Satisfied Are the Program's Stakeholders?	171	(2)
How Efficient Is the Program?	173	(1)
Step 2: Developing Data Collection Instruments	173	(1)
Ease of Use	173	(1)
Appropriateness to the Flow of a Program's Operations	174	(1)
Design With User Input	175	(1)
Step 3: Developing a Data Collection Monitoring System	175	(1)
Number of Cases to Include (Unit of Analysis)	176	(1)
Times to Collect the Data	177	(1)
Methods for Collecting the Data	178	(1)
Step 4: Scoring and Analyzing Data	179	(1)
Step 5: Developing a Feedback System	180	(2)
Step 6: Disseminating and Communicating Results	182	(1)
Summing Up and Looking Ahead	182	(2)
Recap and Online Materials	184	(1)
Study Questions	184	(1)
References, Further Reading, and Resources	185	(6)
Doing an Outcome Evaluation	191	(30)
Purpose of Outcome Evaluations	193	(1)
Uses of Outcome Evaluations	194	(2)
Improving Program Services to Clients	194	(1)
Influencing Decisions?	195	(1)
Generating Knowledge for the Profession	195	(1)
Steps in Outcome Evaluations	196	(21)
Step 1: Conceptualizing and Operationalizing Program Objectives	196	(3)
Step 2: Operationalizing Variables and Stating the Outcomes	199	(2)
Step 3: Designing a Monitoring System	201	(7)
Deciding the Number of Clients to Include (Unit of Analysis)	208	(2)
Deciding When Data Will Be Collected	210	(2)
Deciding How Data Will Be Collected	212	(1)
Step 4: Analyzing and Displaying Data	212	(2)
Step 5: Developing a Feedback System	214	(1)
Step 6: Disseminating and Communicating Results	215	(2)
Summing Up and Looking Ahead	217	(1)
Recap and Online Materials	217	(1)
Study Questions	217	(1)
References, Further Reading, and Resources	218	(3)
Doing an Efficiency Evaluation	221	(26)
Cost Effectiveness Versus Cost Benefit	222	(4)
When to Evaluate for Efficiency	222	(4)
Steps in Conducting a Cost-Benefit Evaluation	226	(11)
Step 1: Deciding on an Accounting Perspective	226	(1)
The Individual Program's Participant's Perspective	226	(1)
The Funding Source's Perspective	227	(1)
Applying the Procedure	227	(1)
Step 2: Specifying the Cost--Benefit Model	228	(1)
Looking at Costs	228	(1)
Looking at Benefits	228	(1)
Applying the Procedure	228	(2)
Step 3: Determining Costs	230	(1)
Direct Costs	230	(1)
Indirect Costs	230	(1)
Applying the Procedure	230	(1)
Step 4: Determining Benefits	231	(1)
Applying the Procedure	232	(1)
Step 5: Adjusting for Present Value	233	(1)
Applying the Procedure	234	(1)
Step 6: Completing the Cost-Benefit Analysis	235	(1)
Applying the Procedure	235	(2)
Cost-Effectiveness Analysis	237	(1)
Applying the Procedure	238	(1)
A Few Words About Efficiency-Focused Evaluations	238	(3)
Summing Up and Looking Ahead	241	(1)
Recap and Online Materials	241	(1)
Study Questions	241	(1)
References, Further Reading, and Resources	242	(5)
Part III Gathering Data and Making Decisions
Measuring Practice and Program Objectives	247	(28)
Why Measurement Is Necessary	247	(3)
Objectivity	249	(1)
Precision	249	(1)
Types of Measuring Instruments	250	(8)
Rating Scales	250	(1)
Graphic Rating Scales	251	(1)
Self-Anchored Rating Scales	251	(2)
Summated Scales	253	(1)
Goal Attainment Scaling (GAS)	254	(2)
Creating Practice Objectives From Program Objectives	256	(2)
Measurement by the Numbers	258	(4)
Standardized Measuring Instruments	262	(6)
Is the Measurement Useful?	268	(2)
Summing Up and Looking Ahead	270	(1)
Recap and Online Materials	270	(1)
Study Questions	270	(1)
References, Further Reading, and Resources	271	(4)
Data Sources, Sampling, and Data Collection Methods	275	(30)
Data Sources	276	(3)
Sampling	279	(4)
Probability Sampling	280	(1)
Nonprobability Sampling	280	(3)
Data Collection Methods	283	(15)
Obtaining Existing Data	289	(1)
Documents and Reports	289	(1)
Data Sets	290	(1)
Obtaining New Data	291	(1)
Face-to-Face Individual Interviews	292	(1)
Surveys	292	(2)
Group Interviews	294	(2)
Observation	296	(2)
Fitting Data Collection to the Program	298	(1)
Ease of Use	298	(1)
Appropriateness to the Flow of Program Operations	299	(1)
Design With User Input	299	(1)
Developing a Data Collection Monitoring System	299	(2)
Summing Up and Looking Ahead	301	(1)
Recap and Online Materials	301	(1)
Study Questions	301	(1)
References, Further Reading, and Resources	302	(3)
Developing a Data Information System	305	(26)
Staff Members' Roles in Developing a Data Information System	306	(2)
Establishing an Organizational Plan	307	(1)
Case-Level Data Collection	308	(2)
Program-Level Data Collection	310	(11)
Data Collection at Intake	311	(3)
Data Collection at Client Contact	314	(3)
Data Collection at Termination	317	(1)
Data Collection to Obtain Feedback	318	(3)
Data Management	321	(3)
Manual Data Management	321	(1)
Computer-Assisted Data Management	322	(1)
Reporting	323	(1)
A Look to the Future	324	(1)
Summing Up and Looking Ahead	324	(1)
Recap and Online Materials	325	(1)
Study Questions	325	(1)
References, Further Reading, and Resources	326	(5)
Using Graphics to Report Evaluation Data	331	(10)
Characteristics of an Effective Graphic	333	(1)
Bar Charts	334	(1)
Pie Charts	334	(1)
Line Graphs	334	(1)
Illustrations	335	(1)
Photographs	336	(1)
Checklist	337	(1)
Summing Up and Looking Ahead	338	(1)
Recap and Online Materials	338	(1)
Study Questions	338	(1)
References, Further Reading, and Resources	339	(2)
Analyzing Qualitative Data	341	(20)
Narrative Data	341	(1)
The Analysis Process	342	(6)
Step 1: Get to Know Your Data	342	(1)
Step 2: Focus the Analysis	343	(1)
Focus by Question or Topic, Time Period, or Event	343	(1)
Focus by Case, Individual, or Group	343	(1)
Step 3: Categorize Information	343	(1)
Preset Categories	344	(1)
Emergent Categories	344	(1)
Step 4: Identify Patterns and Connections Within and Between Categories	345	(1)
Within Category Description	346	(1)
Larger Categories	347	(1)
Relative Importance	347	(1)
Relationships	347	(1)
Step 5: Interpretation---Bringing It All Together	347	(1)
``Nuts and Bolts'' of Narrative Analysis	348	(4)
Data Management Tips	348	(1)
Check Your Data	348	(1)
Add ID Numbers	348	(1)
Prepare Data for Analysis	348	(1)
Make Copies	349	(1)
Identify the Source of All Data	349	(1)
Mark Key Themes	349	(1)
Define Categories	350	(1)
Cut and Sort	351	(1)
Make Connections	352	(1)
Enhancing the Process	352	(1)
Use Several Sources of Data	352	(1)
Track Your Choices	353	(1)
Involve Others	353	(1)
Pitfalls to Avoid	353	(1)
Avoid Generalizing	353	(1)
Choose Quotes Carefully	354	(1)
Address Limitations and Alternative Explanations	354	(1)
Summing Up and Looking Ahead	354	(3)
Recap and Online Materials	357	(1)
Study Questions	357	(1)
References, Further Reading, and Resources	358	(3)
Making Decisions With Data	361	(28)
Objective Data	361	(1)
Subjective Data	362	(2)
Case-Level Decision Making	364	(11)
The Engagement and Problem-Definition Phase	364	(1)
The Practice Objective Setting Phase	365	(1)
The Intervention Phase	366	(1)
Deterioration, or No Change	367	(1)
Insufficient, or Slow Change	368	(2)
Satisfactory Change	370	(2)
The Termination and Follow-Up Phase	372	(3)
Program-Level Decision Making	375	(4)
Process	376	(1)
Outcome	376	(1)
Problems and Cases	376	(2)
Program	378	(1)
Agency	379	(1)
Using Outcome Monitoring Data in Program-Level Decision Making	379	(2)
Acceptable Results	380	(1)
Mixed Results	380	(1)
Inadequate Results	381	(1)
Summing Up and Looking Ahead	381	(1)
Recap and Online Materials	382	(1)
Study Questions	382	(2)
References, Further Reading, and Resources	384	(5)
Part IV Knowing the Contexts of Evaluations
Evaluation Politics, Ethics, and Standards	389	(28)
Politics of Evaluation	389	(1)
Appropriate and Inappropriate Uses of Evaluation	390	(4)
Misuses of Evaluation	390	(1)
Justifying Decisions Already Made	390	(1)
Inappropriate Use of Public Relations	391	(1)
Used for Performance Appraisals	391	(1)
Fulfilling Funding Requirements	392	(1)
Proper Uses of Evaluation	392	(1)
Internal Decision Making	392	(1)
External Decision Making	393	(1)
Political Influences on the Evaluation Process	394	(4)
Manipulating the Evaluation Process	394	(1)
Misdirecting the Evaluation Process	395	(1)
Program Objectives	395	(1)
Sample Selection	396	(1)
Data Collection Methods	397	(1)
Interpretation of Findings	398	(1)
Professional Standards for Evaluation	398	(4)
Utility	400	(1)
Feasibility	401	(1)
Propriety	401	(1)
Accuracy	402	(1)
Principles of Evaluation Practice	402	(9)
Principle 1: Evaluation and Service Delivery Activities Should Be Integrated	406	(3)
Principle 2: Involve From the Beginning as Many Stakeholder Groups as Possible	409	(1)
Principle 3: Involve All Levels of Staff in the Evaluation Process	409	(1)
Principle 4: Make Explicit the Purpose of the Evaluation	409	(1)
Principle 5: Provide a Balanced Report and Disseminate Early and Regularly	410	(1)
Summing Up and Looking Ahead	411	(1)
Recap and Online Materials	411	(1)
Study Questions	411	(2)
References, Further Reading, and Resources	413	(4)
Culturally Appropriate Evaluations	417	(20)
The Impact of Culture	418	(1)
Bridging the Culture Gap	419	(3)
Cultural Awareness	419	(1)
Intercultural Communication	420	(1)
Nonverbal	421	(1)
Verbal	421	(1)
Cultural Frameworks	422	(3)
Orientation to Information	422	(1)
Decision Making	422	(1)
Individualism	423	(1)
Tradition	423	(1)
Pace of Life	424	(1)
Putting It Together: The Practice of Culturally Competent Evaluation	425	(5)
Cultural Awareness	425	(1)
Intercultural Communication Skills	426	(1)
Developing Specific Knowledge About the Culture	426	(1)
Adapting Evaluations	427	(1)
Working with Stakeholders	427	(1)
Adapting Processes	428	(1)
Providing Meaningful Products	429	(1)
Summing Up and Looking Ahead	430	(1)
Recap and Online Materials	431	(1)
Study Questions	431	(1)
References, Further Reading, and Resources	432	(5)
Writing Grant Proposals	437	(16)
Gathering Background Information	438	(1)
Program Concept	438	(1)
Your Program	438	(1)
Program Expenses	439	(1)
Components of a Proposal	439	(8)
Executive Summary	439	(1)
Statement of Need	440	(1)
Program Description	441	(1)
Objectives	442	(1)
Methods, or Interventions	443	(1)
Staffing/Administration	443	(1)
Evaluation	444	(1)
Sustainability	444	(1)
Budget	445	(1)
Expense Budget	445	(1)
Support and Revenue and Statement	446	(1)
Budget Narrative	446	(1)
Organizational Information	447	(1)
Conclusion	447	(1)
Letter Proposal	447	(2)
What Happens Next?	449	(1)
Summing Up	449	(1)
Recap and Online Materials	450	(1)
Study Questions	450	(1)
References, Further Reading, and Resources	451	(2)
Credits	453	(4)
Index	457