Isolation of Results : Defining the Impact of the Program

by Phillips, Jack J.; Aaron, Bruce C.

• ISBN13: 9780787987190
• ISBN10: 0787987190
• Format: Paperback
• Copyright: 2008-02-08
• Publisher: Pfeiffer & Co

Summary

Isolation of Results Isolation of Results is the third of six books in the Measurement and Evaluation Series from Pfeiffer. The proven ROI Methodology-developed by the ROI Institute-provides a practical system for evaluation planning, data collection, data analysis, and reporting. All six books in the series offer the latest tools, most current research, and practical advice for measuring ROI in a variety of settings.To bring credibility to a project it is vital to isolate the effects of a program on business data. Isolation of Results focuses on this critical topic, arguably the most valuable part of the ROI Methodology. The authors acknowledge that other factors can influence results, and this important resource shows a variety of ways in which the effects of the program can be isolated from other influences. The techniques presented include the most reliable and rigorous approaches and also contain the more subjective methods. Isolation of Results offers a comprehensive review of the topic and contains information about using control group arrangements, trend line analysis, forecasting, expert estimation, and adjustments. The book also addresses issues of credibility of the method.

Author Biography

Jack J. Phillips is chairman of the ROI Institute, Inc. A world-renowned expert on measurement and evaluation, Phillips provides consulting services to Fortune 500 companies and workshops for major conference providers throughout the world. Phillips is the author or editor of more than thirty books and more than one hundred articles.

Bruce C. Aaron is responsible for research and evaluation of learning and knowledge management initiatives within Accenture, a global management consulting, technology services, and outsourcing company. During his tenure, Accenture Education has received several awards from professional associations for excellence in measurement and evaluation, including the 2004 ASTD ROI Impact

Study of the Year award.

The ROI Institute, Inc., is a benchmarking, research, and information sharing organization that provides consulting services, workshops, and certification in the ROI Methodology. Widely considered the leading authority on evaluation and measurement of learning and development in organizations, the ROI Institute conducts workshops and offers certification for thousands of practitioners through a variety of strategic partners.

Table of Contents

Acknowledgments

Principles of the ROI Methodology

The Importance of Isolating the Effects of Programs

Challenges in Understanding a Program's Impact

Case Study: What Caused the Improvement?

Preliminary Issues in Isolating Program Effects

Pager: Please do not italicize the Contents H1 items or the sublists to this level

The Need to Isolate Program Effects

Chain of Impact: Initial Evidence of Program Effects

Identification of Factors Other Than the Program: A First Step

Final Thoughts

Use of Control Groups

Control Group Design

Threats to Validity

Basic Control Group Design

Ideal Experimental Design

Posttest-Only Control Group Design

Which Design to Choose

Issues When Considering Control Groups

Viability

Practicality

Ethicial Considerations

Problems with Control Groups: A Case Example

Feasibility

Control Group Example 1: Retail Merchandise Company

Setting

Audience

Solution

Measures That Matter

Selection Criteria

Size of Groups

Duration of Experiment

Control Group Example 2: Federal Information Agency

Setting

Audience

Solution

Measures That Matter

Selection Criteria

Size of Groups

Duration of Experiment

Control Group Example 3: Midwest Electric, Inc

Setting

Needs Assessment

Audience

Solution

Measures That Matter

Selection Criteria

Size of Groups

Duration of Experiment

Control Group Example 4: International Software Company

Setting

Audience

Solution

Measures That Matter

Selection Criteria

Size of Groups

Duration of Experiment

Final Thoughts

Use of Trend Lines and Forecasts

Trend Lines

Forecasts

Trend Line Analysis Example 1: Micro Electronics

Setting

Audience

Solution

Measures That Matter

Conditions Test

Trend Line Analysis Example 2: Healthcare, Inc

Setting

Audience

Solution

Measures That Matter

Conditions Test

Trend Line Analysis Example 3: National Book Company

Setting

Audience

Solution

Measures That Matter

Conditions Test

Final Thoughts

Use of Expert Estimates

Participants' Estimates of Program Impact

Using Focus Groups to Obtain Participant Estimates

Using Questionnaires to Obtain Participant Estimates

Using Interviews to Obtain Participant Estimates

Advantages and Disadvantages of Participant Estimates

Case Study

Pager: Please style the following items as a sublist to the previous list

Setting

Audience

Solution

Measures

Estimates Provided

Credibility Check

Methodology

Pager: End of sublist

Immediate Managers' Estimates of Program Impact

Senior Management's Estimates of Program Impact

Customers? Estimates of Program Impact

Experts' Estimates of Program Impact

Determining the Impact of Other Factors

Estimate Example 1: Global Financial Services

Setting

Audience and Solution

Measures

Estimates Provided

Estimate Example 2: Cracker Box

Setting

Audience and Solution

Measures

Estimates Provided

Estimate Example 3: Public Bank of Malaysia

Setting

Audience and Solution

Measure

Estimates Provided

Estimate Example 4: Multi-National, Inc

Setting

Audience, Solution, and Measures

Estimates Provided

The Power of Estimates

Research

A Demonstration

Participant Reaction

Management Reaction

The Wisdom of Crowds

Key Is

Table of Contents provided by Publisher. All Rights Reserved.