Improving the Army's Assessment of Interactive Multimedia Instruction Courseware (2009)

by Straus, Susan G.; Shanley, Michael G.; Burns, Rachel M.; Waite, Anisah; Crowley, James C.

• ISBN13: 9780833047274
• ISBN10: 0833047272
• Format: Paperback
• Copyright: 2009-09-16
• Publisher: RAND Corporation

Summary

An important component of The Army Distributed Learning Program’s (TADLP’s) performance is the quality of its courses, which consist primarily of asynchronous interactive multimedia instruction (IMI). However, there are no current efforts to assess course quality at the program level. This report outlines a comprehensive approach to evaluating the quality of IMI within TADLP. In addition, it describes how a program-level evaluation program developed at RAND might be implemented within the U.S. Army Training and Doctrine Command (TRADOC). RAND assessed the quality of the learning experience in a sample of recently fielded IMI courses, using criteria based on standards in the training development community. The analysis identifies strengths and deficiencies in technical, production quality, and pedagogical aspects of IMI courseware. This research demonstrates a feasible method for evaluation that can supply TRADOC with metrics concerning IMI quality, points to directions for needed improvements, and provides a basis for assessing improvement initiatives. The authors recommend adoption of this approach, as well as further development of other components of a comprehensive program of IMI training evaluation.

Table of Contents

Preface	p. iii
Tables	p. vii
Summary	p. ix
Acknowledgments	p. xxi
Acronyms	p. xxiii
Introduction	p. 1
Background	p. 1
Purpose and Organization of This Report	p. 3
Quality-Evaluation Approaches and Their Usage in TADLP	p. 5
Approaches to Evaluating Training Quality	p. 5
Training Outcomes	p. 5
Test Evaluation	p. 7
Administrative Data	p. 7
Courseware Evaluation	p. 7
Current Evaluation Efforts in Army Training	p. 8
Measures	p. 9
Conclusion	p. 12
RAND's Approach to IMI Evaluation	p. 13
Courses Evaluated	p. 13
Evaluation Criteria	p. 15
Coding Process	p. 22
IMI Evaluation Findings	p. 23
Interactivity and Course Length	p. 23
Technical Criteria for Courseware	p. 24
Production-Quality Criteria for Courseware	p. 26
Pedagogical Criteria for Courseware	p. 28
Conclusions and Implications for TADLP's Assessment of IMI	p. 41
How IMI Courseware Can Be Improved	p. 41
Recommendations for Courses in Our Sample	p. 41
Strategic Issues in Instructional Design	p. 44
Recommendations for Improving TADLP's Assessment Program	p. 46
Implementing a Program of Courseware Evaluation in TADLP	p. 49
Potential Directions for Other Program-Level Evaluations of IMI Quality	p. 51
Conclusion	p. 54
Appendix
Courseware Evaluation Criteria	p. 57
Summary of Specific Changes for Improving the Quality of IMI Courseware	p. 63
References	p. 65
Table of Contents provided by Ingram. All Rights Reserved.