Ensuring the Quality and Productivity of Education and Professional Development Activities A Review of Approaches and Lessons for DoD

by Gates, Susan M.; Augustine, Catherine H.; Benjamin, Roger; Bikson, Tora K.; Derghazarian, Eric

• ISBN13: 9780833029874
• ISBN10: 0833029878
• Format: Paperback
• Copyright: 2001-08-30
• Publisher: RAND Corporation

Summary

With some 700,000 civilian employees, the Department of Defense is the single largest employer of civil service workers in the U.S. government. The Office of the Chancellor for Education and Professional Development in the Department of Defense (DoD) is charged with ensuring the quality and productivity of education and professional development activities targeted at DoD civilians. At the request of the Chancellor's office, RAND undertook a study to examine the approaches used to evaluate academic quality and productivity in a variety of postsecondary education and training contexts, including corporations, state governments, and universities. The study then considered which approaches might be most relevant to the Chancellor1s office.

Table of Contents

Preface	iii
Figures	ix
Tables	xi
Summary	xiii
Acknowledgments	xix
Acronyms	xxi
Introduction	1	(6)
Background	1	(1)
Motivation	2	(1)
Objectives and Approach	3	(2)
Organization of the Report	5	(2)
Framework	7	(6)
The System of DoD Civilian Education and Professional Development	8	(3)
Framework for Analyzing the Assessment of System Performance	11	(2)
Phase One: System-Level Assessment	13	(10)
Goals of System-Level Assessment	13	(2)
Identifying Misalignment	13	(1)
Optimizing Resource Allocation at the System Level	14	(1)
How Systems Establish a Structure for Such Assessment	15	(1)
How Systems Identify Misalignments	16	(2)
How Systems Allocate Resources	18	(1)
Need for Standardized Data and Course Offerings	19	(1)
Beyond Assessment: Promoting Workforce Improvement	20	(1)
Multiple Benefits	21	(1)
Lessons for the Chancellor's Office	21	(2)
Phase Two: Assessing How Well Providers Meet Customer Needs	23	(21)
Model 1: Intermediary Assesses or Guides Provider's Process of Assessment	24	(4)
ISO 9000	25	(1)
The Academic Audit	26	(2)
Model 2: Intermediary Conducts the Assessment	28	(6)
State Higher Education Governing and Coordinating Boards	29	(1)
U.S. News and World Report	30	(1)
Accrediting Agencies	31	(1)
Malcolm Baldrige National Quality Award	32	(1)
The Problem of Compliance	32	(2)
Model 3: Provider Conducts the Assessment	34	(3)
The University of Phoenix	35	(1)
The Urban Universities Portfolio Project	36	(1)
Model 4: Student Competencies Are Assessed	37	(3)
Microsoft's Technical Certification Programs	38	(1)
The Department of Labors SCANS Initiative	39	(1)
Western Governors University	39	(1)
Strengths and Weaknesses of the Four Approaches	40	(4)
Choosing the Right Model for Phase Two	44	(11)
Purposes of Assessment	44	(3)
Level of Authority	47	(1)
Level of Resources	48	(1)
Centralization of Operations	49	(1)
System Heterogeneity	50	(2)
System Complexity	52	(1)
Summary	53	(1)
Implications for the Chancellor's Office	53	(2)
Three Steps for Assessing Providers	55	(20)
Step One: Identifying Goals	56	(3)
Goal Setting Guides the Assessment Process	56	(1)
How to Set Goals	56	(1)
Level of Stakeholder Involvement	57	(2)
Step Two: Selecting Measures	59	(5)
Input Measures	59	(1)
Process Measures	60	(1)
Outcome Measures	61	(1)
Choosing Measures	61	(3)
Step Three: Evaluate Performance Using Measures	64	(3)
Comparison with External Peers	64	(1)
Comparison with Preset Standards	65	(1)
Comparison with Internal Peers	66	(1)
Comparison with Past Performance	66	(1)
Measurement Validity and Reliability	67	(4)
Validity	67	(2)
Reliability	69	(1)
Implications	70	(1)
Bringing It All Together: Integrating All Three Steps	71	(2)
Relevance of the Three Assessment Steps for the Chancellor's Office	73	(2)
Conclusions and Recommendations	75	(140)
Phase One: DoD Should Devote Attention to the First Phase of Assessment	75	(1)
Phase Two: Recommendations	76	(3)
Consider the Purpose of Assessment	77	(1)
Consider Constraints Within the DoD Education and Professional Development System	77	(1)
Integrate the Three Assessment Steps	78	(1)
Appendix
A. Overview of Appendices	79	(3)
B. State Higher Education Boards	82	(18)
B.1. Texas Higher Education Coordinating Board	88	(7)
B.2. Kentucky Council on Postsecondary Education	95	(5)
C. Process Auditors-Academic Audit	100	(11)
C.1. International Organization for Standardization	106	(5)
D. Accrediting Agencies	111	(19)
D.1. Western Association of Schools and Colleges	123	(7)
E. Professional Societies	130	(21)
F.1. U.S. Department of Transportation	133	(6)
F.2. U.S. Air Force Training and Education	139	(6)
F.3. U.S. Navy Individual and Mission Training	145	(6)
G. Corporate Professional Development and Training	151	(32)
G.1. Lucent Technologies Learning and Performance Center	162	(6)
H.1. U.S. News and World Report	168	(7)
I.1. Baldrige Award	175	(8)
J. Certifiers of Student Competencies	183	(32)
K.1. University of Phoenix	190	(6)
K.2. U.S. Air Force Academy, Department of Management	196	(6)
K.3. Balanced Scorecard	202	(8)
K.4. The Urban Universities Portfolio Project: Assuring Quality for Multiple Publics	210	(5)
Bibliography	215