Generalizability Theory

by Brennan, Robert L.; Fienberg, S.; Lievesley, D.; Rolph, J.

• ISBN13: 9780387952826
• ISBN10: 0387952829
• Format: Hardcover
• Copyright: 2001-10-01
• Publisher: Springer Verlag

Summary

Generalizability theory offers an extensive conceptual framework and a powerful set of statistical procedures for characterizing and quantifying the fallibility of measurements. It liberalizes classical test theory, in part through the application of analysis of variance procedures that focus on variance components. As such, generalizability theory is perhaps the most broadly defined measurement model currently in existence. It is applicable to virtually any scientific field that attends to measurements and their errors, and it enables a multifacteted perspective on measurement error and its components. This book provides the most comprehensive and up-to-date treatment of generalizability theory. In addition, it provides a synthesis of those parts of the statistical literature that are directly applicable to generalizability theory. The principal intended audience is measurement practitioners and graduate students in the behavioral and social sciences, although a few examples and references are provided from other fields. Readers will benefit from some familiarity with classical test theory and analysis of variance, but the treatment of most topics does not presume specific background. Robert L. Brennan is E.F. Lindquist Professor of Educational Measurement at the University of Iowa. He is an acknowledged expert in generalizability theory, has authored numerous publications on the theory, and has taught many courses and workshops on generalizability. The author has been Vice-President of the American Educational Research Association and President of the National Council on Measurement in Education (NCME). He has received NCME Awards for Outstanding Technical Contributions to Educational Measurement and Career Contributions to Educational Measurement.

Author Biography

Robert L. Brennan is E.F. Lindquist Professor of Educational Measurement and Director of Iowa Testing Programs at the University of Iowa.

Table of Contents

Preface	vii
Principal Notational Conventions	xix
Introduction	1	(20)
Framework of Generalizability Theory	4	(14)
Universe of Admissible Observations and G Studies	5	(3)
Infinite Universe of Generalization and D Studies	8	(5)
Different Designs and/or Universes of Generalization	13	(4)
Other Issues and Applications	17	(1)
Overview of Book	18	(1)
Exercises	19	(2)
Single-Facet Designs	21	(32)
G Study p x i Design	22	(2)
G Study Variance Components for p x i Design	24	(5)
Estimating Variance Components	25	(3)
Synthetic Data Example	28	(1)
D Studies for the p x I Design	29	(10)
Error Variances	31	(3)
Coefficients	34	(1)
Synthetic Data Example	35	(2)
Real-Data Examples	37	(2)
Nested Designs	39	(6)
Nesting in Both the G and D Studies	40	(4)
Nesting in the D Study, Only	44	(1)
Summary and Other Issues	45	(5)
Other Indices and Coefficients	47	(1)
Total Score Metric	48	(2)
Exercises	50	(3)
Multifacet Universes of Admissible Observations and G Study Designs	53	(42)
Two-Facet Universes and Designs	54	(6)
Venn Diagrams	54	(3)
Illustrative Designs and Universes	57	(3)
Linear Models, Score Effects, and Mean Scores	60	(7)
Notational System for Main and Interaction Effects	61	(2)
Linear Models	63	(3)
Expressing a Score Effect in Terms of Mean Scores	66	(1)
Typical ANOVA Computations	67	(7)
Observed Mean Scores and Score Effects	67	(2)
Sums of Squares and Mean Squares	69	(1)
Synthetic Data Examples	70	(4)
Random Effects Variance Components	74	(11)
Defining and Interpreting Variance Components	74	(2)
Expected Mean Squares	76	(3)
Estimating Variance Components Using the EMS Procedure	79	(1)
Estimating Variance Components Directly from Mean Squares	80	(3)
Synthetic Data Examples	83	(1)
Negative Estimates of Variance Components	84	(1)
Variance Components for Other Models	85	(7)
Model Restrictions and Definitions of Variance Components	86	(3)
Expected Mean Squares	89	(1)
Obtaining σ2(α\M) from σ2(α)	89	(1)
Example: APL Program	90	(2)
Exercises	92	(3)
Multifacet Universes of Generalization and D Study Designs	95	(46)
Random Models and Infinite Universes of Generalization	96	(14)
Universe Score and Its Variance	97	(3)
D Study Variance Components	100	(1)
Error Variances	100	(4)
Coefficients and Signal-Noise Ratios	104	(3)
Venn Diagrams	107	(1)
Rules and Equations for Any Object of Measurement	108	(1)
D Study Design Structures Different from the G Study	109	(1)
Random Model Examples	110	(10)
p x I x O Design with Synthetic Data Set No. 3	110	(3)
p x (I:O) Design with Synthetic Data Set No. 3	113	(2)
p x (R:T) Design with Synthetic Data Set No. 4	115	(2)
Performance Assessments	117	(3)
Simplified Procedures for Mixed Models	120	(5)
Rules	121	(1)
Venn Diagrams	122	(3)
Mixed Model Examples	125	(10)
p x I x O Design with Items Fixed	125	(1)
p x (R:T) Design with Tasks Fixed	125	(2)
Perspectives on Traditional Reliability Coefficients	127	(3)
Generalizability of Class Means	130	(2)
A Perspective on Validity	132	(3)
Summary and Other Issues	135	(1)
Exercises	136	(5)
Advanced Topics in Univariate Generalizability Theory	141	(38)
General Procedures for D Studies	141	(12)
D Study Variance Components	142	(2)
Universe Score Variance and Error Variances	144	(1)
Examples	145	(4)
Hidden Facets	149	(4)
Stratified Objects of Measurement	153	(4)
Relationships Among Variance Components	155	(1)
Comments	156	(1)
Conventional Wisdom About Group Means	157	(2)
Two Random Facets	157	(2)
One Random Facet	159	(1)
Conditional Standard Errors of Measurement	159	(6)
Single-Facet Designs	160	(4)
Multifacet Random Designs	164	(1)
Other Issues	165	(10)
Covariances as Estimators of Variance Components	166	(2)
Estimators of Universe Scores	168	(3)
Random Sampling Assumptions	171	(3)
Generalizability and Other Theories	174	(1)
Exercises	175	(4)
Variability of Statistics in Generalizability Theory	179	(36)
Standard Errors of Estimated Variance Components	180	(10)
Normal Procedure	181	(1)
Jackknife Procedure	182	(3)
Bootstrap Procedure	185	(5)
Confidence Intervals for Estimated Variance Components	190	(6)
Normal Procedure	190	(1)
Satterthwaite Procedure	190	(1)
Ting et al. Procedure	191	(4)
Jackknife Procedure	195	(1)
Bootstrap Procedure	196	(1)
Variability of D Study Statistics	196	(5)
Absolute Error Variance	197	(1)
Feldt Confidence Interval for Eρ2	198	(2)
Arteaga et al. Confidence Interval for φ	200	(1)
Some Simulation Studies	201	(7)
G Study Variance Components	201	(4)
D Study Statistics	205	(3)
Discussion and Other Issues	208	(3)
Exercises	211	(4)
Unbalanced Random Effects Designs	215	(34)
G Study Issues	216	(11)
Analogous-ANOVA Procedure	217	(3)
Unbalanced i:p Design	220	(2)
Unbalanced p x (i:h) Design	222	(3)
Missing Data in the p x i Design	225	(2)
D Study Issues	227	(13)
Unbalanced I:p Design	228	(3)
Unbalanced p x (I:H) Design	231	(2)
Unbalanced (P:c) x I Design	233	(2)
Missing Data in the p x I Design	235	(2)
Metric Matters	237	(3)
Other Topics	240	(7)
Estimation Procedures	241	(4)
Computer Programs	245	(2)
Exercises	247	(2)
Unbalanced Random Effects Designs-Examples	249	(18)
ACT Science Reasoning	249	(2)
District Means for ITED Vocabulary	251	(6)
Clinical Clerkship Performance	257	(5)
Testlets	262	(3)
Exercises	265	(2)
Multivariate G Studies	267	(34)
Introduction	268	(5)
G Study Designs	273	(11)
Single-Facet Designs	275	(3)
Two-Facet Crossed Designs	278	(3)
Two-Facet Nested Designs	281	(3)
Defining Covariance Components	284	(2)
Estimating Covariance Components for Balanced Designs	286	(9)
An Illustrative Derivation	287	(2)
General Equations	289	(4)
Standard Errors of Estimated Covariance Components	293	(2)
Discussion and Other Topics	295	(3)
Interpreting Covariance Components	295	(2)
Computer Programs	297	(1)
Exercises	298	(3)
Multivariate D Studies	301	(46)
Universes of Generalization and D Studies	301	(9)
Variance-Covariance Matrices for D Study Effects	302	(1)
Variance-Covariance Matrices for Universe Scores and Errors of Measurement	303	(2)
Composites and A Priori Weights	305	(1)
Composites and Effective Weights	306	(1)
Composites and Estimation Weights	307	(1)
Synthetic Data Example	308	(2)
Other Topics	310	(18)
Standard Errors of Estimated Covariance Components	310	(2)
Optimality Issues	312	(2)
Conditional Standard Errors of Measurement for Composites	314	(3)
Profiles and Overlapping Confidence Intervals	317	(3)
Expected Within-Person Profile Variability	320	(4)
Hidden Facets	324	(2)
Collapsed Fixed Facets in Multivariate Analyses	326	(2)
Computer Programs	328	(1)
Examples	328	(15)
ACT Assessment Mathematics	328	(6)
Painting Assessment	334	(5)
Listening and Writing Assessment	339	(4)
Exercises	343	(4)
Multivariate Unbalanced Designs	347	(44)
Estimating G Study Covariance Components	347	(20)
Observed Covariance	349	(1)
Analogous TP Terms	349	(4)
CP Terms	353	(6)
Compound Means	359	(1)
Variance of a Sum	360	(3)
Missing Data	363	(3)
Choosing a Procedure	366	(1)
Examples of G Study and D Study Issues	367	(21)
ITBS Maps and Diagrams Test	367	(6)
ITED Literary Materials Test	373	(7)
District Mean Difference Scores	380	(8)
Discussion and Other Topics	388	(1)
Exercises	388	(3)
Multivariate Regressed Scores	391	(40)
Multiple Linear Regression	392	(3)
Estimating Profiles Through Regression	395	(20)
Two Independent Variables	395	(6)
Synthetic Data Example	401	(3)
Variances and Covariances of Regressed Scores	404	(3)
Standard Errors of Estimate and Tolerance Intervals	407	(2)
Different Sample Sizes and/or Designs	409	(3)
Expected Within-Person Profile Variability	412	(3)
Predicted Composites	415	(11)
Difference Scores	417	(2)
Synthetic Data Example	419	(2)
Different Sample Sizes and/or Designs	421	(1)
Relationships with Estimated Profiles	422	(2)
Other Issues	424	(2)
Comments	426	(1)
Exercises	427	(4)
Appendices	431	(76)
A. Degrees of Freedom and Sums of Squares for Selected Balanced Designs	431	(4)
B. Expected Mean Squares and Estimators of Random Effects Variance Components for Selected Balanced Designs	435	(4)
C. Matrix Procedures for Estimating Variance Components and Their Variability	439	(6)
D. Table for Simplified Use of Satterthwaite's Procedure	445	(4)
E. Formulas for Selected Unbalanced Random Effects Designs	449	(4)
F. Mini-Manual for GENOVA	453	(18)
G. urGENOVA	471	(2)
H. mGENOVA	473	(2)
I. Answers to Selected Exercises	475	(32)
References	507	(14)
Author Index	521	(4)
Subject Index	525