Measures of Interobserver Agreement and Reliability

by Shoukri, Mohamed M.

• ISBN13: 9781584883210
• ISBN10: 1584883219
• Edition: 1st
• Format: Hardcover
• Copyright: 2003-07-28
• Publisher: CRC Press

Summary

Agreement among at least two evaluators is an issue of prime importance to statisticians, clinicians, epidemiologists, psychologists, and many other scientists. Measuring interobserver agreement is a method used to evaluate inconsistencies in findings from different evaluators who collect the same or similar information. Highlighting applications over theory, Measure of Interobserver Agreement provides a comprehensive survey of this method and includes standards and directions on how to run sound reliability and agreement studies in clinical settings and other types of investigations.The author clearly explains how to reduce measurement error, presents numerous practical examples of the interobserver agreement approach, and emphasizes measures of agreement among raters for categorical assessments. The models and methods are considered in two different but closely related contexts: 1) assessing agreement among several raters where the response variable is continuous and 2) where there is a prior decision by the investigators to use categorical scales to judge the subjects enrolled in the study. While the author thoroughly discusses the practical and theoretical issues of case 1, a major portion of this book is devoted to case 2. He explores issues such as two raters randomly judging a group of subjects, interrater bias and its connection to marginal homogeneity, and statistical issues in determining sample size.Statistical analysis of real and hypothetical datasets are presented to demonstrate the various applications of the models in repeatability and validation studies. To help with problem solving, the monograph includes SAS code, both within the book and on the CRC Web site. The author presents information with the right amount mathematical details, making this a cohesive book that reflects new research and the latest developments in the field.

Author Biography

Mohammed M. Shoukri is the Deputy Chairman of the Department of Biostatistics, Epidemiology, and Scientific Computing at King Faisal Specialist Hospital, Riyadh, Saudi Arabia and Research Center and Adjunct Professor of Biostatistics at the University of Western Ontario, Canada.

Table of Contents

Chapter 1 Introduction	1	(4)
Chapter 2 Reliability for continuous scale measurements	5	(18)
2.1 Models for reliability studies	6	(4)
2.2 Inference procedures on the index of reliability for case (1)	10	(1)
2.2.1 Interval estimation	10	(1)
2.2.2 Hypothesis testing	11	(1)
2.3 Analysis of method - comparison studies	11	(8)
2.3.1 Assessing agreement between two methods	11	(3)
2.3.2 An index of agreement: concordance correlation	14	(1)
2.3.3 Estimating relative precision	15	(3)
2.3.3.1 Point estimation	15	(2)
2.3.3.2 Confidence limits on precision parameters	17	(1)
2.3.3.3 Remarks	18	(1)
2.3.4 Testing for bias and precision "simultaneously"	18	(1)
2.3.5 workshop	18	(1)
2.3.5.1 Exercise 1	18	(1)
2.4 Comparing reliability coefficients	19	(4)
2.4.1 Case 1: independent experiments	19	(1)
2.4.2 Case 2: dependent measurements	20	(3)
Chapter 3 Measures of 2 x 2 association and agreement of cross classified data	23	(16)
3.1 Introduction	23	(1)
3.2 Indices of Adjusted Agreement	24	(2)
3.3 Cohen's kappa: chance-corrected measure of agreement	26	(1)
3.4 Intraclass kappa	27	(3)
3.5 The 2 x 2 kappa in the context of association	30	(2)
3.6 Stratified kappa	32	(3)
3.7 Conceptual issues	35	(4)
Chapter 4 Coefficients of agreement for multiple raters and multiple categories	39	(24)
4.1 Introduction	39	(1)
4.2 Multiple categories and two raters	39	(8)
4.2.1 Category distinguishability	39	(4)
4.2.2 Test for interrater bias	43	(4)
4.3 Agreement for multiple raters and dichotomous classification	47	(6)
4.3.1 Exchangeable ratings	47	(2)
4.3.2 Test for interrater bias	49	(2)
4.3.3 Probability models	51	(2)
4.4 Multiple raters and multiple categories	53	(4)
4.5 Testing the homogeneity of kappa statistics from independent studies	57	(6)
Chapter 5 Assessing agreement from dependent data	63	(22)
5.1 Introduction	63	(1)
5.2 Dependent dichotomous assessments	64	(7)
5.2.1 Pooled estimate of interrater agreement	64	(2)
5.2.2 Comparing dependent interrater agreements: Probability models	66	(5)
5.3 Adjusting for covariates	71	(1)
5.3.1 Logistic modeling	72	(2)
5.4 Likelihood-based approach	74	(4)
5.5 Estimating equations approach	78	(1)
5.6 Log-linear and association models	78	(3)
Appendix I	81	(2)
Appendix II	83	(2)
Chapter 6 Sample size requirements for the design of a reliability study	85	(18)
6.1 Introduction	85	(1)
6.2 The case of continuous measurements	85	(6)
6.2.1 Power considerations	85	(3)
6.2.2 Fixed length of a confidence interval	88	(1)
6.2.3 Efficiency requirements	89	(2)
6.3 The nonnormal case	91	(1)
6.4 Cost implications	92	(3)
6.5 The case of dichotomous assessments	95	(8)
6.5.1 Power considerations	95	(1)
6.5.2 Fixed length of a confidence interval	96	(1)
6.5.3 Efficiency requirements	97	(6)
Chapter 7 Workshops	103	(40)
7.1 Workshop #1	103	(6)
7.1.1 SAS output for workshop #1	107	(2)
7.2 Workshop #2	109	(4)
7.2.1 SAS output for workshop #2	111	(2)
7.3 Workshop #3.1	113	(2)
7.3.1 SAS output for workshop #3.1	114	(1)
7.4 Workshop #3.2	115	(6)
7.4.1 SAS output for workshop #3.2	117	(4)
7.5 Workshop #3.3	121	(5)
7.5.1 SAS output for workshop #3.3	124	(2)
7.5.1.1 Fitting the two-way random model	125	(1)
7.6 Workshop #4.1	126	(3)
7.6.1 SAS output for workshop #4.1	127	(2)
7.7 Workshop #4.2	129	(2)
7.7.1 SAS output for workshop #4.2	130	(1)
7.8 Workshop #5.1	131	(6)
7.8.1 SAS output for workshop #5.1	132	(5)
7.9 Workshop #5.2	137	(6)
7.9.1 SAS output for workshop #5.2	139	(4)
Bibliography	143	(6)
Index	149