Multiple Comparisons and Multiple Tests Using the SAS System

by Westfall, Peter H.; Tobias, Randall D.; Rom, Dror; Wolfinger, Russell D.; Hochberg, Yosef

• ISBN13: 9781580253970
• ISBN10: 1580253970
• Format: Paperback
• Copyright: 1999-07-01
• Publisher: Sas Inst

Summary

Your work involves multiple comparison procedures...

Author Biography

Peter H. Westfall, Ph.D., Professor of Statistics, Department of Information Systems and Quantitative Sciences, Texas Tech University Randall D. Tobias, Ph.D., principal developer for the GLM procedure and manager of linear models research and development at SAS Institute Dror Rom, Ph.D., president of Prosoft Software, Inc. Russell D. Wolfinger, Ph.D., Senior Research Statistician at SAS Institute Yosef Hochberg, Ph.D., professor in the School of Mathematical Sciences at Tel Aviv University

Table of Contents

Preface	p. xi
Introduction	p. 1
The Multiplicity Problem	p. 1
Examples of Multiplicity in Practice	p. 2
Multiple Comparisons in a Marketing Experiment	p. 2
Subgroup Analysis in a Clinical Trial	p. 2
Analysis of a Sociological Survey	p. 3
An Epidemiology Example: Data Snooping	p. 3
Industrial Experimentation and Engineering	p. 4
Identifying Clinical Practice Improvement Opportunities for Hospital Surgeries	p. 4
When Are Multiple Comparisons/Multiple Testing Methods (MCPs) Needed?	p. 5
Selecting an MCP Using This Book	p. 6
Statistical Modeling Assumptions	p. 6
Multiple Comparisons/Multiple Testing Objectives	p. 7
The Set (Family) of Elements to Be Tested	p. 9
Controversial Aspects of MCPs	p. 9
Size of the Family	p. 10
Composite Inferences vs. Individual Inferences	p. 10
Bayesian Methods	p. 11
Concepts and Basic Methods for Multiple Comparisons and Tests	p. 13
Introduction	p. 13
Families of Hypotheses or Inferences	p. 16
Error Rates	p. 16
Comparisonwise Error Rate (CER)	p. 16
Familywise Error Rate (FWE)	p. 18
Control of the FWE: Weak and Strong	p. 19
Directional Decisions and (Type III) Error rates	p. 20
False Discovery Rate	p. 21
Bonferroni and Sidak Methods	p. 22
Adjusted p-Values	p. 23
An Example with Multiple p-Values	p. 23
Example: Multiple Comparisons from a Questionnaire	p. 24
Sequentially Rejective Methods	p. 29
Bonferroni-Holm Method	p. 29
Sidak-Holm Method	p. 31
Simes' Modified Bonferroni Procedure	p. 32
Hommel's Procedure	p. 32
Hochberg's Method--A Step-Up Test	p. 32
Rom's Method	p. 33
Benjamini and Hochberg's FDR-Controlling Method	p. 34
Sequential Testing with Fixed Sequences	p. 35
Graphical Presentation of Multiple Testing Results	p. 37
The Schweder-Spjotvoll p-Value Plot	p. 37
A Testing Procedure for Nominal FWE Protection	p. 38
Concluding Remarks	p. 40
Multiple Comparisons among Treatment Means in the One-Way Balanced ANOVA	p. 41
Introduction	p. 41
Overview of Methods	p. 42
The Model and Estimates	p. 42
Simultaneous Confidence Intervals	p. 44
All Pairwise Comparisons	p. 44
Example of Pairwise Comparisons with Simultaneous Confidence Intervals	p. 46
Displaying Pairwise Comparisons Graphically	p. 50
Simultaneous Tests of Hypotheses	p. 51
Pairwise Comparisons with a Control	p. 53
Two-Sided Comparisons with a Control	p. 54
One-Sided Comparisons with a Control	p. 57
Multiple Inferences for Independent Estimates	p. 60
Simultaneous Intervals for the Treatment Means	p. 60
Orthogonal Comparisons	p. 62
Concluding Remarks	p. 64
Multiple Comparisons among Treatment Means in the One-Way Unbalanced ANOVA	p. 65
Introduction	p. 65
All Pairwise Comparisons	p. 66
The Tukey-Kramer Method	p. 67
Graphical Comparisons - LINES Option	p. 69
Simulation-based Methods	p. 70
Tests of Hypotheses--Adjusted p-Values	p. 73
Pairwise Comparisons with Control	p. 76
Distributions	p. 76
Two-Sided Comparisons	p. 78
One-Sided Comparisons	p. 79
Concluding Remarks	p. 80
Multiple Comparisons among Treatment Means in the General Linear Model	p. 83
Introduction	p. 83
The Model and Estimates	p. 85
The Model	p. 86
The Estimates	p. 87
The Multivariate t Distribution	p. 88
Simultaneous Inferences for Estimable Functions	p. 88
All Pairwise Comparisons in ANCOVA Models	p. 89
Confidence Intervals	p. 90
The %SimIntervals Macro	p. 92
Tests of Hypotheses and Adjusted p-values	p. 96
Comparisons with a Control	p. 98
Multiple Comparisons when There are CLASSvar x Covariate Interactions	p. 102
Concluding Remarks	p. 105
Inferences for General Linear Functions of Means	p. 107
Introduction	p. 107
Inferences for Any Finite Set of Linear Functions	p. 108
Comparisons with Control, Including Dose-response Contrasts	p. 108
Example: Evaluating Dose Response of Litter Weights in Rats	p. 109
Inferences for Infinite Sets of Linear Functions	p. 112
ANOVA tests	p. 113
Confidence Bands for Regression Functions	p. 121
Confidence Bands for Partial Functions	p. 124
Confidence Band for Difference of Regression Functions; Interaction with Classification Variable and Covariate	p. 127
Comparing the Discrete Method with the Continuous Method	p. 130
Concluding Remarks	p. 132
Power and Sample Size in Simultaneous Inference	p. 135
Introduction	p. 135
Definitions of Power	p. 137
Complete Power	p. 138
Minimal Power	p. 138
Individual Power	p. 138
Proportional Power	p. 139
Examples Using Individual Power	p. 139
All Pairwise Comparisons	p. 139
Comparisons with a control	p. 141
Simultaneous Confidence Intervals for Means	p. 142
Examples Using Combined Power Definitions	p. 143
All Pairwise Comparisons	p. 144
True FWE and Directional FWE	p. 145
Comparisons with a Control	p. 146
A Macro to Simulate and Graph Combined Power	p. 147
Concluding Remarks	p. 148
Stepwise and Closed Testing Procedures	p. 149
Introduction	p. 149
The Closure Principle	p. 149
Step-down Procedures for Testing All Subset Homogeneity Hypotheses	p. 151
Tukey-Welsch	p. 151
Example: Comparing Cholesterol Reduction Using Five Treatments	p. 152
More about REGWQ	p. 154
Power of the REGWQ Method	p. 154
Begun and Gabriel	p. 155
On Shortcut Closure Procedures	p. 157
A Closed Procedure for Combination Tests	p. 158
Caveats Concerning Closed Testing Methods	p. 160
Step-down Procedures for Testing against a Common Control	p. 160
Step-down and Closed Testing Procedures for Dose-Response Analysis	p. 162
Tukey's Trend Test	p. 163
A Closed Testing Procedure for Dose Response	p. 165
Williams' Test	p. 167
Closed Tests for General Contrasts	p. 168
All Pairwise Comparisons--Cholesterol Reduction Data Revisited	p. 170
Using the %SimTests Macro with General Contrasts	p. 171
Using the %SimTests Macro with General Contrasts and Covariates	p. 172
Example: Evaluating Dose Response of Litter Weights in Rats--Revisited	p. 173
Concluding Remarks	p. 174
Simultaneous Inference in Two-Way and Higher-Way ANOVA and ANCOVA	p. 177
Introduction	p. 177
Two-Way ANOVA	p. 178
Balanced with Replication	p. 178
The Cell Means Model	p. 178
Simultaneous Inference for Both Sets of Main Effects	p. 180
Interaction Contrasts	p. 181
Balanced ANOVA without Replication	p. 183
Unbalanced Designs	p. 187
Incomplete Two-way Designs	p. 189
More Complex ANOVAS	p. 192
Examples	p. 192
Example: Effect of Protein in Diet on Weight Gain in Pigs: Three Way ANOVA with a Covariate	p. 193
Example: Sub-group and Whole-group Analysis of a Respiratory Therapy Drug: Three-Way ANOVA with Weighted Contrasts	p. 195
Multiple F-Tests	p. 198
Concluding Remarks	p. 201
Multiple Comparisons in Heteroscedastic, Mixed, and Multivariate Models Using PROC MIXED	p. 203
Introduction	p. 203
Multiple Comparisons in the Heteroscedastic ANOVA	p. 204
Simultaneous Intervals	p. 204
Simultaneous Tests	p. 206
Multiple Comparisons Among Treatments When Blocks Are Random	p. 208
RCBD with One Observation per Cell	p. 209
Incomplete Blocks	p. 210
Comparing Means When Random Factors Interact	p. 212
Repeated Measures Analysis	p. 213
Repeated Measures Experimental Design	p. 213
Multivariate Analysis	p. 217
Testing Mean Differences in the MANOVA Model--Multiple Outcomes	p. 217
Multiple Comparisons in MANCOVA using PROC MIXED	p. 220
Multiple Comparisons of Simple Effects in Repeated Measures	p. 222
Concluding Remarks	p. 225
Multiple Comparisons of Means in Univariate and Multivariate Models, Allowing Nonnormality, Using PROC MULTTEST	p. 227
Introduction	p. 227
Univariate Means Tests Using PROC MULTTEST	p. 228
Bootstrap Resampling	p. 228
Permutation Resampling	p. 231
A Caveat: The Subset Pivotality Condition, Heteroscedasticity and Excess Type I Errors	p. 234
Incorporating Covariates with PROC MULTTEST	p. 235
Testing Means from Multivariate Data Using PROC MULTTEST	p. 235
Bootstrap and Permutation Resampling	p. 235
Multiple Endpoints and the Subset Pivotality Condition	p. 237
Missing Value Handling	p. 237
Inferences for Multiple Contrasts and Multiple Variables Simultaneously	p. 237
Concluding Remarks	p. 238
Multiple Comparisons with Binary Data Using PROC MULTTEST	p. 241
Introduction	p. 241
Multivariate Two-Sample Binary Outcomes	p. 241
Resampling-based Multiplicity Adjustment	p. 242
Multiplicity Adjustments Ignoring Correlations	p. 246
Multiple Pairwise Comparisons with Binary Data	p. 247
The Good and the Bad of Multiple Pairwise Comparisons: Two Examples	p. 248
Using the Discrete Bonferroni Method to Avoid the Subset Pivotality Pitfall	p. 250
Closed Binary Comparisons against a Control Using PROC MULTTEST	p. 251
Improving the Power of Multiple Binary Tests	p. 256
Comparison with PROC MULLTEST	p. 259
Multiple Linear Contrast Tests	p. 260
Multiple Animal Carcinogenicity Tests	p. 262
Miscellaneous PROC MULTTEST Applications	p. 266
Freeman-Tukey (FT) Test	p. 266
Mixing Binary and Continuous Variables	p. 268
Multiple Comparisons of Survival Functions	p. 268
Concluding Remarks	p. 270
Bayesian Multiple Comparisons and Multiple Tests	p. 273
Introduction	p. 273
The Variance Component Model	p. 275
Analysis of an Incomplete Block Design	p. 275
The Model	p. 275
Generating the Sample	p. 276
Simultaneous Intervals	p. 277
Multiple Hypothesis Testing: A Loss Function Approach	p. 278
Multiple Bayesian Tests of Point Nulls	p. 281
Concluding Remarks	p. 285
Additional Topics	p. 287
Introduction	p. 287
Large-Sample Multiple Comparisons Using PROC LOGISTIC, PROC LIFEREG, PROC PHREG, PROC CATMOD, PROC GENMOD	p. 287
Multiple Comparisons with the Best	p. 289
Infinitely Many Comparisons with Multivariate Data	p. 291
Interim Analysis and Repeated Significance Tests in Clinical Trials	p. 293
Concluding Remarks	p. 295
Macro Code	p. 297
The %Rom Macro	p. 298
The %HochBen Macro	p. 300
The %SimIntervals Macro	p. 304
The %MakeGLMStats Macro	p. 308
The %IndividualPower Macro	p. 310
The %SimPower Macro	p. 312
The %PlotSimPower Macro	p. 319
The %BegGab Macro	p. 321
The %RCC Macro	p. 327
The %Williams Macro	p. 332
The %SimTests Macro	p. 335
The %RomEx Macro	p. 343
The %RomMC Macro	p. 351
The %BayesIntervals Macro	p. 358
The %BayesTests Macro	p. 361
The %MCB Macro	p. 365
The %MCW Macro	p. 370
The %UMCB Macro	p. 372
The %UMCW Macro	p. 379
References	p. 383
Index	p. 389
Table of Contents provided by Ingram. All Rights Reserved.