Statistical and Methodological Myths and Urban Legends: Doctrine, Verity and Fable in Organizational and Social Sciences

by Lance; Charles E.

• ISBN13: 9780805862379
• ISBN10: 0805862374
• Edition: 1st
• Format: Hardcover
• Copyright: 2008-10-03
• Publisher: Routledge

Summary

The objective of this book is to provide an up-to-date review of commonly undertaken methodological and statistical practices that are sustained, in part, upon sound rationale and justification and, in part, upon unfounded lore. The practices themselves are not necessarily intrinscially faulty. Rather, it is often the reasoning why or rationalization used to justify the practices that is questionable. In this book, a group of scholars look at statistical and urban myths and legends and suggest what the state of practice should be. This book meets an important need and will be of interest to researchers, students and scholars in the fields of organizational and social sciences. Book jacket.

Table of Contents

Preface	p. xv
About the Editors	p. xvii
Acknowledgments	p. xix
Introduction	p. 1
Statistical Issues
Missing Data Techniques and Low Response Rates: The Role of Systematic Nonresponse Parameters	p. 7
Organization of the Chapter	p. 8
Levels, Problems, and Mechanisms of Missing Data	p. 8
Three Levels of Missing Data	p. 9
Two Problems Caused by Missing Data (External Validity and Statistical Power)	p. 9
Missingness Mechanisms (MCAR, MAR, and MNAR)	p. 9
Missing Data Treatments	p. 11
A Fundamental Principle of Missing Data Analysis	p. 11
Missing Data Techniques (Listwise and Pairwise Deletion, ML, and MI)	p. 13
Systematic Nonresponse Parameters (d[subscript miss] and f[superscript 2 subscript miss])	p. 14
Theory of Survey Nonresponse	p. 17
Missing Data Legends	p. 21
"Low Response Rates Invalidate Results"	p. 21
"When in Doubt, Use Listwise or Pairwise Deletion"	p. 24
Applications	p. 26
Longitudinal Modeling	p. 26
Within-Group Agreement Estimation	p. 27
Meta-analysis	p. 27
Social Network Analysis	p. 28
Moderated Regression	p. 29
Conclusions	p. 29
Future Research on d[subscript miss] and f[superscript 2 subscript miss]	p. 30
Missing Data Techniques	p. 31
References	p. 31
Appendix	p. 35
Derivation of Response Rate Bias for the Correlation (Used to Generate Figure 1.1c)	p. 35
The Partial Revival of a Dead Horse? Comparing Classical Test Theory and Item Response Theory	p. 37
Basic Statement of the Two Theories	p. 38
Classical Test Theory	p. 38
Item Response Theory	p. 40
Criticisms and Limitations of CTT	p. 44
Lack of Population Invariance	p. 44
Person and Item Parameters on Different Scales	p. 45
Correlations Between Item Parameters	p. 46
Reliability as a Monolithic Concept	p. 47
Criticisms and Limitations of IRT	p. 48
Large Sample Sizes	p. 48
Strong Assumptions	p. 49
Complicated Programs	p. 50
Times to Use CTT	p. 50
Small Sample Sizes	p. 50
Multidimensional Data?	p. 51
CTT Supports Other Methodologies	p. 52
Times to Use IRT	p. 53
Focus on Particular Range of Construct	p. 53
Conduct Goodness-of-Fit Studies	p. 53
IRT Supports Many Psychometric Tools	p. 55
Conclusions	p. 56
References	p. 57
Four Common Misconceptions in Exploratory Factor Analysis	p. 61
The Choice Between Component and Common Factor Analysis Is Inconsequential	p. 62
The Component Versus Common Factor Debate: Methodological Arguments	p. 66
The Component Versus Common Factor Debate: Philosophical Arguments	p. 68
Differences in Results from Component and Common Factor Analysis	p. 69
Orthogonal Rotation Results in Better Simple Structure Than Oblique Rotation	p. 71
Oblique or Orthogonal Rotation?	p. 71
Do Orthogonal Rotations Result in Better Simple Structure?	p. 72
The Minimum Sample Size Needed for Factor Analysis Is... (Insert Your Favorite Guideline)	p. 74
New Sample Size Guidelines	p. 76
The "Eigenvalues Greater Than One" Rule Is the Best Way of Choosing the Number of Factors	p. 79
Discussion	p. 83
References	p. 85
Dr. StrangeLOVE, or: How I Learned to Stop Worrying and Love Omitted Variables	p. 89
Theoretical and Mathematical Definition of the Omitted Variables Problem	p. 91
Violated Assumptions	p. 96
More Complex Models	p. 97
Path Coefficient Bias Versus Significance Testing	p. 100
Minimizing the Risk of LOVE	p. 102
Experimental Control	p. 102
More Inclusive Models	p. 103
Use Previous Research to Justify Assumptions	p. 103
Consideration of Research Purpose	p. 104
References	p. 105
The Truth(s) on Testing for Mediation in the Social and Organizational Sciences	p. 107
Baron and Kenny's (1986) Four-Step Test of Mediation	p. 110
Condition/Step 1	p. 111
Condition/Step 2	p. 111
Condition/Step 3	p. 111
Condition/Step 4	p. 112
The Urban Legend: Baron and Kenny's Four-Step Test Is an Optimal and Sufficient Test for Mediation Hypotheses	p. 113
The Kernel of Truth About the Urban Legends	p. 113
Debunking the Legends	p. 116
A Test of a Mediation Hypothesis Should Consist of the Four Steps Articulated by Baron and Kenny (1986)	p. 116
Baron and Kenny's (1986) Four-Step Procedure Is the Optimal Test of Mediation Hypotheses	p. 120
Fulfilling the Conditions Articulated in the Baron and Kenny (1986) Four-Step Test Is Sufficient for Drawing Conclusions About Mediated Relationships	p. 122
Suggestions for Testing Mediation Hypotheses	p. 124
Structural Equation Modeling (SEM) as an Analytic Framework	p. 124
Summary of Tests of Mediation	p. 127
A Heuristic Framework for Classifying Mediation Models	p. 129
Summary	p. 135
Conclusion	p. 136
Author Note	p. 136
References	p. 137
Seven Deadly Myths of Testing Moderation in Organizational Research	p. 143
The Seven Myths	p. 144
Product Terms Create Multicollinearity Problems	p. 144
Coefficients on First-Order Terms Are Meaningless	p. 146
Measurement Error Poses Little Concern When First-Order Terms Are Reliable	p. 148
Product Terms Should Be Tested Hierarchically	p. 150
Curvilinearity Can Be Disregarded When Testing Moderation	p. 151
Product Terms Can Be Treated as Causal Variables	p. 156
Testing Moderation in Structural Equation Modeling Is Impractical	p. 158
Myths Beyond Moderation	p. 159
Conclusion	p. 160
References	p. 160
Alternative Model Specifications in Structural Equation Modeling: Facts, Fictions, and Truth	p. 165
The Core of the Issue	p. 167
AMS Strategies	p. 170
Equivalent Models	p. 170
Nested Models	p. 174
Nonnested Alternative Models	p. 177
Summary	p. 179
AMS in Practice	p. 181
Summary	p. 186
References	p. 187
On the Practice of Allowing Correlated Residuals Among Indicators in Structural Equation Models	p. 193
Unraveling the Urban Legend	p. 195
Extent of the Problem	p. 195
Origins	p. 196
A Brief Review of Structural Equation Modeling	p. 197
Indicator Residuals	p. 199
Model Fit	p. 200
An Example	p. 202
Why Correlated IRs Improve Fit	p. 204
Problems With Correlated Residuals	p. 207
Recommendations	p. 209
Summary and Conclusions	p. 211
References	p. 212
Methodological Issues
Qualitative Research: The Redheaded Stepchild in Organizational and Social Science Research?	p. 219
Definitional Issues	p. 221
Philosophical Differences in Qualitative and Quantiative Research	p. 222
Quantitative and Qualitative Conceptualizations of Validity	p. 223
Caveats and Assumptions	p. 225
Beliefs Associated With Qualitative Research	p. 225
Qualitative Research Does Not Utilize the Scientific Method	p. 225
Qualitative Research Lacks Methodological Rigor	p. 226
Qualitative Research Contributes Little to the Advancement of Knowledge	p. 228
Evaluating the Beliefs Associated With Qualitative Research	p. 229
Qualitative Research Does Not Utilize the Scientific Method	p. 234
Qualitative Research Is Methodologically Weak	p. 236
Qualitative Research Has Weak Internal Validity	p. 236
Qualitative Research Has Weak Construct Validity	p. 237
Qualitative Research Has Weak External Validity	p. 238
Qualitative Research Contributes Little to the Advancement of Knowledge	p. 239
The Future of Qualitative Research in the Social and Organizational Sciences	p. 240
Concluding Thoughts	p. 241
Author Note	p. 242
References	p. 242
Do Samples Really Matter That Much?	p. 247
Kernel of Truth	p. 248
Background	p. 251
History of the Concern	p. 251
The Research Base	p. 253
Why Do Samples Seem to Matter So Much?	p. 255
People Confuse Random Sampling With Random Assignment	p. 255
People Focus on the Wrong Things	p. 257
People Rely on Superficial Similarities	p. 259
Concluding Thoughts	p. 260
Author Note	p. 262
References	p. 262
Sample Size Rules of Thumb: Evaluating Three Common Practices	p. 267
Determine Whether Sample Size Is Appropriate by Conducting a Power Analysis Using Cohen's Definitions of Small, Medium, and Large Effect Size	p. 269
Discussion	p. 271
Increase the A Priori Type I Error Rate to .10 Because of Your Small Sample Size	p. 273
Discussion	p. 275
Sample Size Should Include at Least 5 Observations per Estimated Parameter in Covariance Structure Analyses	p. 277
Discussion	p. 279
Discussion	p. 280
Author Note	p. 283
References	p. 284
When Small Effect Sizes Tell a Big Story, and When Large Effect Sizes Don't	p. 287
Effect Size Defined	p. 289
The Urban Legend	p. 290
The Kernel of Truth	p. 291
Quine and Ontological Relativism	p. 292
Contextualization	p. 295
Inauspicious Designs	p. 296
Phenomena With Obscured Consequences	p. 299
Phenomena That Challenge Fundamental Assumptions	p. 300
The Flip Side: Trivial "Large" Effects	p. 302
Conclusion	p. 305
References	p. 306
So Why Ask Me? Are Self-Report Data Really That Bad?	p. 309
The Urban Legend of Self-Report Data and Its Historical Roots	p. 310
Construct Validity of Self-Report Data	p. 313
Interpreting the Correlations in Self-Report Data	p. 316
Social Desirability Responding in Self-Report Data	p. 319
Value of Data Collected From Non-Self-Report Measures	p. 325
Conclusion and Moving Forward	p. 330
References	p. 332
If It Ain't Trait It Must Be Method: (Mis)application of the Multitrait-Multimethod Design in Organizational Research	p. 337
Background	p. 338
Literature Review	p. 342
Range of Traits Studied	p. 342
Range of Methods Studied	p. 343
Not All "Measurement Methods" Are Created Equal	p. 344
The Case of Multisource Performance Appraisal	p. 345
The Case of AC Construct Validity	p. 347
Other Cases	p. 349
So, Are Any "Method" Facets Really Method Facets?	p. 350
Discriminating Method From Substance, or "If It Looks Like a Method and Quacks Like a Method..."	p. 351
References	p. 353
Chopped Liver? OK. Chopped Data? Not OK	p. 361
Urban Legends Regarding Chopped Data	p. 362
Urban Legends Associated With the Occurrence of Chopped Data	p. 363
Urban Legends Associated With Chopped Data Techniques	p. 364
Urban Legends Associated With Chopped Data Justifications	p. 365
Literature Review	p. 366
Chopped Data Through the Years	p. 367
Prevalence of Chopped Data	p. 370
The Occurrence of Chopped Data Over Time	p. 371
Chopped Data Across Disciplines	p. 372
Types of Chopped Data Approaches	p. 372
Evaluating Justifications for Using Chopped Data	p. 374
Insufficient or Faulty Justifications (Myths)	p. 374
Legitimate Justifications (Truths)	p. 376
Advantages of, Disadvantages of, and Recommendations for Using Chopped Data	p. 377
(Perceived) Advantages of Chopping Data	p. 378
Disadvantages of Chopping Data	p. 378
Recommendations When Faced With Chopping Data	p. 382
Conclusion	p. 383
References	p. 383
Subject Index	p. 387
Author Index	p. 401
Table of Contents provided by Ingram. All Rights Reserved.