Evaluating Learning Algorithms: A Classification Perspective

by Nathalie Japkowicz , Mohak Shah

• ISBN13: 9780521196000
• ISBN10: 0521196000
• Format: Hardcover
• Copyright: 2011-01-17
• Publisher: Cambridge University Press

Summary

The field of machine learning has matured to the point where many sophisticated learning approaches can be applied to practical applications. Thus it is of critical importance that researchers have the proper tools to evaluate learning approaches and understand the underlying issues. This book examines various aspects of the evaluation process with an emphasis on classification algorithms. The authors describe several techniques for classifier performance assessment, error estimation and resampling, obtaining statistical significance as well as selecting appropriate domains for evaluation. They also present a unified evaluation framework and highlight how different components of evaluation are both significantly interrelated and interdependent. The techniques presented in the book are illustrated using R and WEKA facilitating better practical insight as well as implementation. Aimed at researchers in the theory and applications of machine learning, this book offers a solid basis for conducting performance evaluations of algorithms in practical settings.

Author Biography

Nathalie Japkowicz is a Professor of Computer Science at the School of Information Technology and Engineering of the University of Ottawa. She also taught machine learning and artificial intelligence at Dalhousie University and Ohio State University. Along with machine learning evaluation, her research interests include one-class learning, the class imbalance problem, and learning in the presence of concept drifts. Mohak Shah is a Postdoctoral Fellow, at McGill University...He earned a PhD in Computer Science from the University of Ottawa in 2006 and was a Postdoctoral Fellow at CHUL Genomics Research Center in Quebec prior to joining McGill. His research interests span machine learning and statistical learning theory as well as their application to various domains.

Table of Contents

Preface	p. xi
Acronyms	p. xv
Introduction	p. 1
The De Facto Culture	p. 3
Motivations for This Book	p. 6
The De Facto Approach	p. 7
Broader Issues with Evaluation Approaches	p. 12
What Can We Do?	p. 16
Is Evaluation an End in Itself?	p. 18
Purpose of the Book	p. 19
Other Takes on Evaluation	p. 20
Moving Beyond Classification	p. 20
Thematic Organization	p. 21
Machine Learning and Statistics Overview	p. 23
Machine Learning Overview	p. 23
Statistics Overview	p. 42
Summary	p. 72
Bibliographic Remarks	p. 73
Performance Measures I	p. 74
Overview of the Problem	p. 75
An Ontology of Performance Measures	p. 81
Illustrative Example	p. 82
Performance Metrics with a Multiclass Focus	p. 85
Performance Metrics with a Single-Class Focus	p. 94
Illustration of the Confusion-Matrix-Only-Based Metrics Using WEKA	p. 107
Summary	p. 108
Bibliographic Remarks	p. 109
Performance Measures II	p. 111
Graphical Performance Measures	p. 112
Receiver Operating Characteristic (ROC) Analysis	p. 112
Other Visual Analysis Methods	p. 131
Continuous and Probabilistic Classifiers	p. 137
Specialized Metrics	p. 143
Illustration of the Ranking and Probabilistic Approaches Using R, ROCR, and WEKA	p. 146
Summary	p. 159
Bibliographic Remarks	p. 159
Error Estimation	p. 161
Introduction	p. 163
Holdout Approach	p. 164
What Implicitly Guides Resampling?	p. 167
Simple Resampling	p. 171
A Note on Model Selection	p. 177
Multiple Resampling	p. 178
Discussion	p. 185
Illustrations Using R	p. 187
Summary	p. 202
Bibliographic Remarks	p. 202
Appendix: Proof of Equation (5.5)	p. 204
Statistical Significance Testing	p. 206
The Purpose of Statistical Significance Testing	p. 207
The Limitations of Statistical Significance Testing	p. 210
An Overview of Relevant Statistical Tests	p. 213
A Note on Terminology	p. 215
Comparing Two Classifiers on a Single Domain	p. 217
Comparing Two Classifiers on Multiple Domains	p. 231
Comparing Multiple Classifiers on Multiple Domains	p. 239
Statistical Tests for Two Classifiers on a Single Domain Based on Resampling Techniques	p. 258
llustration of the Statistical Tests Application Using R	p. 263
Summary	p. 289
Bibliographic Remarks	p. 290
Datasets and Experimental Framework	p. 292
Repository-Based Approach	p. 294
Making Sense of Our Repositories: Metalearning	p. 300
Artificial Data Approach	p. 301
Community Participation: Web-Based Solutions	p. 304
Summary	p. 306
Bibliographic Remarks	p. 306
Recent Developments	p. 308
Performance Metrics	p. 309
Frameworks for Performance Metrics	p. 312
Combining Metrics	p. 317
Insights from Statistical Learning Theory	p. 323
Other Developments	p. 329
Summary	p. 330
Appendix: Proof of Theorems 8.1 and 8.2	p. 330
Conclusion	p. 335
An Evaluation Framework Template	p. 336
Concluding Remarks	p. 349
Bibliographic Remarks	p. 350
Statistical Tables	p. 351
The Z Table	p. 351
The t Table	p. 352
The x2 Table	p. 353
The Table of Critical Values for the Signed Test	p. 355
The Wilcoxon Table	p. 356
The F-Ratio Table	p. 357
The Friedman Table	p. 361
The Table of Critical Values for the Tukey Test	p. 362
The Table of Critical Values for the Dunnett Test	p. 363
Additional Information on the Data	p. 364
Two Case Studies	p. 368
Illustrative Case Study 1	p. 368
Illustrative Case Study 2	p. 375
Bibliography	p. 393
Index	p. 403
Table of Contents provided by Ingram. All Rights Reserved.