Forecast Verification A Practitioner's Guide in Atmospheric Science

by Jolliffe, Ian T.; Stephenson, David B.

• ISBN13: 9780470660713
• ISBN10: 0470660716
• Edition: 2nd
• Format: Hardcover
• Copyright: 2011-12-30
• Publisher: Wiley

Summary

Forecast Verification: A Practioner&'s Guide in Atmospheric Science, 2nd Edition provides an indispensible guide to this area of active research by combining depth of information with a range of topics to appeal both to professional practitioners and researchers and postgraduates. The editors have succeeded in presenting chapters by a variety of the leading experts in the field while still retaining a cohesive and highly accessible style. The book balances explanations of concepts with clear and useful discussion of the main application areas. & & Reviews of first edition: & "This book will provide a good reference, and I recommend it especially for developers and evaluators of statistical forecast systems." ( Bulletin of the American Meteorological Society; April 2004) & "&…a good mixture of theory and practical applications&…well organized and clearly written..." (Royal Statistical Society, Vol.168, No.1, January 2005)

Author Biography

Professor Ian Jolliffe is currently a Honorary Visiting Professor in SECaM. He took early retirement in 2004 form the University of Aberdeen, where he is Professor Emeritus and where he spent 12 years as Professor of Statistics before his retirement. Before Aberdeen, he was Lecturer, then Senior Lecturer in Statistics at the University of Kent for 22 years, apart from two one-year visiting posts at the Dalhousie University and the University of Guelph.
He gained his PhD in Statistics in 1970, and has been an active teacher and researcher in Applied Statistics since then. His thesis was on variable selection in principal component analysis (PCA), and topics related to PCA have formed a substantial strand of his research throughout his career.
He has a long-standing interest in applications of Statistics to atmospheric science, and forecast verification is now one of his main research interests. He has co-authored two other books and has over 85 other publications; roughly one third of these are directly related to applications in climate or weather. He has successfully supervised 20 research students, 16 of them at PhD level.

Table of Contents

List of Contributors	p. xi
Preface	p. xiii
Preface to the First Edition	p. xv
Introduction	p. 1
A brief history and current practice	p. 1
History	p. 1
Current practice	p. 2
Reasons for forecast verification and its benefits	p. 3
Types of forecast and verification data	p. 4
Scores, skill and value	p. 5
Skill scores	p. 6
Artificial skill	p. 6
Statistical significance	p. 7
Value added	p. 8
Data quality and other practical considerations	p. 8
Summary	p. 9
Basic concepts	p. 11
Introduction	p. 11
Types of predictand	p. 11
Exploratory methods	p. 12
Numerical descriptive measures	p. 15
Probability, random variables and expectations	p. 20
Joint, marginal and conditional distributions	p. 20
Accuracy, association and skill	p. 22
Properties of verification measures	p. 22
Verification as a regression problem	p. 23
The Murphy-Winkler framework	p. 25
Dimensionality of the verification problem	p. 28
Deterministic forecasts of binary events	p. 31
Introduction	p. 31
Theoretical considerations	p. 33
Some basic descriptive statistics	p. 33
A general framework for verification: the distributions-oriented approach	p. 34
Performance measures in terms of factorizations of the joint distribution	p. 37
Diagrams for visualizing performance measures	p. 38
Case study: verification of cloud-fraction forecasts	p. 41
Signal detection theory and the ROC	p. 42
The signal detection model	p. 43
The relative operating characteristic (ROC)	p. 44
Metaverification: criteria for assessing performance measures	p. 45
Desirable properties	p. 45
Other properties	p. 49
Performance measures	p. 50
Overview of performance measures	p. 51
Sampling uncertainty and confidence intervals for performance measures	p. 55
Optimal threshold probabilities	p. 57
Acknowledgements	p. 59
Deterministic forecasts of multi-category events	p. 61
Introduction	p. 61
The contingency table: notation, definitions, and measures of accuracy	p. 62
Notation and definitions	p. 62
Measures of accuracy	p. 64
Skill scores	p. 64
Desirable attributes	p. 65
Gandin and Murphy equitable scores	p. 66
Gerrity equitable scores	p. 69
LEPSCAT	p. 71
SEEPS	p. 72
Summary remarks on scores	p. 73
Sampling variability of the contingency table and skill scores	p. 73
Deterministic forecasts of continuous variables	p. 77
Introduction	p. 77
Forecast examples	p. 77
First-order moments	p. 79
Bias	p. 79
Mean Absolute Error	p. 80
Bias correction and artificial skill	p. 81
Mean absolute error and skill	p. 81
Second- and higher-order moments	p. 82
Mean Squared Error	p. 82
MSE skill score	p. 82
MSE of scaled forecasts	p. 83
Correlation	p. 84
An example: testing the 'limit of predictability’	p. 86
Rank correlations	p. 87
Comparison of moments of the marginal distributions	p. 88
Graphical summaries	p. 90
Scores based on cumulative frequency	p. 91
Linear Error in Probability Space (LEPS)	p. 91
Quantile-quantile plots	p. 92
Conditional quantile plots	p. 92
Summary and concluding remarks	p. 94
Forecasts of spatial fields	p. 95
Introduction	p. 95
Matching methods	p. 96
Traditional verification methods	p. 97
Standard continuous and categorical approaches	p. 97
S1 and anomaly correlation	p. 98
Distributional methods	p. 99
Motivation for alternative approaches	p. 100
Neighbourhood methods	p. 103
Comparing neighbourhoods of forecasts and observations	p. 104
Comparing spatial forecasts with point observations	p. 104
Scale separation methods	p. 105
Feature-based methods	p. 108
Feature-matching techniques	p. 108
Structure-Amplitude-Location (SAL) technique	p. 110
Field deformation methods	p. 111
Location metrics	p. 111
Field deformation	p. 112
Comparison of approaches	p. 113
New approaches and applications: the future	p. 114
Summary	p. 116
Probability forecasts	p. 119
Introduction	p. 119
Probability theory	p. 120
Basic concepts from probability theory	p. 120
Probability forecasts, reliability and sufficiency	p. 121
Probabilistic scoring rules	p. 122
Definition and properties of scoring rules	p. 122
Commonly used scoring rules	p. 124
Decomposition of scoring rules	p. 125
The relative operating characteristic (ROC)	p. 126
Evaluation of probabilistic forecasting systems from data	p. 128
Three examples	p. 128
The empirical ROC	p. 130
The empirical score as a measure of performance	p. 130
Decomposition of the empirical score	p. 131
Binning forecasts and the leave-one-out error	p. 132
Testing reliability	p. 134
Reliability analysis for forecast A: the reliability diagram	p. 134
Reliability analysis for forecast B: the chi-squared test	p. 136
Reliability analysis for forecast C: the PIT	p. 138
Acknowledgements	p. 139
Ensemble forecasts	p. 141
Introduction	p. 141
Example data	p. 142
Ensembles interpreted as discrete samples	p. 143
Reliability of ensemble forecasts	p. 144
Multidimensional reliability	p. 152
Discrimination	p. 157
Ensembles interpreted as probabilistic forecasts	p. 159
Probabilistic interpretation of ensembles	p. 159
Probabilistic skill metrics applied to ensembles	p. 160
Effect of ensemble size on skill	p. 163
Summary	p. 166
Economic value and skill	p. 167
Introduction	p. 167
The cost/loss ratio decision model	p. 168
Value of a deterministic binary forecast system	p. 169
Probability forecasts	p. 172
Comparison of deterministic and probabilistic binary forecasts	p. 174
The relationship between value and the ROC	p. 175
Overall value and the Brier Skill Score	p. 178
Skill, value and ensemble size	p. 180
Applications: value and forecast users	p. 182
Summary	p. 183
Deterministic forecasts of extreme events and warnings	p. 185
Introduction	p. 185
Forecasts of extreme events	p. 186
Challenges	p. 186
Previous studies	p. 187
Verification measures for extreme events	p. 189
Modelling performance for extreme events	p. 191
Extreme events: summary	p. 194
Warnings	p. 195
Background	p. 195
Format of warnings and observations for verification	p. 196
Verification of warnings	p. 197
Warnings: summary	p. 200
Acknowledgements	p. 201
Seasonal and longer-range forecasts	p. 203
Introduction	p. 203
Forecast formats	p. 204
Deterministic and probabilistic formats	p. 204
Defining the predictand	p. 206
Inclusion of climatological forecasts	p. 206
Measuring attributes of forecast quality	p. 207
Skill	p. 207
Other attributes	p. 215
Statistical significance and uncertainty estimates	p. 216
Measuring the quality of individual forecasts	p. 217
Decadal and longer-range forecast verification	p. 218
Summary	p. 220
Epilogue: new directions in forecast verification	p. 221
Introduction	p. 221
Review of key concepts	p. 221
Forecast evaluation in other disciplines	p. 223
Statistics	p. 223
Finance and economics	p. 225
Medical and clinical studies	p. 226
Current research and future directions	p. 228
Acknowledgements	p. 230
Appendix: Verification Software	p. 231
What is good software?	p. 231
Correctness	p. 232
Documentation	p. 232
Open source/closed source/commercial	p. 232
Large user base	p. 232
Types of verification users	p. 232
Students	p. 233
Researchers	p. 233
Operational forecasters	p. 233
Institutional use	p. 233
Types of software and programming languages	p. 233
Spreadsheets	p. 235
Statistical programming languages	p. 235
Institutional supported software	p. 238
Model Evaluation Tool (MET)	p. 238
Ensemble Verification System (EVS)	p. 239
EUMETCAL Forecast Verification Training Module	p. 239
Displays of verification information	p. 239
National Weather Service Performance Management	p. 240
Forecast Evaluation Tool	p. 240
Glossary	p. 241
References	p. 251
Index	p. 267
Table of Contents provided by Publisher. All Rights Reserved.