Uncertain Judgements Eliciting Experts' Probabilities

by O'Hagan, Anthony; Buck, Caitlin E.; Daneshkhah, Alireza; Eiser, J. Richard; Garthwaite, Paul H.; Jenkinson, David J.; Oakley, Jeremy E.; Rakow, Tim

• ISBN13: 9780470029992
• ISBN10: 0470029994
• Edition: 1st
• Format: Hardcover
• Copyright: 2006-09-01
• Publisher: WILEY

Summary

Elicitation is the process of extracting expert knowledge about some unknown quantity or quantities, and formulating that information as a probability distribution. Elicitation is important in situations, such as modelling the safety of nuclear installations or assessing the risk of terrorist attacks, where expert knowledge is essentially the only source of good information. It also plays a major role in other contexts by augmenting scarce observational data, through the use of Bayesian statistical methods. However, elicitation is not a simple task, and practitioners need to be aware of a wide range of research findings in order to elicit expert judgements accurately and reliably. Uncertain Judgements introduces the area, before guiding the reader through the study of appropriate elicitation methods, illustrated by a variety of multi-disciplinary examples. This is achieved by: Presenting a methodological framework for the elicitation of expert knowledge incorporating findings from both statistical and psychological research. Detailing techniques for the elicitation of a wide range of standard distributions, appropriate to the most common types of quantities. Providing a comprehensive review of the available literature and pointing to the best practice methods and future research needs. Using examples from many disciplines, including statistics, psychology, engineering and health sciences. Including an extensive glossary of statistical and psychological terms. An ideal source and guide for statisticians and psychologists with interests in expert judgement or practical applications of Bayesian analysis, Uncertain Judgements will also benefit decision-makers, risk analysts, engineers and researchers in the medical and social sciences.

Author Biography

Professor Anthony O’Hagan is the Director of The Centre for Bayesian Statistics in Health Economics at the University of Sheffield. The Centre is a collaboration between the Department of Probability and Statistics and the School of Health and Related Research (ScHARR). The Department of Probability and Statistics is internationally respected for its research in Bayesian statistics, while ScHARR is one of the leading UK centres for economic evaluation.

Prof O’Hagan is an internationally leading expert in Bayesian Statistics.

Co-authors:

Professor Paul Gathwaite – Open University, Prof of Statistics, Maths and Computing

Dr Jeremy Oakley – Sheffield University

Professor John Brazier – Director of Health Economics Group, University of Sheffield

Dr Tim Rakow – University of Essex, Psychology Department

Dr Alireza Daneshkhah – University of Sheffield, Medical Statistics Department

Dr Jim Chilcott - School of Health Research, University of Sheffield, Department of OR

Table of Contents

Preface.

1 Fundamentals of Probability and Judgement.

1.1 Introduction.

1.2 Probability and elicitation.

1.2.1 Probability.

1.2.2 Random variables and probability distributions.

1.2.3 Summaries of distributions.

1.2.4 Joint distributions.

1.2.5 Bayes' theorem.

1.2.6 Elicitation.

1.3 Uncertainty and the interpretation of probability.

1.3.1 Aleatory and epistemic uncertainty.

1.3.2 Frequency and personal probabilities.

1.3.3 An extended example.

1.3.4 Implications for elicitation.

1.4 Elicitation and the psychology of judgement.

1.4.1 Judgement - absolute or relative?

1.4.2 Beyond perception.

1.4.3 Implications for elicitation.

1.5 What use are such judgements?

1.5.1 Normative theories of probability.

1.5.2 Coherence.

1.5.3 Do elicited probabilities have the desired interpretation?

1.6 Conclusions.

1.6.1 Elicitation practice.

1.6.2 Research questions.

2 The elicitation Context.

2.1 How and who?

2.1.1 Choice of format.

2.2 What is an expert?

2.3 The elicitation process.

2.3.1 Roles within the elicitation process.

2.3.2 A model for the elicitation process.

2.4 Conventions in Chapters 3 to 9. 

2.5 Conclusions.

2.5.1 Elicitation practice.

2.5.2 Research questions.

3 The psychology of Judgement Under Uncertainty.

3.1 Introduction.

3.1.1 Why psychology?

3.1.2 Chapter overview.

3.2 Understanding the task and the expert.

3.2.1 Cognitive capabilities: the proper view of human information processing?

3.2.2 Constructive processes: the proper view of the process?

3.3 Understanding research on human judgement.

3.3.1 Experts versus the rest: the proper focus or research?

3.3.2 Early research on subjective probability: 'conservatism' in Bayesian probability revision.

3.4 The heuristic and biases research programme.

3.4.1 Availability.

3.4.2 Representativeness.

3.4.3 Do frequency representations remove the biases attributed to availability and representativeness?

3.4.4 Anchoring and adjusting.

3.4.5 Support Theory.

3.4.6 The affect heuristic.

3.4.7 Critique of the heuristics and biased approach.

3.5 Experts and expertise.

3.5.1 The heuristics and biased approach.

3.5.2 The cognitive science approach.

3.5.3 'The middle way'.

3.6 Three meta theories of judgement.

3.6.1 The cognitive continuum.

3.6.2 The inside versus the outside view.

3.6.3 The naive intuitive statistician metaphor.

3.7 Conclusions.

3.7.1 Elicitation practice.

3.7.2 Research questions.

4 The Elicitation of Probabilities.

4.1 Introduction.

4.2 The Calibration of Subjective Probabilities.

4.2.1 Research methods in calibration research.

4.2.1 Calibration research: General findings.

4.2.3 Calibration research in applied setting.

4.2.4 A case study in probability judgement: calibration research in medicine.

4.3 The calibration in subjective probabilities: theories and explanations.

4.3.1 Explanation of probability judgement in calibration tasks.

4.3.2 Theories of the calibration of subjective probabilities.

4.4 Representations and methods.

4.4.1 Different modes for representing uncertainty.

4.4.2 Different formats for eliciting responses.

4.4.3 Key lessons.

4.5 Debiasing.

4.5.1 General principles for debiasing judgement.

4.5.2 'Managing noise'.

4.5.3 Redressing insufficient regressiveness in prediction.

4.5.4 A caveat concerning post hoc corrections.

4.6 Conclusions.

4.6.1 Elicitation practice.

4.6.2 Research questions.

5 Eliciting Distributions - General.

5.1 From probabilities to distributions.

5.1.1 From a few to infinity.

5.1.2 Summaries.

5.1.3 Fitting.

5.1.4 Overview.

5.2 Eliciting univariate distributions.

5.2.1 Summaries based on probabilities.

5.2.2 Proportions.

5.2.3 Other summaries.

5.3 Eliciting multivariate distributions.

5.3.1 Structuring.

5.3.2 Eliciting association.

5.3.3 Joint and conditional probabilities.

5.3.4 Regression.

5.3.5 Many variables.

5.4 Uncertainty and imprecision.

5.4.1 Qualifying alicitation error.

5.4.2 Sensivity analysis.

5.4.3 Feedback and overfitting.

5.5 Conclusions.

5.5.1 Elicitation practice.

5.5.2 Research questions.

6 Eliciting and Fitting a Parametric Distribution.

6.1 Introduction.

6.2 Outline of this chapter.

6.3 Eliciting opinion about a proportion.

6.4 Eliciting opinion about a general scalar quantity.

6.5 Eliciting opinion about a set of proportions.

6.6 Eliciting opinion about the parameters of a multivariate normal distribution.

6.7 Eliciting opinion about the parameters of a linear regression model.

6.8 Eliciting opinion about the parameters of a generalized linear model.

6.9 Elicitation methods for other problems.

6.10 Deficiencies in existing research.

6.11 Conclusions.

6.11.1 Elicitation practice.

6.11.2 Research questions.

7 Eliciting distributions - Uncertainty and Imprecision.

7.1 Introduction.

7.2 Imprecise probabilities.

7.3 Incomplete information.

7.4 Summary.

7.5 Conclusions.

7.5.1 Elicitation practice.

7.5.2 Research questions.

8 Evaluating Elicitation.

8.1 Introduction.

8.1.1 Good elicitation.

8.1.2 Innacurate knowledge.

8.1.3 Automatic calibration.

8.1.4 Lessons of the psychological literature.

8.1.5 Outline of this chapter.

8.2 Scoring rules.

8.2.1 Scoring rules for discrete probability distributions.

8.2.2 Scoring rules for continuous probability distributions.

8.3 Coherence, feedback and overfitting.

8.3.1 Coherence and calibration.

8.3.2 Feedback and overfitting.

8.4 Conclusions.

8.4.1 Elicitation practice.

8.4.2 Research questions.

9 Multiple experts.

9.1 Introduction.

9.2 Mathematical aggregation.

9.2.1 Bayesian methods.

9.2.2 Opinion pooling.

9.2.3 Cooke's method.

9.2.4 Performance of mathematical aggregation.

9.3 Behavioural aggregation.

9.3.1 Group elicitation.

9.3.2 Other methods of behavioural aggregation.   

9.3.3 Performance of behavioural methods.

9.4 Discussion.

9.5 Elicitation practice.

9.6 Research questions.

10 Published Examples of the Formal Elicitation of Expert Opinion.

10.1 Some applications.

10.2 An example of an elicitation interview - eliciting engine sales.

10.3 Medicine.

10.3.1 Diagnosis and treatment decisions.

10.3.2 Clinical trials.

10.3.3 Survival analysis.

10.3.4 Clinical psychology.

10.4 The nuclear industry.

10.5 Veterinary science.

10.6 Agriculture.

10.7 Meteorology.

10.8 Business studies, economics and finance.

10.9 Other professions.

10.10 Other examples of the elicitation of subjective probabilities.

11 Guidance on Best Practice.

12 Areas for Research.

Bibliography.

Glossary.