Elicitation in the classical model

John Quigley, Abigail Colson, Willy Aspinall, R.M. Cooke

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

The Classical Model (CM) is a performance-based approach for mathematically aggregating judgements from multiple experts, when reasoning about target questions under uncertainty. Individual expert performance is assessed against a set of seed questions, items from their field, for which the analyst knows or will know the true values, but the experts do not; the experts are, however, expected to provide accurate and informative distributional judgements that capture these values reliably. Performance is measured according as metrics for each expert’s statistical accuracy and informativeness, and the two metrics are convolved to deter-mine a weight for each expert, with which to modulate their contribution when pooling them together for a final combined assessment of the desired target values. This chapter provides mathematical and practical details of the CM, including describing the method for measuring expert performance and discussing approaches for devising good seed questions.
LanguageEnglish
Title of host publicationElicitation
Subtitle of host publicationThe Science and Art of Structuring Judgement
EditorsLuis Dias, Alec Morton, John Quigley
Place of PublicationNew York
PublisherSpringer
ISBN (Print)978-3-319-65051-7
DOIs
Publication statusAccepted/In press - 19 Jun 2017

Fingerprint

Value capture
Informativeness
Pooling
Uncertainty
Analysts

Keywords

  • classical model
  • uncertainty
  • seed questions

Cite this

Quigley, J., Colson, A., Aspinall, W., & Cooke, R. M. (Accepted/In press). Elicitation in the classical model. In L. Dias, A. Morton, & J. Quigley (Eds.), Elicitation: The Science and Art of Structuring Judgement New York: Springer. https://doi.org/10.1007/978-3-319-65052-4
Quigley, John ; Colson, Abigail ; Aspinall, Willy ; Cooke, R.M. / Elicitation in the classical model. Elicitation: The Science and Art of Structuring Judgement. editor / Luis Dias ; Alec Morton ; John Quigley. New York : Springer, 2017.
@inbook{d23d29a713dd43d1a05c1edd72570931,
title = "Elicitation in the classical model",
abstract = "The Classical Model (CM) is a performance-based approach for mathematically aggregating judgements from multiple experts, when reasoning about target questions under uncertainty. Individual expert performance is assessed against a set of seed questions, items from their field, for which the analyst knows or will know the true values, but the experts do not; the experts are, however, expected to provide accurate and informative distributional judgements that capture these values reliably. Performance is measured according as metrics for each expert’s statistical accuracy and informativeness, and the two metrics are convolved to deter-mine a weight for each expert, with which to modulate their contribution when pooling them together for a final combined assessment of the desired target values. This chapter provides mathematical and practical details of the CM, including describing the method for measuring expert performance and discussing approaches for devising good seed questions.",
keywords = "classical model, uncertainty, seed questions",
author = "John Quigley and Abigail Colson and Willy Aspinall and R.M. Cooke",
year = "2017",
month = "6",
day = "19",
doi = "10.1007/978-3-319-65052-4",
language = "English",
isbn = "978-3-319-65051-7",
editor = "Luis Dias and Morton, {Alec } and Quigley, {John }",
booktitle = "Elicitation",
publisher = "Springer",

}

Quigley, J, Colson, A, Aspinall, W & Cooke, RM 2017, Elicitation in the classical model. in L Dias, A Morton & J Quigley (eds), Elicitation: The Science and Art of Structuring Judgement. Springer, New York. https://doi.org/10.1007/978-3-319-65052-4

Elicitation in the classical model. / Quigley, John; Colson, Abigail; Aspinall, Willy; Cooke, R.M.

Elicitation: The Science and Art of Structuring Judgement. ed. / Luis Dias; Alec Morton; John Quigley. New York : Springer, 2017.

Research output: Chapter in Book/Report/Conference proceedingChapter

TY - CHAP

T1 - Elicitation in the classical model

AU - Quigley, John

AU - Colson, Abigail

AU - Aspinall, Willy

AU - Cooke, R.M.

PY - 2017/6/19

Y1 - 2017/6/19

N2 - The Classical Model (CM) is a performance-based approach for mathematically aggregating judgements from multiple experts, when reasoning about target questions under uncertainty. Individual expert performance is assessed against a set of seed questions, items from their field, for which the analyst knows or will know the true values, but the experts do not; the experts are, however, expected to provide accurate and informative distributional judgements that capture these values reliably. Performance is measured according as metrics for each expert’s statistical accuracy and informativeness, and the two metrics are convolved to deter-mine a weight for each expert, with which to modulate their contribution when pooling them together for a final combined assessment of the desired target values. This chapter provides mathematical and practical details of the CM, including describing the method for measuring expert performance and discussing approaches for devising good seed questions.

AB - The Classical Model (CM) is a performance-based approach for mathematically aggregating judgements from multiple experts, when reasoning about target questions under uncertainty. Individual expert performance is assessed against a set of seed questions, items from their field, for which the analyst knows or will know the true values, but the experts do not; the experts are, however, expected to provide accurate and informative distributional judgements that capture these values reliably. Performance is measured according as metrics for each expert’s statistical accuracy and informativeness, and the two metrics are convolved to deter-mine a weight for each expert, with which to modulate their contribution when pooling them together for a final combined assessment of the desired target values. This chapter provides mathematical and practical details of the CM, including describing the method for measuring expert performance and discussing approaches for devising good seed questions.

KW - classical model

KW - uncertainty

KW - seed questions

UR - http://www.springer.com/gb/book/9783319650517

U2 - 10.1007/978-3-319-65052-4

DO - 10.1007/978-3-319-65052-4

M3 - Chapter

SN - 978-3-319-65051-7

BT - Elicitation

A2 - Dias, Luis

A2 - Morton, Alec

A2 - Quigley, John

PB - Springer

CY - New York

ER -

Quigley J, Colson A, Aspinall W, Cooke RM. Elicitation in the classical model. In Dias L, Morton A, Quigley J, editors, Elicitation: The Science and Art of Structuring Judgement. New York: Springer. 2017 https://doi.org/10.1007/978-3-319-65052-4