### Abstract

Language | English |
---|---|

Pages | 109-120 |

Number of pages | 11 |

Journal | Reliability Engineering and System Safety |

Volume | 163 |

Early online date | 24 Feb 2017 |

DOIs | |

Publication status | E-pub ahead of print - 24 Feb 2017 |

### Fingerprint

### Keywords

- expert judgment
- calibration
- out-of-sample validation
- classical model

### Cite this

*Reliability Engineering and System Safety*,

*163*, 109-120. https://doi.org/10.1016/j.ress.2017.02.003

}

*Reliability Engineering and System Safety*, vol. 163, pp. 109-120. https://doi.org/10.1016/j.ress.2017.02.003

**Cross validation for the classical model of structured expert judgment.** / Colson, Abigail R.; Cooke, Roger M.

Research output: Contribution to journal › Article

TY - JOUR

T1 - Cross validation for the classical model of structured expert judgment

AU - Colson, Abigail R.

AU - Cooke, Roger M.

PY - 2017/2/24

Y1 - 2017/2/24

N2 - We update the 2008 TU Delft structured expert judgment database with data from 33 professionally contracted Classical Model studies conducted between 2006 and March 2015 to evaluate its performance relative to other expert aggregation models. We briefly review alternative mathematical aggregation schemes, including harmonic weighting, before focusing on linear pooling of expert judgments with equal weights and performance-based weights. Performance weighting outperforms equal weighting in all but 1 of the 33 studies in-sample. True out-of-sample validation is rarely possible for Classical Model studies, and cross validation techniques that split calibration questions into a training and test set are used instead. Performance weighting incurs an “out-of-sample penalty” and its statistical accuracy out-of-sample is lower than that of equal weighting. However, as a function of training set size, the statistical accuracy of performance-based combinations reaches 75% of the equal weight value when the training set includes 80% of calibration variables. At this point the training set is sufficiently powerful to resolve differences in individual expert performance. The information of performance-based combinations is double that of equal weighting when the training set is at least 50% of the set of calibration variables. Previous out-of-sample validation work used a Total Out-of-Sample Validity Index based on all splits of the calibration questions into training and test subsets, which is expensive to compute and includes small training sets of dubious value. As an alternative, we propose an Out-of-Sample Validity Index based on averaging the product of statistical accuracy and information over all training sets sized at 80% of the calibration set. Performance weighting outperforms equal weighting on this Out-of-Sample Validity Index in 26 of the 33 post-2006 studies; the probability of 26 or more successes on 33 trials if there were no difference between performance weighting and equal weighting is 0.001.

AB - We update the 2008 TU Delft structured expert judgment database with data from 33 professionally contracted Classical Model studies conducted between 2006 and March 2015 to evaluate its performance relative to other expert aggregation models. We briefly review alternative mathematical aggregation schemes, including harmonic weighting, before focusing on linear pooling of expert judgments with equal weights and performance-based weights. Performance weighting outperforms equal weighting in all but 1 of the 33 studies in-sample. True out-of-sample validation is rarely possible for Classical Model studies, and cross validation techniques that split calibration questions into a training and test set are used instead. Performance weighting incurs an “out-of-sample penalty” and its statistical accuracy out-of-sample is lower than that of equal weighting. However, as a function of training set size, the statistical accuracy of performance-based combinations reaches 75% of the equal weight value when the training set includes 80% of calibration variables. At this point the training set is sufficiently powerful to resolve differences in individual expert performance. The information of performance-based combinations is double that of equal weighting when the training set is at least 50% of the set of calibration variables. Previous out-of-sample validation work used a Total Out-of-Sample Validity Index based on all splits of the calibration questions into training and test subsets, which is expensive to compute and includes small training sets of dubious value. As an alternative, we propose an Out-of-Sample Validity Index based on averaging the product of statistical accuracy and information over all training sets sized at 80% of the calibration set. Performance weighting outperforms equal weighting on this Out-of-Sample Validity Index in 26 of the 33 post-2006 studies; the probability of 26 or more successes on 33 trials if there were no difference between performance weighting and equal weighting is 0.001.

KW - expert judgment

KW - calibration

KW - out-of-sample validation

KW - classical model

UR - http://www.sciencedirect.com/science/journal/09518320

U2 - 10.1016/j.ress.2017.02.003

DO - 10.1016/j.ress.2017.02.003

M3 - Article

VL - 163

SP - 109

EP - 120

JO - Reliability Engineering and System Safety

T2 - Reliability Engineering and System Safety

JF - Reliability Engineering and System Safety

SN - 0951-8320

ER -