Estimating the probability of rare events: addressing zero failure data

Research output: Contribution to journalArticle

17 Citations (Scopus)

Abstract

Traditional statistical procedures for estimating the probability of an event result in an estimate of zero when no events are realized. Alternative inferential procedures have been proposed for the situation where zero events have been realized but often these are ad hoc, relying on selecting methods dependent on the data that have been realized. Such data-dependent inference decisions violate fundamental statistical principles, resulting in estimation procedures whose benefits are difficult to assess. In this article, we propose estimating the probability of an event occurring through minimax inference on the probability that future samples of equal size realize no more events than that in the data on which the inference is based. Although motivated by inference on rare events, the method is not restricted to zero event data and closely approximates the maximum likelihood estimate (MLE) for nonzero data. The use of the minimax procedure provides a risk adverse inferential procedure where there are no events realized. A comparison is made with the MLE and regions of the underlying probability are identified where this approach is superior. Moreover, a comparison is made with three standard approaches to supporting inference where no event data are realized, which we argue are unduly pessimistic. We show that for situations of zero events the estimator can be simply approximated with 1/2.5n, where n is the number of trials.
LanguageEnglish
Pages1120-1132
Number of pages13
JournalRisk Analysis
Volume31
Issue number7
Early online date14 Jan 2011
DOIs
Publication statusPublished - 1 Jul 2011

Fingerprint

Maximum likelihood
Likelihood Functions
Sampling Studies
Sample Size

Keywords

  • zero event
  • binomial
  • minimax

Cite this

@article{eb46fe50e7cc4f5a93d36465061d8213,
title = "Estimating the probability of rare events: addressing zero failure data",
abstract = "Traditional statistical procedures for estimating the probability of an event result in an estimate of zero when no events are realized. Alternative inferential procedures have been proposed for the situation where zero events have been realized but often these are ad hoc, relying on selecting methods dependent on the data that have been realized. Such data-dependent inference decisions violate fundamental statistical principles, resulting in estimation procedures whose benefits are difficult to assess. In this article, we propose estimating the probability of an event occurring through minimax inference on the probability that future samples of equal size realize no more events than that in the data on which the inference is based. Although motivated by inference on rare events, the method is not restricted to zero event data and closely approximates the maximum likelihood estimate (MLE) for nonzero data. The use of the minimax procedure provides a risk adverse inferential procedure where there are no events realized. A comparison is made with the MLE and regions of the underlying probability are identified where this approach is superior. Moreover, a comparison is made with three standard approaches to supporting inference where no event data are realized, which we argue are unduly pessimistic. We show that for situations of zero events the estimator can be simply approximated with 1/2.5n, where n is the number of trials.",
keywords = "zero event, binomial, minimax",
author = "John Quigley and Matthew Revie",
year = "2011",
month = "7",
day = "1",
doi = "10.1111/j.1539-6924.2010.01568.x",
language = "English",
volume = "31",
pages = "1120--1132",
journal = "Risk Analysis",
issn = "0272-4332",
number = "7",

}

Estimating the probability of rare events : addressing zero failure data. / Quigley, John; Revie, Matthew.

In: Risk Analysis, Vol. 31, No. 7, 01.07.2011, p. 1120-1132.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Estimating the probability of rare events

T2 - Risk Analysis

AU - Quigley, John

AU - Revie, Matthew

PY - 2011/7/1

Y1 - 2011/7/1

N2 - Traditional statistical procedures for estimating the probability of an event result in an estimate of zero when no events are realized. Alternative inferential procedures have been proposed for the situation where zero events have been realized but often these are ad hoc, relying on selecting methods dependent on the data that have been realized. Such data-dependent inference decisions violate fundamental statistical principles, resulting in estimation procedures whose benefits are difficult to assess. In this article, we propose estimating the probability of an event occurring through minimax inference on the probability that future samples of equal size realize no more events than that in the data on which the inference is based. Although motivated by inference on rare events, the method is not restricted to zero event data and closely approximates the maximum likelihood estimate (MLE) for nonzero data. The use of the minimax procedure provides a risk adverse inferential procedure where there are no events realized. A comparison is made with the MLE and regions of the underlying probability are identified where this approach is superior. Moreover, a comparison is made with three standard approaches to supporting inference where no event data are realized, which we argue are unduly pessimistic. We show that for situations of zero events the estimator can be simply approximated with 1/2.5n, where n is the number of trials.

AB - Traditional statistical procedures for estimating the probability of an event result in an estimate of zero when no events are realized. Alternative inferential procedures have been proposed for the situation where zero events have been realized but often these are ad hoc, relying on selecting methods dependent on the data that have been realized. Such data-dependent inference decisions violate fundamental statistical principles, resulting in estimation procedures whose benefits are difficult to assess. In this article, we propose estimating the probability of an event occurring through minimax inference on the probability that future samples of equal size realize no more events than that in the data on which the inference is based. Although motivated by inference on rare events, the method is not restricted to zero event data and closely approximates the maximum likelihood estimate (MLE) for nonzero data. The use of the minimax procedure provides a risk adverse inferential procedure where there are no events realized. A comparison is made with the MLE and regions of the underlying probability are identified where this approach is superior. Moreover, a comparison is made with three standard approaches to supporting inference where no event data are realized, which we argue are unduly pessimistic. We show that for situations of zero events the estimator can be simply approximated with 1/2.5n, where n is the number of trials.

KW - zero event

KW - binomial

KW - minimax

U2 - 10.1111/j.1539-6924.2010.01568.x

DO - 10.1111/j.1539-6924.2010.01568.x

M3 - Article

VL - 31

SP - 1120

EP - 1132

JO - Risk Analysis

JF - Risk Analysis

SN - 0272-4332

IS - 7

ER -