Estimating fault numbers remaining after testing

Research output: Contribution to conferencePaper

Abstract

Testing is an essential component of the software development process, but also one which is exceptionally difficult to manage and control. For example, it is well understood that testing techniques are not guaranteed to detect all faults, but more frustrating is that after the application of a testing technique the tester has little or no knowledge of how many faults might still be left undiscovered. This paper investigates the performance of a range of capture-recapture models to determine the accuracy with which they predict the number of defects remaining after testing. The models are evaluated with data from two empirical testing-related studies and from one larger publicly available project and the factors affecting the accuracy of the models are analysed. The paper also considers how additional information (such as structural coverage data) may be used to
improve the accuracy of the estimates. The results demonstrate that diverse sets of faults resulting from different testers using different techniques tend to produce the most accurate results, and also illustrate the sensitivity of the estimators to the patterns of fault data.

Conference

Conference2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST)
CountryLuxembourg
CityLuxembourg
Period18/03/1322/06/13

Fingerprint

Testing
Software engineering
Defects

Keywords

  • fault numbers
  • testing
  • software

Cite this

Roper, M. (2013). Estimating fault numbers remaining after testing. Paper presented at 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), Luxembourg, Luxembourg.
Roper, Marc. / Estimating fault numbers remaining after testing. Paper presented at 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), Luxembourg, Luxembourg.
@conference{f82b214e43af441f912b30784a18ce91,
title = "Estimating fault numbers remaining after testing",
abstract = "Testing is an essential component of the software development process, but also one which is exceptionally difficult to manage and control. For example, it is well understood that testing techniques are not guaranteed to detect all faults, but more frustrating is that after the application of a testing technique the tester has little or no knowledge of how many faults might still be left undiscovered. This paper investigates the performance of a range of capture-recapture models to determine the accuracy with which they predict the number of defects remaining after testing. The models are evaluated with data from two empirical testing-related studies and from one larger publicly available project and the factors affecting the accuracy of the models are analysed. The paper also considers how additional information (such as structural coverage data) may be used toimprove the accuracy of the estimates. The results demonstrate that diverse sets of faults resulting from different testers using different techniques tend to produce the most accurate results, and also illustrate the sensitivity of the estimators to the patterns of fault data.",
keywords = "fault numbers, testing, software",
author = "Marc Roper",
year = "2013",
language = "English",
note = "2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST) ; Conference date: 18-03-2013 Through 22-06-2013",

}

Roper, M 2013, 'Estimating fault numbers remaining after testing' Paper presented at 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), Luxembourg, Luxembourg, 18/03/13 - 22/06/13, .

Estimating fault numbers remaining after testing. / Roper, Marc.

2013. Paper presented at 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), Luxembourg, Luxembourg.

Research output: Contribution to conferencePaper

TY - CONF

T1 - Estimating fault numbers remaining after testing

AU - Roper, Marc

PY - 2013

Y1 - 2013

N2 - Testing is an essential component of the software development process, but also one which is exceptionally difficult to manage and control. For example, it is well understood that testing techniques are not guaranteed to detect all faults, but more frustrating is that after the application of a testing technique the tester has little or no knowledge of how many faults might still be left undiscovered. This paper investigates the performance of a range of capture-recapture models to determine the accuracy with which they predict the number of defects remaining after testing. The models are evaluated with data from two empirical testing-related studies and from one larger publicly available project and the factors affecting the accuracy of the models are analysed. The paper also considers how additional information (such as structural coverage data) may be used toimprove the accuracy of the estimates. The results demonstrate that diverse sets of faults resulting from different testers using different techniques tend to produce the most accurate results, and also illustrate the sensitivity of the estimators to the patterns of fault data.

AB - Testing is an essential component of the software development process, but also one which is exceptionally difficult to manage and control. For example, it is well understood that testing techniques are not guaranteed to detect all faults, but more frustrating is that after the application of a testing technique the tester has little or no knowledge of how many faults might still be left undiscovered. This paper investigates the performance of a range of capture-recapture models to determine the accuracy with which they predict the number of defects remaining after testing. The models are evaluated with data from two empirical testing-related studies and from one larger publicly available project and the factors affecting the accuracy of the models are analysed. The paper also considers how additional information (such as structural coverage data) may be used toimprove the accuracy of the estimates. The results demonstrate that diverse sets of faults resulting from different testers using different techniques tend to produce the most accurate results, and also illustrate the sensitivity of the estimators to the patterns of fault data.

KW - fault numbers

KW - testing

KW - software

UR - http://www.ieee.org/conferences_events/conferences/conferencedetails/index.html?Conf_ID=19854

M3 - Paper

ER -

Roper M. Estimating fault numbers remaining after testing. 2013. Paper presented at 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), Luxembourg, Luxembourg.