Building test oracles by clustering failures

Rafig Almaghairbe, Marc Roper

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

3 Citations (Scopus)
140 Downloads (Pure)

Abstract

In recent years, software testing research has produced notable advances in the area of automated test data generation, but the corresponding oracle problem (a mechanism for determine the (in)correctness of an executed test case) is still a major problem. In this paper, we present a preliminary study which investigates the application of anomaly detection techniques (based on clustering) to automatically build an oracle using a system’s input/output pairs, based on the hypothesis that failures will tend to group into small clusters. The fault detection capability of the approach is evaluated on two systems and the findings reveal that failing outputs do indeed tend to congregate in small clusters, suggesting that the approach is feasible and has the potential to reduce by an order of magnitude the numbers of outputs that would need to be manually examined following a test run.
Original languageEnglish
Title of host publication2015 IEEE/ACM 10th International Workshop on Automation of Software Test (AST)
PublisherIEEE
Pages3-7
Number of pages5
DOIs
Publication statusPublished - 2015

Fingerprint

Software testing
Fault detection

Keywords

  • software testing
  • automated test data generation
  • oracle problem
  • anomaly detection techniques
  • fault detection

Cite this

Almaghairbe, R., & Roper, M. (2015). Building test oracles by clustering failures. In 2015 IEEE/ACM 10th International Workshop on Automation of Software Test (AST) (pp. 3-7). IEEE. https://doi.org/10.1109/AST.2015.8
Almaghairbe, Rafig ; Roper, Marc. / Building test oracles by clustering failures. 2015 IEEE/ACM 10th International Workshop on Automation of Software Test (AST) . IEEE, 2015. pp. 3-7
@inproceedings{afbdeb33598445dea40c2285c18631b0,
title = "Building test oracles by clustering failures",
abstract = "In recent years, software testing research has produced notable advances in the area of automated test data generation, but the corresponding oracle problem (a mechanism for determine the (in)correctness of an executed test case) is still a major problem. In this paper, we present a preliminary study which investigates the application of anomaly detection techniques (based on clustering) to automatically build an oracle using a system’s input/output pairs, based on the hypothesis that failures will tend to group into small clusters. The fault detection capability of the approach is evaluated on two systems and the findings reveal that failing outputs do indeed tend to congregate in small clusters, suggesting that the approach is feasible and has the potential to reduce by an order of magnitude the numbers of outputs that would need to be manually examined following a test run.",
keywords = "software testing, automated test data generation, oracle problem, anomaly detection techniques, fault detection",
author = "Rafig Almaghairbe and Marc Roper",
note = "{\circledC} 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.",
year = "2015",
doi = "10.1109/AST.2015.8",
language = "English",
pages = "3--7",
booktitle = "2015 IEEE/ACM 10th International Workshop on Automation of Software Test (AST)",
publisher = "IEEE",

}

Almaghairbe, R & Roper, M 2015, Building test oracles by clustering failures. in 2015 IEEE/ACM 10th International Workshop on Automation of Software Test (AST) . IEEE, pp. 3-7. https://doi.org/10.1109/AST.2015.8

Building test oracles by clustering failures. / Almaghairbe, Rafig ; Roper, Marc.

2015 IEEE/ACM 10th International Workshop on Automation of Software Test (AST) . IEEE, 2015. p. 3-7.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Building test oracles by clustering failures

AU - Almaghairbe, Rafig

AU - Roper, Marc

N1 - © 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

PY - 2015

Y1 - 2015

N2 - In recent years, software testing research has produced notable advances in the area of automated test data generation, but the corresponding oracle problem (a mechanism for determine the (in)correctness of an executed test case) is still a major problem. In this paper, we present a preliminary study which investigates the application of anomaly detection techniques (based on clustering) to automatically build an oracle using a system’s input/output pairs, based on the hypothesis that failures will tend to group into small clusters. The fault detection capability of the approach is evaluated on two systems and the findings reveal that failing outputs do indeed tend to congregate in small clusters, suggesting that the approach is feasible and has the potential to reduce by an order of magnitude the numbers of outputs that would need to be manually examined following a test run.

AB - In recent years, software testing research has produced notable advances in the area of automated test data generation, but the corresponding oracle problem (a mechanism for determine the (in)correctness of an executed test case) is still a major problem. In this paper, we present a preliminary study which investigates the application of anomaly detection techniques (based on clustering) to automatically build an oracle using a system’s input/output pairs, based on the hypothesis that failures will tend to group into small clusters. The fault detection capability of the approach is evaluated on two systems and the findings reveal that failing outputs do indeed tend to congregate in small clusters, suggesting that the approach is feasible and has the potential to reduce by an order of magnitude the numbers of outputs that would need to be manually examined following a test run.

KW - software testing

KW - automated test data generation

KW - oracle problem

KW - anomaly detection techniques

KW - fault detection

UR - http://tech.brookes.ac.uk/AST2015/

U2 - 10.1109/AST.2015.8

DO - 10.1109/AST.2015.8

M3 - Conference contribution book

SP - 3

EP - 7

BT - 2015 IEEE/ACM 10th International Workshop on Automation of Software Test (AST)

PB - IEEE

ER -

Almaghairbe R, Roper M. Building test oracles by clustering failures. In 2015 IEEE/ACM 10th International Workshop on Automation of Software Test (AST) . IEEE. 2015. p. 3-7 https://doi.org/10.1109/AST.2015.8