Automatically classifying test results by semi-supervised learning

Rafig Almaghairbe, Marc Roper

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

2 Citations (Scopus)

Abstract

A key component of software testing is deciding whether a test case has passed or failed: an expensive and error-prone manual activity. We present an approach to automatically classify passing and failing executions using semi-supervised learning on dynamic execution data (test inputs/outputs and execution traces). A small proportion of the test data is labelled as passing or failing and used in conjunction with the unlabelled data to build a classifier which labels the remaining outputs (classify them as passing or failing tests). A range of learning algorithms are investigated using several faulty versions of three systems along with varying types of data (inputs/outputs alone, or in combination with execution traces) and different labelling strategies (both failing and passing tests, and passing tests alone). The results show that in many cases labelling just a small proportion of the test cases – as low as 10% – is sufficient to build a classifier that is able to correctly categorise the large majority of the remaining test cases. This has important practical potential: when checking the test results from a system a developer need only examine a small proportion of these and use this information to train a learning algorithm to automatically classify the remainder.
LanguageEnglish
Title of host publication2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)
Place of Publication[Piscataway, NJ]
PublisherIEEE
Pages116-126
Number of pages11
ISBN (Print)978-1-4673-9003-3
DOIs
Publication statusPublished - 8 Dec 2016
Event27th International Symposium on Software Reliability Engineering - Ottawa, Canada
Duration: 23 Oct 201627 Oct 2016

Conference

Conference27th International Symposium on Software Reliability Engineering
Abbreviated titleISSRE 2016
CountryCanada
CityOttawa
Period23/10/1627/10/16

Fingerprint

Supervised learning
Labeling
Learning algorithms
Classifiers
Software testing
Information use
Labels

Keywords

  • software testing
  • test case
  • semi-supervised learning
  • dynamic execution data
  • test oracles
  • learning algorithms

Cite this

Almaghairbe, R., & Roper, M. (2016). Automatically classifying test results by semi-supervised learning. In 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE) (pp. 116-126). [Piscataway, NJ]: IEEE. https://doi.org/10.1109/ISSRE.2016.22
Almaghairbe, Rafig ; Roper, Marc. / Automatically classifying test results by semi-supervised learning. 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). [Piscataway, NJ] : IEEE, 2016. pp. 116-126
@inproceedings{c9592a3c9de34435b476f1646c050e3b,
title = "Automatically classifying test results by semi-supervised learning",
abstract = "A key component of software testing is deciding whether a test case has passed or failed: an expensive and error-prone manual activity. We present an approach to automatically classify passing and failing executions using semi-supervised learning on dynamic execution data (test inputs/outputs and execution traces). A small proportion of the test data is labelled as passing or failing and used in conjunction with the unlabelled data to build a classifier which labels the remaining outputs (classify them as passing or failing tests). A range of learning algorithms are investigated using several faulty versions of three systems along with varying types of data (inputs/outputs alone, or in combination with execution traces) and different labelling strategies (both failing and passing tests, and passing tests alone). The results show that in many cases labelling just a small proportion of the test cases – as low as 10{\%} – is sufficient to build a classifier that is able to correctly categorise the large majority of the remaining test cases. This has important practical potential: when checking the test results from a system a developer need only examine a small proportion of these and use this information to train a learning algorithm to automatically classify the remainder.",
keywords = "software testing, test case, semi-supervised learning, dynamic execution data, test oracles, learning algorithms",
author = "Rafig Almaghairbe and Marc Roper",
note = "{\circledC} 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.",
year = "2016",
month = "12",
day = "8",
doi = "10.1109/ISSRE.2016.22",
language = "English",
isbn = "978-1-4673-9003-3",
pages = "116--126",
booktitle = "2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)",
publisher = "IEEE",

}

Almaghairbe, R & Roper, M 2016, Automatically classifying test results by semi-supervised learning. in 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). IEEE, [Piscataway, NJ], pp. 116-126, 27th International Symposium on Software Reliability Engineering, Ottawa, Canada, 23/10/16. https://doi.org/10.1109/ISSRE.2016.22

Automatically classifying test results by semi-supervised learning. / Almaghairbe, Rafig; Roper, Marc.

2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). [Piscataway, NJ] : IEEE, 2016. p. 116-126.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Automatically classifying test results by semi-supervised learning

AU - Almaghairbe, Rafig

AU - Roper, Marc

N1 - © 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

PY - 2016/12/8

Y1 - 2016/12/8

N2 - A key component of software testing is deciding whether a test case has passed or failed: an expensive and error-prone manual activity. We present an approach to automatically classify passing and failing executions using semi-supervised learning on dynamic execution data (test inputs/outputs and execution traces). A small proportion of the test data is labelled as passing or failing and used in conjunction with the unlabelled data to build a classifier which labels the remaining outputs (classify them as passing or failing tests). A range of learning algorithms are investigated using several faulty versions of three systems along with varying types of data (inputs/outputs alone, or in combination with execution traces) and different labelling strategies (both failing and passing tests, and passing tests alone). The results show that in many cases labelling just a small proportion of the test cases – as low as 10% – is sufficient to build a classifier that is able to correctly categorise the large majority of the remaining test cases. This has important practical potential: when checking the test results from a system a developer need only examine a small proportion of these and use this information to train a learning algorithm to automatically classify the remainder.

AB - A key component of software testing is deciding whether a test case has passed or failed: an expensive and error-prone manual activity. We present an approach to automatically classify passing and failing executions using semi-supervised learning on dynamic execution data (test inputs/outputs and execution traces). A small proportion of the test data is labelled as passing or failing and used in conjunction with the unlabelled data to build a classifier which labels the remaining outputs (classify them as passing or failing tests). A range of learning algorithms are investigated using several faulty versions of three systems along with varying types of data (inputs/outputs alone, or in combination with execution traces) and different labelling strategies (both failing and passing tests, and passing tests alone). The results show that in many cases labelling just a small proportion of the test cases – as low as 10% – is sufficient to build a classifier that is able to correctly categorise the large majority of the remaining test cases. This has important practical potential: when checking the test results from a system a developer need only examine a small proportion of these and use this information to train a learning algorithm to automatically classify the remainder.

KW - software testing

KW - test case

KW - semi-supervised learning

KW - dynamic execution data

KW - test oracles

KW - learning algorithms

UR - http://issre.net/

UR - http://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000700

U2 - 10.1109/ISSRE.2016.22

DO - 10.1109/ISSRE.2016.22

M3 - Conference contribution book

SN - 978-1-4673-9003-3

SP - 116

EP - 126

BT - 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)

PB - IEEE

CY - [Piscataway, NJ]

ER -

Almaghairbe R, Roper M. Automatically classifying test results by semi-supervised learning. In 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). [Piscataway, NJ]: IEEE. 2016. p. 116-126 https://doi.org/10.1109/ISSRE.2016.22