Using machine learning to classify test outcomes

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

7 Downloads (Pure)

Abstract

When testing software it has been shown that there are substantial benefits to be gained from approaches which exercise unusual or unexplored interactions with a system - techniques such as random testing, fuzzing, and exploratory testing. However, such approaches have a drawback in that the outputs of the tests need to be manually checked for correctness, representing a significant burden for the software engineer. This paper presents a strategy to support the process of identifying which tests have passed or failed by combining clustering and semi-supervised learning. We have shown that by using machine learning it is possible to cluster test cases in such a way that those corresponding to failures concentrate into smaller clusters. Examining the test outcomes in cluster-size order has the effect of prioritising the results: those that are checked early on have a much higher probability of being a failing test. As the software engineer examines the results (and confirms or refutes the initial classification), this information is employed to bootstrap a secondary learner to further improve the accuracy of the classification of the (as yet) unchecked tests. Results from experimenting with a range of systems demonstrate the substantial benefits that can be gained from this strategy, and how remarkably accurate test output classifications can be derived from examining a relatively small proportion of results.
Original languageEnglish
Title of host publication2019 IEEE International Conference On Artificial Intelligence Testing (AITest)
Place of PublicationPiscataway, N.J.
PublisherIEEE
Pages99-100
Number of pages2
ISBN (Electronic)978-1-7281-0492-8
ISBN (Print)978-1-7281-0493-5
DOIs
Publication statusPublished - 20 May 2019

Fingerprint

Learning systems
Engineers
Software testing
Supervised learning
Testing

Keywords

  • semisupervised learning
  • pattern classification
  • software engineering
  • machine learning
  • program testing

Cite this

Roper, R. (2019). Using machine learning to classify test outcomes. In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest) (pp. 99-100). Piscataway, N.J.: IEEE. https://doi.org/10.1109/AITest.2019.00009
Roper, Richard. / Using machine learning to classify test outcomes. 2019 IEEE International Conference On Artificial Intelligence Testing (AITest). Piscataway, N.J. : IEEE, 2019. pp. 99-100
@inproceedings{cbe3556cf6084ddc9a99adda12ba95bc,
title = "Using machine learning to classify test outcomes",
abstract = "When testing software it has been shown that there are substantial benefits to be gained from approaches which exercise unusual or unexplored interactions with a system - techniques such as random testing, fuzzing, and exploratory testing. However, such approaches have a drawback in that the outputs of the tests need to be manually checked for correctness, representing a significant burden for the software engineer. This paper presents a strategy to support the process of identifying which tests have passed or failed by combining clustering and semi-supervised learning. We have shown that by using machine learning it is possible to cluster test cases in such a way that those corresponding to failures concentrate into smaller clusters. Examining the test outcomes in cluster-size order has the effect of prioritising the results: those that are checked early on have a much higher probability of being a failing test. As the software engineer examines the results (and confirms or refutes the initial classification), this information is employed to bootstrap a secondary learner to further improve the accuracy of the classification of the (as yet) unchecked tests. Results from experimenting with a range of systems demonstrate the substantial benefits that can be gained from this strategy, and how remarkably accurate test output classifications can be derived from examining a relatively small proportion of results.",
keywords = "semisupervised learning, pattern classification, software engineering, machine learning, program testing",
author = "Richard Roper",
note = "{\circledC} 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.",
year = "2019",
month = "5",
day = "20",
doi = "10.1109/AITest.2019.00009",
language = "English",
isbn = "978-1-7281-0493-5",
pages = "99--100",
booktitle = "2019 IEEE International Conference On Artificial Intelligence Testing (AITest)",
publisher = "IEEE",

}

Roper, R 2019, Using machine learning to classify test outcomes. in 2019 IEEE International Conference On Artificial Intelligence Testing (AITest). IEEE, Piscataway, N.J., pp. 99-100. https://doi.org/10.1109/AITest.2019.00009

Using machine learning to classify test outcomes. / Roper, Richard.

2019 IEEE International Conference On Artificial Intelligence Testing (AITest). Piscataway, N.J. : IEEE, 2019. p. 99-100.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Using machine learning to classify test outcomes

AU - Roper, Richard

N1 - © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

PY - 2019/5/20

Y1 - 2019/5/20

N2 - When testing software it has been shown that there are substantial benefits to be gained from approaches which exercise unusual or unexplored interactions with a system - techniques such as random testing, fuzzing, and exploratory testing. However, such approaches have a drawback in that the outputs of the tests need to be manually checked for correctness, representing a significant burden for the software engineer. This paper presents a strategy to support the process of identifying which tests have passed or failed by combining clustering and semi-supervised learning. We have shown that by using machine learning it is possible to cluster test cases in such a way that those corresponding to failures concentrate into smaller clusters. Examining the test outcomes in cluster-size order has the effect of prioritising the results: those that are checked early on have a much higher probability of being a failing test. As the software engineer examines the results (and confirms or refutes the initial classification), this information is employed to bootstrap a secondary learner to further improve the accuracy of the classification of the (as yet) unchecked tests. Results from experimenting with a range of systems demonstrate the substantial benefits that can be gained from this strategy, and how remarkably accurate test output classifications can be derived from examining a relatively small proportion of results.

AB - When testing software it has been shown that there are substantial benefits to be gained from approaches which exercise unusual or unexplored interactions with a system - techniques such as random testing, fuzzing, and exploratory testing. However, such approaches have a drawback in that the outputs of the tests need to be manually checked for correctness, representing a significant burden for the software engineer. This paper presents a strategy to support the process of identifying which tests have passed or failed by combining clustering and semi-supervised learning. We have shown that by using machine learning it is possible to cluster test cases in such a way that those corresponding to failures concentrate into smaller clusters. Examining the test outcomes in cluster-size order has the effect of prioritising the results: those that are checked early on have a much higher probability of being a failing test. As the software engineer examines the results (and confirms or refutes the initial classification), this information is employed to bootstrap a secondary learner to further improve the accuracy of the classification of the (as yet) unchecked tests. Results from experimenting with a range of systems demonstrate the substantial benefits that can be gained from this strategy, and how remarkably accurate test output classifications can be derived from examining a relatively small proportion of results.

KW - semisupervised learning

KW - pattern classification

KW - software engineering

KW - machine learning

KW - program testing

U2 - 10.1109/AITest.2019.00009

DO - 10.1109/AITest.2019.00009

M3 - Conference contribution book

SN - 978-1-7281-0493-5

SP - 99

EP - 100

BT - 2019 IEEE International Conference On Artificial Intelligence Testing (AITest)

PB - IEEE

CY - Piscataway, N.J.

ER -

Roper R. Using machine learning to classify test outcomes. In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest). Piscataway, N.J.: IEEE. 2019. p. 99-100 https://doi.org/10.1109/AITest.2019.00009