Automatically classifying test results by semi-supervised learning

Rafig Almaghairbe, Marc Roper

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

6 Citations (Scopus)
149 Downloads (Pure)


A key component of software testing is deciding whether a test case has passed or failed: an expensive and error-prone manual activity. We present an approach to automatically classify passing and failing executions using semi-supervised learning on dynamic execution data (test inputs/outputs and execution traces). A small proportion of the test data is labelled as passing or failing and used in conjunction with the unlabelled data to build a classifier which labels the remaining outputs (classify them as passing or failing tests). A range of learning algorithms are investigated using several faulty versions of three systems along with varying types of data (inputs/outputs alone, or in combination with execution traces) and different labelling strategies (both failing and passing tests, and passing tests alone). The results show that in many cases labelling just a small proportion of the test cases – as low as 10% – is sufficient to build a classifier that is able to correctly categorise the large majority of the remaining test cases. This has important practical potential: when checking the test results from a system a developer need only examine a small proportion of these and use this information to train a learning algorithm to automatically classify the remainder.
Original languageEnglish
Title of host publication2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)
Place of Publication[Piscataway, NJ]
Number of pages11
ISBN (Print)978-1-4673-9003-3
Publication statusPublished - 8 Dec 2016
Event27th International Symposium on Software Reliability Engineering - Ottawa, Canada
Duration: 23 Oct 201627 Oct 2016


Conference27th International Symposium on Software Reliability Engineering
Abbreviated titleISSRE 2016


  • software testing
  • test case
  • semi-supervised learning
  • dynamic execution data
  • test oracles
  • learning algorithms


Dive into the research topics of 'Automatically classifying test results by semi-supervised learning'. Together they form a unique fingerprint.

Cite this