Home > Research > Publications & Outputs > Which Software Faults Are Tests Not Detecting?

Electronic data

  • Full paper

    Rights statement: © ACM, 2020. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering http://doi.acm.org/10.1145/3383219.3383236

    Accepted author manuscript, 174 KB, PDF document

    Available under license: CC BY-NC: Creative Commons Attribution-NonCommercial 4.0 International License

Links

Text available via DOI:

View graph of relations

Which Software Faults Are Tests Not Detecting?

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSNConference contribution/Paperpeer-review

Published

Standard

Which Software Faults Are Tests Not Detecting? / Petric, J.; Hall, T.; Bowes, David.

PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering. New York : ACM, 2020. p. 160-169.

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSNConference contribution/Paperpeer-review

Harvard

Petric, J, Hall, T & Bowes, D 2020, Which Software Faults Are Tests Not Detecting? in PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering. ACM, New York, pp. 160-169. https://doi.org/10.1145/3383219.3383236

APA

Petric, J., Hall, T., & Bowes, D. (2020). Which Software Faults Are Tests Not Detecting? In PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering (pp. 160-169). ACM. https://doi.org/10.1145/3383219.3383236

Vancouver

Petric J, Hall T, Bowes D. Which Software Faults Are Tests Not Detecting? In PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering. New York: ACM. 2020. p. 160-169 https://doi.org/10.1145/3383219.3383236

Author

Petric, J. ; Hall, T. ; Bowes, David. / Which Software Faults Are Tests Not Detecting?. PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering. New York : ACM, 2020. pp. 160-169

Bibtex

@inproceedings{8333810d0f4846428e167ff933f59fda,
title = "Which Software Faults Are Tests Not Detecting?",
abstract = "Context: Software testing plays an important role in assuring the reliability of systems. Assessing the efficacy of testing remains challenging with few established test effectiveness metrics. Those metrics that have been used (e.g. coverage and mutation analysis) have been criticised for insufficiently differentiating between the faults detected by tests. Objective: We investigate how effective tests are at detecting different types of faults and whether some types of fault evade tests more than others. Our aim is to suggest to developers specific ways in which their tests need to be improved to increase fault detection. Method: We investigate seven fault types and analyse how often each goes undetected in 10 open source systems. We statistically look for any relationship between the test set and faults. Results: Our results suggest that the fault detection rates of unit tests are relatively low, typically finding only about a half of all faults. In addition, conditional boundary and method call removals are less well detected by tests than other fault types. Conclusions: We conclude that the testing of these open source systems needs to be improved across the board. In addition, despite boundary cases being long known to attract faults, tests covering boundaries need particular improvement. Overall, we recommend that developers do not rely only on code coverage and mutation score to measure the effectiveness of their tests. ",
keywords = "software testing, test effectiveness, unit tests, Fault detection, Open source software, Open systems, Software reliability, Code coverage, Fault detection rate, Fault types, Mutation analysis, Mutation score, Open source system, Software fault, Test effectiveness, Software testing",
author = "J. Petric and T. Hall and David Bowes",
note = "{\textcopyright} ACM, 2020. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering http://doi.acm.org/10.1145/3383219.3383236",
year = "2020",
month = apr,
day = "15",
doi = "10.1145/3383219.3383236",
language = "English",
isbn = "9781450377317",
pages = "160--169",
booktitle = "PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering",
publisher = "ACM",

}

RIS

TY - GEN

T1 - Which Software Faults Are Tests Not Detecting?

AU - Petric, J.

AU - Hall, T.

AU - Bowes, David

N1 - © ACM, 2020. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering http://doi.acm.org/10.1145/3383219.3383236

PY - 2020/4/15

Y1 - 2020/4/15

N2 - Context: Software testing plays an important role in assuring the reliability of systems. Assessing the efficacy of testing remains challenging with few established test effectiveness metrics. Those metrics that have been used (e.g. coverage and mutation analysis) have been criticised for insufficiently differentiating between the faults detected by tests. Objective: We investigate how effective tests are at detecting different types of faults and whether some types of fault evade tests more than others. Our aim is to suggest to developers specific ways in which their tests need to be improved to increase fault detection. Method: We investigate seven fault types and analyse how often each goes undetected in 10 open source systems. We statistically look for any relationship between the test set and faults. Results: Our results suggest that the fault detection rates of unit tests are relatively low, typically finding only about a half of all faults. In addition, conditional boundary and method call removals are less well detected by tests than other fault types. Conclusions: We conclude that the testing of these open source systems needs to be improved across the board. In addition, despite boundary cases being long known to attract faults, tests covering boundaries need particular improvement. Overall, we recommend that developers do not rely only on code coverage and mutation score to measure the effectiveness of their tests.

AB - Context: Software testing plays an important role in assuring the reliability of systems. Assessing the efficacy of testing remains challenging with few established test effectiveness metrics. Those metrics that have been used (e.g. coverage and mutation analysis) have been criticised for insufficiently differentiating between the faults detected by tests. Objective: We investigate how effective tests are at detecting different types of faults and whether some types of fault evade tests more than others. Our aim is to suggest to developers specific ways in which their tests need to be improved to increase fault detection. Method: We investigate seven fault types and analyse how often each goes undetected in 10 open source systems. We statistically look for any relationship between the test set and faults. Results: Our results suggest that the fault detection rates of unit tests are relatively low, typically finding only about a half of all faults. In addition, conditional boundary and method call removals are less well detected by tests than other fault types. Conclusions: We conclude that the testing of these open source systems needs to be improved across the board. In addition, despite boundary cases being long known to attract faults, tests covering boundaries need particular improvement. Overall, we recommend that developers do not rely only on code coverage and mutation score to measure the effectiveness of their tests.

KW - software testing

KW - test effectiveness

KW - unit tests

KW - Fault detection

KW - Open source software

KW - Open systems

KW - Software reliability

KW - Code coverage

KW - Fault detection rate

KW - Fault types

KW - Mutation analysis

KW - Mutation score

KW - Open source system

KW - Software fault

KW - Test effectiveness

KW - Software testing

U2 - 10.1145/3383219.3383236

DO - 10.1145/3383219.3383236

M3 - Conference contribution/Paper

SN - 9781450377317

SP - 160

EP - 169

BT - PROCEEDINGS of EASE 2020: Evaluation and Assessment in Software Engineering

PB - ACM

CY - New York

ER -