Home > Research > Publications & Outputs > Some automatically generated patches are more l...

Research

Electronic data

  • Some Automatically Generated Patches are More Likely to be Correct than Others An Analysis of Defects4J Patch Features

    Accepted author manuscript, 436 KB, PDF document

    Available under license: CC BY: Creative Commons Attribution 4.0 International License

Links

Text available via DOI:

View graph of relations

Some automatically generated patches are more likely to be correct than others: an analysis of Defects4J patch features

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Conference contribution/Paper › peer-review

Published

Standard

Some automatically generated patches are more likely to be correct than others: an analysis of Defects4J patch features. / Bennett, Gaz; Hall, Tracy; Bowes, David.
Proceedings - International Workshop on Automated Program Repair, APR 2022. New York: ACM, 2022. p. 46-52.

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Conference contribution/Paper › peer-review

Harvard

Bennett, G, Hall, T & Bowes, D 2022, Some automatically generated patches are more likely to be correct than others: an analysis of Defects4J patch features. in Proceedings - International Workshop on Automated Program Repair, APR 2022. ACM, New York, pp. 46-52, APR '22: Proceedings of the Third International Workshop on Automated Program Repair, Pittsburgh, Pennsylvania, United States, 19/05/22. https://doi.org/10.1145/3524459.3527348

APA

Bennett, G., Hall, T., & Bowes, D. (2022). Some automatically generated patches are more likely to be correct than others: an analysis of Defects4J patch features. In Proceedings - International Workshop on Automated Program Repair, APR 2022 (pp. 46-52). ACM. https://doi.org/10.1145/3524459.3527348

Vancouver

Bennett G, Hall T, Bowes D. Some automatically generated patches are more likely to be correct than others: an analysis of Defects4J patch features. In Proceedings - International Workshop on Automated Program Repair, APR 2022. New York: ACM. 2022. p. 46-52 Epub 2022 May 19. doi: 10.1145/3524459.3527348

Author

Bennett, Gaz ; Hall, Tracy ; Bowes, David. / Some automatically generated patches are more likely to be correct than others : an analysis of Defects4J patch features. Proceedings - International Workshop on Automated Program Repair, APR 2022. New York : ACM, 2022. pp. 46-52

Bibtex

@inproceedings{fb2fecbb6a1046afae52469032afa130,
title = "Some automatically generated patches are more likely to be correct than others: an analysis of Defects4J patch features",
abstract = "Defects4J is a popular dataset against which many Java Automatic Program Repair (APR) tools benchmark their performance. However, recent evidence suggests that some APR tools overfit to Defects4J, producing plausible patches which are incorrect. What we do not currently know is whether there is any commonality in the features of these plausible patches that turn out not to be correct. We compare the features of Defects4J's human written patches in terms of those correctly patched by existing APR tools and those incorrectly patched. We found that 48.4% of Defects4J v1.5 have been automatically patched by existing APR tools; of which only 28.9% have been correctly patched leaving 19.5% incorrectly patched. We found patches of defects that added a method call, added a variable, or wrapped existing code with new code, such as a try/catch block were significantly associated with incorrect patches. Editing only a single line was significantly associated with correct patches. Our results suggest that current tools are weak at generating multi-line patches and synthesising new code especially when wrapping existing code. Our results highlight potential future areas of development for new APR approaches, such as developing a tool that effectively repairs defects that require a try/catch block. Our replication Package is available online11Replication Package available at: https://github.com/IncorrectDefects/ReplicationPackage.",
author = "Gaz Bennett and Tracy Hall and David Bowes",
year = "2022",
month = oct,
day = "26",
doi = "10.1145/3524459.3527348",
language = "English",
pages = "46--52",
booktitle = "Proceedings - International Workshop on Automated Program Repair, APR 2022",
publisher = "ACM",
note = "APR '22: Proceedings of the Third International Workshop on Automated Program Repair, APR '22 ; Conference date: 19-05-2022",
url = "https://dl.acm.org/doi/proceedings/10.1145/3524459",

}

RIS

TY - GEN

T1 - Some automatically generated patches are more likely to be correct than others

T2 - APR '22: Proceedings of the Third International Workshop on Automated Program Repair

AU - Bennett, Gaz

AU - Hall, Tracy

AU - Bowes, David

PY - 2022/10/26

Y1 - 2022/10/26

N2 - Defects4J is a popular dataset against which many Java Automatic Program Repair (APR) tools benchmark their performance. However, recent evidence suggests that some APR tools overfit to Defects4J, producing plausible patches which are incorrect. What we do not currently know is whether there is any commonality in the features of these plausible patches that turn out not to be correct. We compare the features of Defects4J's human written patches in terms of those correctly patched by existing APR tools and those incorrectly patched. We found that 48.4% of Defects4J v1.5 have been automatically patched by existing APR tools; of which only 28.9% have been correctly patched leaving 19.5% incorrectly patched. We found patches of defects that added a method call, added a variable, or wrapped existing code with new code, such as a try/catch block were significantly associated with incorrect patches. Editing only a single line was significantly associated with correct patches. Our results suggest that current tools are weak at generating multi-line patches and synthesising new code especially when wrapping existing code. Our results highlight potential future areas of development for new APR approaches, such as developing a tool that effectively repairs defects that require a try/catch block. Our replication Package is available online11Replication Package available at: https://github.com/IncorrectDefects/ReplicationPackage.

AB - Defects4J is a popular dataset against which many Java Automatic Program Repair (APR) tools benchmark their performance. However, recent evidence suggests that some APR tools overfit to Defects4J, producing plausible patches which are incorrect. What we do not currently know is whether there is any commonality in the features of these plausible patches that turn out not to be correct. We compare the features of Defects4J's human written patches in terms of those correctly patched by existing APR tools and those incorrectly patched. We found that 48.4% of Defects4J v1.5 have been automatically patched by existing APR tools; of which only 28.9% have been correctly patched leaving 19.5% incorrectly patched. We found patches of defects that added a method call, added a variable, or wrapped existing code with new code, such as a try/catch block were significantly associated with incorrect patches. Editing only a single line was significantly associated with correct patches. Our results suggest that current tools are weak at generating multi-line patches and synthesising new code especially when wrapping existing code. Our results highlight potential future areas of development for new APR approaches, such as developing a tool that effectively repairs defects that require a try/catch block. Our replication Package is available online11Replication Package available at: https://github.com/IncorrectDefects/ReplicationPackage.

U2 - 10.1145/3524459.3527348

DO - 10.1145/3524459.3527348

M3 - Conference contribution/Paper

SP - 46

EP - 52

BT - Proceedings - International Workshop on Automated Program Repair, APR 2022

PB - ACM

CY - New York

Y2 - 19 May 2022

ER -