Source-linked AI summary
Automated Program Repair: Emerging trends pose and expose problems for benchmarks
Joseph Renzullo, Pemma Reiter, Westley Weimer, Stephanie Forrest
TL;DR
ML-based APR evaluations face inconsistent reporting and benchmarks that raise concerns about comparison and generalization. This review examines ML approaches, training-test interactions, and benchmark issues, concluding that leakage threatens validity and that larger studies alone will not resolve evaluation problems.
Problem
Different APR studies use different correctness criteria, making results difficult to compare, while benchmark designs can constrain what evaluations measure.
Source: §1 · p. 1, §6.1 · p. 9Method
The authors review recent APR research from five leading software-engineering venues and examine ML approaches, data interactions, benchmarks, and standardization proposals.
Source: §1.1 · p. 2, §1 · p. 2Results
The review identifies data leakage as a threat to ML-based APR result validity and finds that varied reporting makes comparisons challenging or infeasible.
Source: §7 · p. 10Takeaways & Limitations
The authors conclude that larger empirical studies and comparisons will not shed light until problems inherent in APR evaluation are addressed.
Source: §7 · p. 10Because GPT training corpora are unavailable, third parties cannot directly apply OpenAI’s decontamination policy to new benchmarks.
Source: §4.3 · p. 6
Abstract
from arXiv · showhide
Machine learning (ML) now pervades the field of Automated Program Repair (APR). Algorithms deploy neural machine translation and large language models (LLMs) to generate software patches, among other tasks. But, there are important differences between these applications of ML and earlier work. Evaluations and comparisons must take care to ensure that results are valid and likely to generalize. A challenge is that the most popular APR evaluation benchmarks were not designed with ML techniques in mind. This is especially true for LLMs, whose large and often poorly-disclosed training datasets may include problems on which they are evaluated.
1 INTRODUCTION
APR research uses varied techniques and evaluation practices, making results difficult to compare; this review examines ML in APR and the challenges posed by data leakage and benchmarks. It reviews work from five leading software-engineering venues published between January 2018 and September 2023.
1 INTRODUCTION
- Different correctness criteria—from passing tests to semantic equivalence with a developer fix—make results difficult to compare.VFix, VRepair, and SynFix use different criteria for correctness.Source: §1 · p. 1
- APR evaluations also differ in fault-localization assumptions and stopping rules, including stopping after one solution or when a time budget expires.Sequencer assumes perfect line-level localization, while iFixR uses human-generated bug reports.Source: §1 · p. 2
- The review surveys recent ML applications to APR, data interactions between training and testing, benchmark roles, and proposals for competitions and standardization.The introduction notes that benchmarks alone do not standardize evaluation.Source: §1 · pp. 1–2
1.1 Methodology
- The authors reviewed APR papers from five leading software-engineering venues published from January 1, 2018, through September 1, 2023, identifying 118 papers.Their search combined metadata keywords with manual abstract review; 65 identified papers were cross-listed in Monperrus’s living review.Source: §1.1 · p. 2
2 BACKGROUND
This background introduces transformer and language-model concepts alongside data bias, data leakage, and beam search as context for ML-based APR.
2.1 Transformer Models and Atention
- Transformer encoder-decoder models represent inputs in a latent space and generate outputs; attention relates inputs and outputs when feature positions are unknown.Attention is relevant to APR because source-code inputs vary in structure and length.Source: §2.1 · p. 2
2.2 Language Models
- PLMs are pretrained on large corpora and can be fine-tuned, while LLMs are very large language models that are being adopted in APR.The passage characterizes LLMs as usually having a billion or more parameters.Source: §2.2 · p. 3
2.3 Data Bias
- Data bias can distort ML results when dataset features are unequally represented, but its manifestation in APR remains understudied.The reviewed papers did not explicitly discuss data bias or its handling.Source: §2.3 · p. 3
2.4 Data leakage
- Data leakage includes training information relevant to prediction and duplicated or near-duplicated content, and can overestimate performance on unseen data.Duplication can occur at sentence, document, and dataset levels for language, and at statement, function, class, and file levels for code.Source: §2.4 · p. 3
2.5 Beam Search
- Beam search uses beam size k to control search breadth by retaining the k remaining sequences with the largest output weights at each evaluation step.Source: §2.5 · p. 3
3 DIVERSE ML APPROACHES PERVADE APR
The review finds ML increasingly prevalent in APR and describes shifts in language-model use and architecture. It also notes variation in how transformer-based tools report attention mechanisms.
3 DIVERSE ML APPROACHES PERVADE APR
- 29% of the 118 reviewed APR papers list ML-related keywords; fewer than one in six did so in 2018, compared with more than three in four in 2023.Source: §3 · p. 3
3.1 The Rise of LLMs
- LLMs are becoming popular in APR, usually for patch generation, amid interest in their apparent emergent capabilities.The reviewed passage notes that these capabilities and their contributing factors are not well understood.Source: §3.1 · p. 4
3.2 Code-Specific Language Models
- Specialized code language models trained on source code have shown promise in code completion and program synthesis, and preliminary work applies them to APR.The passage names CodeBERT, CodeT5, and PLBART among models used in APR tools.Source: §3.2 · p. 4
3.3 Architectures
- 20 of 29 reviewed ML papers use transformers; APR’s ML paradigm has shifted from NMT translation toward LLM gap-filling.The earlier approach translates buggy code into repaired code; the described LLM approach deletes a buggy line and generates code to fill the gap.Source: §3.3 · pp. 4–5
3.4 Atention Mechanisms
- Transformer-based APR tools vary in how they describe attention: one omits a specific model, five explain it in detail, and others provide briefer accounts or inherit an NMT model.The review suggests this variability may reflect the lack of accepted attention taxonomies.Source: §3.4 · p. 5
4 DATA LEAKAGE CONTAMINATES MODELS AND BENCHMARKS
The review finds uneven handling of leakage and contamination in ML-based APR, while benchmark contamination can be difficult to detect or remove. It also highlights near-duplicate code as a threat to evaluation and generalization.
4.1 Data Leakage
- APR studies use temporal, project-based, and random train-test segmentation, but approaches vary, and RewardRepair does not segment or decontaminate its syntactic training content.SelfAPR and CURE use temporal boundaries; CURE trains on pre-2006 projects and manually removes duplicate bugs.Source: §4.1 · p. 5
4.2 Contamination
- Five reviewed papers using LLMs for APR do not consider data leakage or contamination, while three others report deduplicating their corpora using syntax, semantics, or an unspecified method.CURE’s policy assumes independence across GitHub projects and between commits and repairs; SelfAPR checks samples individually and retains unique ones.Source: §4.2 · p. 6
4.3 Decontaminating Benchmarks
- Decontamination or even contamination measurement is often unavailable: training corpora may be undisclosed, and API access can prevent after-the-fact entropy-gap analysis.GPT-3 used n-grams and GPT-4 multisample substring matching, but third parties cannot directly apply GPT’s policy without its training corpora.Source: §4.3 · p. 6
4.4 Memorization and Generalization
- Memorization means generating training sequences verbatim without capturing their meaning; duplicate or near-duplicate removal reduces verbatim memorization on average.Memorization may help repair bugs seen during training but is problematic when a model is expected to generalize.Source: §4.4 · p. 6
- Few language models provide a package that detects contaminated test samples and produces a clean benchmark; proposed approaches include canary strings and perturbing names during validation.The review also notes that file-level code deduplication can leave duplicated excerpts, such as functions, intact; small-fragment translation methods may be especially susceptible.Source: §4.4 · p. 6, §4.5 · p. 7
4.6 Impact of Near-Duplicates on Code Tasks
- Near-duplicate code in training and benchmark data can inflate performance metrics by up to 100% compared with de-duplicated corpora.Specialized source-code similarity techniques perform better than general textual similarity measures.Source: §4.6 · p. 7
5 BENCHMARKS INSUFFICIENTLY STANDARDIZE EVALUATIONS
The review finds that APR benchmark evaluations vary in reporting, task assumptions, correctness criteria, and data resources. Build and search-efficiency constraints further limit reproducibility and comparison.
5.1 Performance Reporting
- Many APR papers omit repair run-time details or report incompatible efficiency measures, limiting comparisons across benchmark evaluations.ManyBugs papers usually report run-time alongside repair success, while Defects4J reporting varies or is omitted.Source: §5.1 · p. 7
5.2 Fault Localization
- Full automated repair includes fault localization and solution prioritization, unlike patch generation alone; papers do not report these assumptions uniformly.Some evaluations assume perfect fault localization, while others estimate the defect location.Source: §5.2 · pp. 7–8
5.3 Correctness
- Correctness proxies often compare patches with one human repair, restricting acceptance to solutions resembling that example despite other ways to satisfy functionality.A single reference repair makes correctness unusually strict given that functionality can be implemented in many ways.Source: §5.3 · p. 8
5.4 APR Benchmarks Systematically Differ from ML Datasets
- APR benchmarks have too few curated bug-fix pairs to train ML tools, while larger ML datasets generally lack replicable build environments and test suites.ML datasets can be large enough for training but are not curated to the same extent as APR benchmarks.Source: §5.4 · p. 8
5.5 Access to source code, bug information, and repair information
- Benchmarks often link to public repositories but cannot feasibly contain full development histories, which some APR tools need for historical information retrieval.CodeFlaws and IntroClass are exceptions because their histories are short enough to include.Source: §5.5 · p. 8
5.6 Reproducible program builds
- Five sampled papers report results on ManyBugs, but only two present new evaluations; 44 report results on Defects4J, whose Maven infrastructure contrasts with ManyBugs’ incomplete build support.Some authors unable to execute ManyBugs used syntactic comparison with the human repair as an evaluation proxy.Source: §5.6 · p. 9
5.7 Consistent reporting measures
- NPR techniques lack a standard search-efficiency metric, and beam size alone omits generative vocabulary size and sequence length.Vocabulary size is not always reported, making comparisons difficult; effective search sizes vary widely across models.Source: §5.7 · p. 9
6 DISCUSSION
StandUp4NPR standardizes access, resources, runtime, and evaluation, but its data restrictions and no-feedback design constrain what it can compare. APR-COMP 2024 offers common access and resource limits, while its functional-repair track inherits existing benchmark issues.
6.1 Standup4NPR
- StandUp4NPR processes existing datasets and benchmarks with filtering intended to minimize exact duplicates.Its standardized interface can impose resource restrictions to facilitate tool comparisons.Source: §6.1 · p. 9
- StandUp4NPR standardizes data access, resource usage, runtime, and evaluation criteria, but restricting inputs to its training examples excludes independent-data curation and auxiliary inputs.Its comparisons therefore focus on how effectively architectures use the supplied training data.Source: §6.1 · p. 9
- StandUp4NPR assumes patches are generated simultaneously and evaluated independently, so search-based techniques that rely on feedback cannot be compared under its framework.GenProg and ARJA use mutation performance to affect later steps.Source: §6.1 · p. 10
6.2 APR-COMP 2024
- APR-COMP 2024's functional-repair track samples existing benchmarks, inheriting their issues and making its content knowable to model designers in advance.The competition also provides a standard API, common evaluation-time and resource limits, and public-pull-request submissions that require an executable object.Source: §6.2 · p. 10
7 CONCLUSION
The conclusion argues that popular APR benchmarks are aging as ML approaches rise, while data leakage and inconsistent reporting challenge evaluation. It cautions that larger empirical studies will not clarify tool success until these evaluation problems are addressed.
7 CONCLUSION
- The rise of ML approaches in APR poses problems for evaluating popular benchmarks that have concentrated the field's evaluation effort.The paper describes APR as fast-evolving and notes its increasing industrial uptake.Source: §7 · p. 10
- Data leakage threatens result validity, especially for language models with unavailable or undisclosed training corpora, while varied reporting makes cross-evaluation comparisons difficult or infeasible.The authors conclude that more and larger empirical studies and comparisons will not shed light until these evaluation problems are addressed.Source: §7 · p. 10