Source-linked AI summary
You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems
Kui Liu, Anil Koyuncu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, Yves Le Traon
TL;DR
APR benchmarking can be biased because fault localization determines whether repair systems can reach buggy code, yet published comparisons often omit important FL details. The paper analyzes FL configurations and localizability on Defects4J, evaluates configuration effects with kPAR, and finds that many bugs are not localized and that fixed-bug comparisons can therefore be misleading.
Problem
APR comparisons may be biased when studies omit fault-localization implementation details, even though FL determines whether systems can attempt the correct patch.
Method
The study audits FL configurations, measures Defects4J fault localizability, evaluates configuration changes with kPAR, and provides FL and benchmark resources for comparison.
Results
Approximately 62% (246/395) of Defects4J bugs had never received a plausible automatically generated patch from state-of-the-art APR tools, and FL configurations can bias comparisons.
Takeaways & Limitations
APR performance reports should disclose FL procedures and compare systems with respect to bugs that can actually be localized.
Takeaways & Limitations
The study focuses on Java bugs in Defects4J and uses the single automatic testing framework GZoltar, although these choices are common in the APR literature.
Abstract
from arXiv · showhide
Properly benchmarking Automated Program Repair (APR) systems should contribute to the development and adoption of the research outputs by practitioners. To that end, the research community must ensure that it reaches significant milestones by reliably comparing state-of-the-art tools for a better understanding of their strengths and weaknesses. In this work, we identify and investigate a practical bias caused by the fault localization (FL) step in a repair pipeline. We propose to highlight the different fault localization configurations used in the literature, and their impact on APR systems when applied to the Defects4J benchmark. Then, we explore the performance variations that can be achieved by `tweaking' the FL step. Eventually, we expect to create a new momentum for (1) full disclosure of APR experimental procedures with respect to FL, (2) realistic expectations of repairing bugs in Defects4J, as well as (3) reliable performance comparison among the state-of-the-art APR systems, and against the baseline performance results of our thoroughly assessed kPAR repair tool. Our main findings include: (a) only a subset of Defects4J bugs can be currently localized by commonly-used FL techniques; (b) current practice of comparing state-of-the-art APR systems (i.e., counting the number of fixed bugs) is potentially misleading due to the bias of FL configurations; and (c) APR authors do not properly qualify their performance achievement with respect to the different tuning parameters implemented in APR systems.
I. INTRODUCTION
APR repairs semantic bugs through a pipeline of fault localization, patch generation, and patch validation, but comparisons can be biased when fault-localization details differ. The paper investigates these differences, bug localizability, and performance variation under alternative configurations, contributing tools, benchmarks, and baseline results for more reliable assessment.
- APR pipeline: Semantic-bug repair requires fault localization, followed by patch generation and validation in an iterative APR pipeline.APR systems revisit generation and validation across suspicious locations until a valid patch is found or termination occurs.
- Fault-localization bias: Fault-localization accuracy can boost or degrade APR performance because a system cannot generate the correct patch without identifying the buggy code locations.The paper highlights differences in FL versions, configurations, and assumptions, including experiments where the buggy method is known.
- Research questions: The study examines how APR systems use fault localization, how many benchmark bugs are localizable, and how performance varies under different FL configurations.It inspects tools, formulas, granularity, and the number of suspicious locations, then evaluates kPAR as a comparable baseline.
- Research questions: 246 Defects4J bugs had not been fixed by any state-of-the-art APR tool, motivating investigation of whether fault localization contributes to repair misses.The authors frame this as a practical localizability issue rather than only a limitation involving difficult bugs.
- Contributions: The contributions include exposing comparison bias, releasing a configurable Java FL toolkit, refining the benchmark to localizable bugs, and publishing kPAR results across FL configurations.These resources are intended to support more reliable comparisons and fuller reporting of APR performance.
II. BACKGROUND
Fault localization identifies suspicious program entities from test executions and ranks them for APR. The background describes spectrum-based localization, the widespread use of Ochiai, and the resulting convention of assessing tools by reported bug-fix counts.
- Fault localization: Fault localization is the first APR step and identifies suspicious files, methods, or lines whose rankings guide patch generation.Broader granularity increases the candidate space, while precise and accurate locations are preferred.
- Spectrum-based fault localization: Spectrum-based fault localization ranks program entities by suspiciousness scores computed from test-execution traces.The technique is also called coverage-based fault localization and is widely used in APR systems.
- APR assessment: APR studies commonly assess performance by counting bugs for which tools generate patches that pass all provided test cases.This reported-fix-count practice is the basis of comparisons examined later in the paper.
- Ochiai: Ochiai is widely used as the spectrum-based ranking metric, and empirical studies report it as one of the most effective techniques for object-oriented fault localization.The paper notes that 11 of 14 APR tools explicitly rely on Ochiai.
- Ochiai: Ochiai’s suspiciousness calculation uses counts of failing and passing tests that execute a statement and failing tests that do not execute it.Fault-localization tools use these scores to produce a ranked list of suspicious statements.
B. APR Performance Assessment
APR performance is commonly summarized by the number of bugs whose generated patches pass all given tests, but this measure can be biased by fault-localization choices and can overstate patch effectiveness. The paper therefore grounds its assessment in explicit experimental procedures and FL-aware comparisons.
- Performance measure: A patch is counted as successful when it makes the program pass all supplied passing and failing test cases.The literature commonly reports APR performance using the number of bugs with such patches.
- Performance measure: Passing all tests does not establish patch correctness because tests may miss broken behavior or newly introduced faults, and developers may reject patches for other reasons.Such patches are called plausible; correctness requires further investigation, often by comparison with the developer patch.
- Fault-localization bias: Fault-localization technique selection can bias performance assessment, so the study uses reported fixed-bug counts as a starting point for examining this problem.Table II compiles results from current state-of-the-art APR tools on Defects4J, with sources varying across tools.
- Experiment design: The experiments use common tools and processes from the literature to establish the basis for the study’s implementation and conclusions.This design choice supports comparison with existing APR assessments.
A. Definition of Fault Locality
The study defines fault locality at file, method, and line granularities by matching ranked suspicious lines against bug locations. It represents bugs and suspicious lines explicitly to determine whether and where an FL tool identifies the fault.
- Granularity: Fault locality is evaluated at the file, method, and line levels.File-level matches any line in the buggy file; method-level matches any line in the buggy method; line-level matches a buggy line.
- Localization criterion: A bug is localized when any reported suspicious line matches a ground-truth bug location at the selected granularity.The study records the matching location's rank in the ordered suspicious-location list.
- Bug representation: Each bug location is represented as a tuple containing a file name, methods, and line numbers.The representation accommodates bugs outside methods, including type or field declaration faults.
- Suspicious-line representation: Each suspicious line is represented by a file name, line number, and rank within the suspiciousness-ordered list.Localization uses the earliest matching suspicious line in that ranked list.
- Experimental basis: The experiments use Defects4J because it provides buggy Java programs, test cases, and developer fixes, with 395 bugs and 22,954 test cases.GZoltar executes the tests and produces coverage matrices for fault-localization ranking; experiments compare versions 0.1.1 and 1.6.0.
D. Implementation of a Baseline APR System
The paper implements kPAR as a shareable baseline APR system and examines how fault-localization choices and supplementary information vary across state-of-the-art tools. These variations are relevant because prior comparisons often leave their impact on repair performance unexamined.
- Motivation: Existing APR systems are difficult to reuse as references because code and implementation details are often unavailable, repair steps are coupled, and contributions are hard to isolate.These constraints motivate implementing a simpler baseline system.
- Baseline implementation: The authors build and share kPAR as a reproducible baseline based on PAR's repair patterns.kPAR implements six patterns learned from 60,000 human-written patches and is evaluated under four fault-localization configurations.
- FL configurations: Among 14 studied APR tools, 10 use GZoltar, and 11 explicitly use Ochiai to compute fault-localization suspiciousness.Most tools use GZoltar 0.1.1, while SimFix uses a recent framework version.
- Fault granularity: Only LSRepair operates at method-level fault locality, whereas other systems require line-level fault information for patch generation.Method-level localization may exclude faults outside methods but can reduce the number of suspicious statements considered.
- Supplementary information: Four APR systems use supplementary information to assist fault localization, but comparisons do not discuss its performance impact.Examples include known faulty methods, stack traces, and predicate switching.
- Assessment: State-of-the-art tools add adaptations to improve fault-localization accuracy, yet their contribution to overall repair performance is not properly qualified.The paper identifies this omission as a source of biased comparisons.
B. Localizability of Defects4J Bugs
Defects4J localizability varies substantially with the fault-localization configuration and ranking depth. The commonly used setup misses about one third of bugs, while newer tooling and broader ranked lists expose additional repair opportunities.
- Overall repair coverage: ∼62% (246/395) of Defects4J bugs had never received a plausible automatically generated patch from the state of the art considered.The reported total includes results from prior APR systems and LSRepair; 136 bugs had plausible patches and 83 had patches validated as correct.
- GZoltar versions: Up to 132 of 395 bugs are not localized with GZoltar 0.1.1 at line granularity, compared with 74 using GZoltar 1.6.0.The newer version therefore provides an opportunity to attempt fixes for 58 more bugs.
- Main finding: One third of Defects4J bugs cannot be localized by the commonly used automated fault-localization tool.The recent GZoltar version supplies coverage information that helps localize more than 50 additional bugs.
- Ranking metrics: Fault-localization performance is consistent across the different ranking metrics evaluated.The study reports results at Top-1 and Top-10, with additional rankings through Top-200 and all localized cases available in the replication package.
- Ranking depth: 45 bugs are localized by Ochiai at Top-1, 140 at Top-10, and 214 at Top-100.APR tools may process only part of the ranked list; examples include Top-50 for SketchFix and Top-200 for ELIXIR.
C. Impact of Effective Ranking in Fault Localization
The study examines whether fault-localization ranking positions correlate with APR repair success and finds that correctly fixed bugs are generally localized more accurately. It also shows that FL configurations and undocumented adaptations can make cross-system comparisons misleading.
- Localization accuracy and repair outcomes: Correctly fixed bugs are more accurately localized than overfitting-fixed or unfixed bugs across file, method, and line granularities.Unfixed bugs tend to be poorly localized even at file level.
- Localization accuracy and repair outcomes: APR tools are prone to correctly fixing the subset of Defects4J bugs that can be accurately localized.
- Localization configurations: Some reported fixes for non-localizable bugs rely on improved FL, targeted localization, coarser granularity, or non-explicit localization processes.Examples include SimFix using GZoltar 1.6.0, HDRepair assuming the faulty method, and LSRepair operating at method level.
- Benchmarking implications: APR systems do not fully disclose their fault-localization tuning parameters, preventing reliable replication and comparison.
- Benchmarking implications: When comparisons are restricted to localizable bugs, state-of-the-art APR systems can correctly fix fewer bugs than reported in the literature.The study therefore proposes comparing performance on a common set of localizable bugs.
D. Evaluating kPAR with Specific FL Configurations
The study evaluates kPAR under progressively more constrained fault-localization assumptions to isolate how localization affects repair performance. kPAR fixes more bugs as accurate locations are supplied, while multi-location and partially passing-test scenarios expose additional repair trade-offs.
- Configurations: kPAR is evaluated on Defects4J using Normal FL, File Assumption, Method Assumption, and Line Assumption configurations.The latter configurations assume increasingly specific knowledge of the faulty file, method, or line.
- Configuration results: 36 bugs are correctly fixed by kPAR with Line Assumption, its maximum performance across the tested configurations.This result is described as a record performance in the literature without accounting for FL bias.
- Configuration results: File Assumption adds four bugs beyond Normal FL, Method Assumption adds Cl-10, and Line Assumption adds 13 further bugs.
- Repair trade-offs: Accuracy of fault localization has a direct and substantial impact on APR repair-pipeline performance.For multi-location Chart-14, fixing one location still leaves failed tests, so the patch is not considered plausible.
- Repair trade-offs: Partial patches can prevent a fully correct patch, while multiple faulty locations can prevent kPAR from completing a full fix when faults share a failing test.The study identifies a trade-off between repairing multi-location bugs and bugs failing multiple test cases.
V. DISCUSSION
The discussion consolidates the study's conclusions into assessment guidelines and identifies threats to validity before turning to related work.
- Discussion: The study's conclusions are reformulated into guidelines for assessing APR systems.
- Discussion: The authors enumerate associated threats to validity before discussing related work.
- Discussion: The discussion follows the study's conclusions with validity assessment and related-work discussion.
A. APR Assessment Guidelines
The paper proposes guidelines for making APR evaluations more transparent, comparable, and informative about the roles of fault localization and patch generation. It distinguishes assessing patch generation under perfect localization from assessing realistic end-to-end repair.
- Transparency and comparison: APR reports should fully disclose fault-localization parameters and preferably use a standard, replicable localization configuration.
- Transparency and comparison: Authors should qualify APR performance gains by identifying the contributions of the different ingredients in their approaches.
- Evaluation scope: Patch-generation contributions can be assessed under perfect fault localization, whereas realistic industry-oriented evaluation should test the full pipeline without localization assumptions.
- Evaluation scope: APR studies should characterize whether exact fault locations are necessary or whether surrounding context and heuristics can support patch generation.
B. Threats to Validity
The study’s validity is constrained by its focus on Defects4J’s Java bugs and by methodological choices that may not represent all APR systems. The authors argue these threats are limited because the study compares performance differences and uses choices common in APR research.
- External validity: The study focuses on Defects4J bugs targeting Java, whose test suites may be insufficient for localizing faults.This limits the study’s external validity.
- Internal validity: Using only GZoltar threatens internal validity because not all APR systems use it for fault localization.The authors treat the choice as mitigated by its prevalence in the APR literature.
- Internal validity: Selecting 14 state-of-the-art APR systems also constrains internal validity, although the selection reflects systems commonly studied in APR research.The authors present commonality with the literature as a mitigation.
C. Related Work
Prior work examined fault localization metrics and fault-space effects, but did not systematically connect fault localization techniques to APR performance across many systems. This study addresses that gap and calls for fairer, more reproducible benchmarking practices.
- Prior fault-localization studies: Earlier work studied fault localization effectiveness using APR performance as a proxy and found Jaccard outperforming other ranking metrics.The present study notes that Ochiai remains the common technique used in APR.
- Fault-space studies: Fault-space research examined plausible and correct patches but was limited to GenProg and the Ochiai technique.Fault space is defined as a ranked list of suspicious program entities.
- This study: This study compares 14 APR systems and examines exact fault locations in relation to the possibility of generating plausible patches.It further targets biases among APR systems.
- Novelty: The authors describe their work as the first systematic study of how fault-localization techniques affect automated program-repair pipeline performance.This positions the study as extending prior work on fault space and localization metrics.
- Contribution: The replication package provides a Defects4J fault-localization dissection, a tunable toolkit, and a baseline Java APR system to support fair and reproducible experiments.The authors also call for guidelines covering APR performance assessment and reporting.