Source-linked AI summary
Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair
Xuemeng Cai, Jiakun Liu, Linhan Yang, Wei Ma, Lingxiao Jiang
TL;DR
Existing APR evaluations focus mainly on final patches, leaving hallucination in intermediate repair artifacts insufficiently examined. This study analyzes both levels across three tasks and finds hallucinations remain prevalent, while artifact quality is associated with—but does not establish causation for—repair success.
Problem
Existing APR studies are largely result-centric, and final test outcomes provide incomplete evidence because passing patches may still be semantically incorrect or overfit available tests.
Method
The study analyzes repair hallucination in final patches and understanding hallucination in intermediate artifacts across three tasks on 832 Defects4J bugs.
Results
Hallucinations remain prevalent across repair and understanding tasks, including 21.0%–55.9% test-passing patches, 72.7% repair hallucinations, and frequent intermediate-artifact errors.
Takeaways & Limitations
Evaluating only final patches is insufficient; APR analysis should also examine intermediate artifacts and strengthen project grounding and execution evidence.
Takeaways & Limitations
The observed association between intermediate-artifact accuracy and repair success is correlational and does not establish that improving an artifact will improve repair success.
Abstract
from arXiv · showhide
Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence. We examine repair hallucination in final patches and understanding hallucination in intermediate artifacts through three tasks, namely triggering testcase identification, line coverage prediction, and additional testcase generation.We then evaluate three representative LLMs on 832 Defects4J bugs through automatic evaluation and manual analysis. Our results show that both repair and understanding hallucinations remain prevalent. Across models and settings, only 21.0%-55.9% of generated patches pass the developer-written test suite. Moreover, although more accurate intermediate artifacts are generally associated with successful repairs, this relationship does not always hold. Manual analysis of 812 sampled repairs identifies repair hallucinations in 72.7% of cases, including patches that pass all available tests; incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively. Meanwhile, models frequently misidentify triggering testcases, mispredict line coverage involving branching control flow, and generate additional testcases with missing bug-triggering conditions or incorrect expected behavior.
1 Introduction
The study addresses the limits of patch-centric APR evaluation by examining hallucination in final repairs and intermediate artifacts. Across three tasks and large-scale experiments, hallucinations remain prevalent, while artifact quality is associated with—but does not establish—repair success.
- Motivation: Existing APR evaluations focus mainly on final patches, providing limited insight into whether models understand and reason about buggy behavior.APR requires understanding buggy behavior, relating failures to execution, and synthesizing a patch.
- Study Design: Hallucination is analyzed as outputs not faithfully grounded in repair evidence, covering both final patches and intermediate artifacts.The intermediate artifacts include triggering-testcase identification, line coverage prediction, and additional testcase generation.
- Study Design: Three representative LLMs are evaluated on Defects4J through automatic evaluation and manual analysis of patches and intermediate artifacts.The study characterizes hallucination manifestations across APR tasks and models and examines potential contributing factors.
- Results: 21.0%–55.9% of generated patches pass the developer-written test suite, while manual analysis finds repair hallucinations in 72.7% of 812 sampled repairs.Incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively.
- Results: More accurate intermediate artifacts are generally associated with successful repairs, but this relationship does not establish that artifact improvements cause repair success.The study therefore treats intermediate-artifact quality as an additional signal rather than proof of correctness.
2 Related Work
Prior APR research demonstrates automated repair feasibility but struggles to distinguish plausible from semantically correct patches. Related work on hallucination, agent trajectories, and security motivates this study’s execution-grounded analysis of final patches and intermediate artifacts in general APR settings.
- Automated Program Repair: APR research spans search-, template-, constraint-, and learning-guided paradigms that demonstrate feasibility on real-world bugs.Examples include GenProg, PAR, TBar, SemFix, SPR, Prophet, CoCoNuT, and SelfAPR.
- Automated Program Repair: Passing the available test suite does not guarantee semantic correctness because developer-written tests may incompletely specify intended behavior.This makes final test outcomes incomplete evidence of repair correctness.
- LLM-based Repair: LLM-based APR has expanded through infilling, prompting, retrieval, natural-language interaction, and test feedback, but evaluations remain largely patch-centric.Patch-centric evaluation provides limited insight into whether repairs are grounded in the buggy program’s behavior.
- Hallucination: Code-intelligence studies define hallucination as plausible output that is not faithfully grounded in input or an external source of truth.Such hallucinations can conflict with repository context, task requirements, APIs, or program semantics.
- Positioning: This study complements trajectory-level and security-focused work by examining semantic faithfulness to execution-grounded evidence in general, non-adversarial APR.It evaluates whether coherent repair outputs are supported before final patch generation.
3 Methodology
The study distinguishes repair hallucination in final patches from understanding hallucination in intermediate artifacts, then evaluates both through three complementary APR tasks and human annotation.
- Conceptual organization: The methodology defines repair hallucination as an invalid or developer-inconsistent final patch and understanding hallucination as intermediate information inconsistent with execution-grounded evidence.Passing available tests does not exclude repair hallucination when patches overfit tests or deviate from intended semantics.
- Task design: The three tasks progress from identifying failure evidence, to reasoning about execution, to constructing new behavioral evidence.They cover triggering testcase identification, line coverage prediction, and additional testcase generation.
- Line coverage prediction: Line coverage prediction evaluates whether the model correctly identifies executed lines for both buggy and model-patched programs under triggering tests.The study compares predicted and observed execution-line sets using precision, recall, and F1, with F1 as the primary metric.
- Additional testcase generation: Additional testcase generation tests whether new tests expose the bug and remain consistent with the developer-written fixed behavior.The task also checks consistency with the model-generated patch, guarding against mutually consistent but incorrect repair semantics.
- Triggering testcase identification: Triggering testcase identification compares predicted triggers with developer-written tests that fail on the original buggy version.The model receives buggy functions and relevant tests, then produces predicted triggers and a repair patch.
4 Experimental Setup
The study evaluates three representative LLMs on Defects4J using a baseline repair setting and three task-specific APR settings that generate intermediate artifacts. It defines task inputs, execution-based evaluation, and human-annotation sampling across triggering-testcase identification, line-coverage prediction, and additional-testcase generation.
- Studied Models: Three representative LLMs—GPT-5, DeepSeek-R1, and Claude Sonnet 4.5—are evaluated for code generation, reasoning, and instruction following.The models were accessed through provider APIs in a zero-shot setting without fine-tuning or external tools.
- Dataset: Defects4J supplies buggy programs, developer-written fixes, and test suites for evaluating patches and intermediate artifacts.The selected bugs must compile, test successfully in the study environment, contain triggering testcases, and satisfy structural filtering requirements.
- Implementation and Evaluation: The automated pipeline constructs inputs, queries models, parses outputs, applies patches, executes tests, and collects line coverage in isolated environments.JaCoCo provides line-level coverage, and builds, tests, generated testcases, and coverage collection have 300-second timeouts.
- Task Design: The study compares baseline repair with triggering-testcase identification, line-coverage prediction, and additional-testcase generation.The baseline uses only developer-modified functions, whereas task-specific settings require intermediate artifacts alongside repair patches.
- Task Design: Triggering-testcase inputs use slices of at most 300 relevant methods, while line-coverage and additional-testcase tasks operate at the bug level without slicing.Generated additional testcases are treated as independent instances, even when they share a model-generated patch.
- Annotation: Human annotation samples cases from task-model groups, with population units defined as sliced instances, bug-level instances, or generated additional-testcase instances.Automatic evaluation is supplemented by manual inspection because execution outcomes alone do not explain how hallucinations manifest or why outputs are incorrect.
5 Results
Results show that hallucinations remain common in both final patches and intermediate APR artifacts. More accurate intermediate artifacts are generally associated with successful repairs, but passing tests do not guarantee faithful understanding.
- 21.0%–55.9% of repair attempts produce plausible patches across models and settings, with line coverage prediction achieving the highest rate for every model.The highest rates are 55.9% for GPT-5, 48.4% for DeepSeek, and 37.0% for Claude.
- GPT-5 consistently produces the most plausible patches, increasing from 219 baseline patches to 378–465 across the three task settings.DeepSeek increases from 175 to 310–403 plausible patches, while Claude improves more modestly from a baseline of 178.
- Passing repairs generally have more accurate intermediate artifacts, but overlaps between passing and failed repairs show that this association is not deterministic.For triggering-testcase identification, GPT-5 reaches F1 scores of 0.607 for passing repairs versus 0.214 for failed repairs; analogous gaps appear for Claude and DeepSeek.
- Valid generated triggering testcases account for only 30.5% of Claude’s outputs, 34.6% of DeepSeek’s, and 48.4% of GPT-5’s outputs.Pass repairs contain more valid triggering testcases than Fail repairs, but 45.7%–67.7% of Pass-associated generated testcases remain invalid across models.
- The study examines hallucinations in final repairs and intermediate APR artifacts through a taxonomy of observed hallucination types.The analysis treats hallucination as artifacts that appear plausible or coherent but are not faithfully grounded in available repair evidence.
6 Discussion
The discussion argues that APR evaluation should inspect intermediate artifacts alongside final patches, while strengthening grounding and execution evidence. It also documents validity threats and cautions that artifact quality is associated with, but does not establish, repair success.
- Implications: Intermediate artifacts and broader validation evidence should supplement final-patch testing in APR evaluation.The proposed artifacts can support patch-confidence estimates and stage-aware diagnosis of failures.
- Implications: Repository-aware retrieval, program analysis, coverage, slicing, and execution traces can strengthen project grounding and execution-path reasoning.These sources target hallucinations involving context, causal localization, and branch-dependent behavior.
- Implications: Intermediate artifacts should be treated as verifiable behavioral claims rather than inherently reliable explanations.The paper connects this view to generated tests, execution feedback, runtime traces, and developer-facing explanations.
- Implications: Different hallucination categories require different validation mechanisms, including compilation checks, dynamic analysis, additional testing, and semantic comparison.The taxonomy is intended to support failure-specific validation and model-improvement pipelines.
- Implications: Generated testcases may preserve regressions even when they do not reveal a bug, whereas tests passing only on a model patch may expose overfitting.Thus, testcase outcomes require interpretation beyond simple bug-revelation labels.
- Threats to Validity: The study uses one response per task instance, samples rather than exhaustively annotates artifacts, and evaluates three models on the Java Defects4J benchmark.These choices constrain reproducibility, conclusion validity, model generalization, and external validity.
- Threats to Validity: Manual interpretation may introduce annotation bias, while implementation and parsing errors may also affect evaluation despite mitigation procedures.The authors use independent annotation, codebook refinement, agreement measurement, isolated environments, and execution logs.
- Threats to Validity: Developer-written tests and generated-test validity criteria do not guarantee complete repair semantics, so passing or bug-revealing outcomes can remain incomplete.A patch may pass insufficient tests, and a generated testcase may reveal a bug without capturing the full intended behavior.
7 Conclusion
The conclusion frames hallucination in APR as affecting both final patches and intermediate artifacts, evaluated across three tasks and 832 Defects4J bugs. Hallucinations remain prevalent, with frequent repair hallucinations and substantial contributions from incorrect causal localization and repair strategy.
- Conclusion: The study evaluates repair hallucination in final patches and understanding hallucination in intermediate artifacts across three tasks on 832 Defects4J bugs.The tasks are triggering testcase identification, line coverage prediction, and additional testcase generation.
- Conclusion: Hallucinations remain prevalent at both repair and understanding levels across the evaluated models and tasks.The conclusion reports patch-passing, triggering-testcase identification, and additional-testcase validity ranges.
- Conclusion: 21.0%–55.9% of generated patches pass the developer-written test suite, while exact triggering-testcase identification ranges from 23.0% to 40.9%.Only 30.5%–48.4% of generated additional testcases are valid.
- Conclusion: GPT-5 generally exhibits the lowest tendency toward hallucination, although accurate intermediate artifacts are not always accompanied by successful repairs.This conclusion preserves the reported association without treating it as deterministic.
- Conclusion: 72.7% of 812 manually analyzed repair outputs contain repair hallucinations.The analysis identifies 590 hallucinated cases, including patches that may pass available tests.
- Conclusion: Incorrect Causal Localization and Incorrect Repair Strategy account for 45.9% and 18.5% of repair hallucinations, respectively.The conclusion also identifies insufficient project-context grounding, imprecise causal localization, and unstable execution-path reasoning as potential contributors.
Statements and Declarations
The statements and declarations report funding, author contributions, data availability, and competing-interest information. A replication package containing prompts, results, labels, and code is publicly available.
- Funding: The research is supported by Singapore’s Ministry of Education Academic Research Fund Tier 3, Award ID MOET32020-0004.The funding statement notes that the expressed opinions and conclusions belong to the authors.
- Author Contributions: The authors report contributions spanning study design, methodology, experiments, analysis, annotation, validation, and manuscript preparation.The statement assigns distinct contributions across the listed authors.
- Data Availability: A replication package containing prompts, experimental results, annotation labels, and code is provided to support reproducibility.The package is described in the data availability statement.
- Data Availability: The replication package is publicly available through the listed GitHub repository.The statement provides the repository link for access.
- Competing Interests: The authors declare no competing interests relevant to the article.