PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair

Xuemeng Cai, Jiakun Liu, Linhan Yang, Wei Ma, Lingxiao Jiang

arXiv:2609.04909v1cs.SEcs.AI

TL;DR

Existing automated program repair evaluations focus largely on final patches, leaving limited insight into hallucination in the intermediate artifacts guiding repair. This study analyzes both layers across three tasks and 832 Defects4J bugs, finding hallucinations prevalent, with 72.7% of 812 sampled repairs exhibiting repair hallucination.

  • Problem

    Existing automated program repair evaluations are largely result-centric, providing limited insight into hallucination in intermediate artifacts and whether final patches reflect misunderstandings of buggy behavior.

    Source: §1 · p. 2, §2.1 · p. 5

  • Method

    The study analyzes hallucination in final patches and intermediate artifacts across three understanding tasks, evaluating three LLMs on 832 Defects4J bugs through automatic and manual analysis.

    Source: §1 · p. 3, §3.1 · p. 8

  • Results

    Repair and understanding hallucinations remain prevalent; manual analysis found repair hallucinations in 72.7% of 812 sampled repairs.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Task-specific intermediate-artifact scores can provide additional signals for identifying potentially unreliable patches, but cannot independently determine repair correctness.

    Source: §1 · p. 4

    The observed association between intermediate-artifact accuracy and repair success does not establish that improving an artifact causes better repair outcomes.

    Source: d. Conclusion validity · p. 43

Abstract

from arXiv · show

Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence. We examine repair hallucination in final patches and understanding hallucination in intermediate artifacts through three tasks, namely triggering testcase identification, line coverage prediction, and additional testcase generation.We then evaluate three representative LLMs on 832 Defects4J bugs through automatic evaluation and manual analysis. Our results show that both repair and understanding hallucinations remain prevalent. Across models and settings, only 21.0%-55.9% of generated patches pass the developer-written test suite. Moreover, although more accurate intermediate artifacts are generally associated with successful repairs, this relationship does not always hold. Manual analysis of 812 sampled repairs identifies repair hallucinations in 72.7% of cases, including patches that pass all available tests; incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively. Meanwhile, models frequently misidentify triggering testcases, mispredict line coverage involving branching control flow, and generate additional testcases with missing bug-triggering conditions or incorrect expected behavior.

1 Introduction · 2 Related Work

This study argues that evaluating only final APR patches misses hallucinations in the intermediate reasoning process, and it jointly analyzes repair and understanding hallucinations using execution-grounded evidence. Results across Defects4J show widespread, distinct failure patterns that patch outcomes alone cannot capture.

1 Introduction

  • The framework defines repair hallucination in final patches and understanding hallucination in intermediate artifacts, enabling joint analysis of hallucination manifestations and their relationship to repair quality.The empirical study evaluates three representative LLMs on Defects4J using automatic evaluation and manual analysis.Source: §1 · p. 3
  • The study evaluates understanding through triggering-testcase identification, line-coverage prediction, and additional-testcase generation, each paired with patch generation.These tasks expose program understanding through failure identification, execution reasoning, and behavioral generalization.Source: §1 · p. 3
  • 21.0%–55.9% of generated patches pass developer-written tests, while repair hallucinations occur in 72.7% of 812 sampled repairs, including some passing patches.Incorrect causal localization and repair strategies account for 45.9% and 18.5% of repair hallucinations, respectively.Source: §1 · p. 4
  • Models exactly identify triggering testcases for only 23.0%–40.9% of bugs, while 88.6% of inspected low-scoring coverage cases involve branch-related control flow.Additional testcase hallucinations include omitted bug-triggering conditions, and more accurate intermediate artifacts generally—but not deterministically—correlate with successful repairs.Source: §1 · p. 4

2.1 Automated Program Repair

  • Traditional and learning-based APR techniques can generate real-world repairs, but passing available tests does not establish semantic correctness because tests may incompletely specify intended behavior.LLM-based APR improves repair through mechanisms including code infilling, prompting, retrieval, natural-language interaction, and iterative test feedback, while remaining largely patch-centric in evaluation.Source: §2.1 · pp. 4–5

2.2 Hallucination in LLM-based Code Intelligence

  • Hallucination denotes plausible outputs that are not faithfully grounded, and existing code-intelligence studies primarily analyze final generated code rather than intermediate APR understanding and reasoning.This study instead examines whether coherent repairs are supported by execution-grounded evidence before patch synthesis.Source: §2.2 · pp. 5–6

2.3 LLM Agents for Automated Program Repair

  • Agentic APR systems increasingly structure repair into localization, reproduction, repair, and validation, while trajectory-level studies examine reasoning failures, tool interaction, and runtime evidence.Security studies also show that functionally passing patches may remain vulnerable, but this study addresses general non-adversarial hallucination instead.Source: §2.3 · pp. 6–7

3 Methodology · b. Automatic Evaluation

The study separates repair hallucination in final patches from understanding hallucination in intermediate artifacts, evaluating both through complementary APR tasks and execution-grounded criteria. Automatic evaluation measures artifact correctness, while human annotation examines semantic deviations that tests alone may miss.

3 Methodology

  • The methodology combines automatic evaluation of three APR tasks with human annotation to quantify hallucination prevalence, characterize its forms, and investigate contributing factors.The study workflow first evaluates generated patches and intermediate artifacts automatically, then samples outputs for manual analysis.Source: §3 · p. 7, §3.1 · pp. 8–9

3.1 Overview

  • Repair hallucination concerns invalid or developer-inconsistent final patches, whereas understanding hallucination concerns intermediate artifacts inconsistent with execution-grounded evidence.Repair hallucination includes uncompilable, test-failing, overfitting, or semantically unintended patches; understanding hallucination reflects misunderstanding of failure evidence, execution, or repair semantics.Source: §3.1 · p. 7
  • The three tasks progress from identifying triggering failure evidence, through predicting dynamic execution, to generating new behavioral evidence while pairing each artifact with a repair patch.They assess triggering testcase identification, line coverage prediction, and additional testcase generation.Source: §3.1 · pp. 7–8

3.2 Research Questions

  • The study asks how often hallucinations occur, which types and distributions arise, and what empirical factors may contribute across APR tasks and models.These correspond to research questions RQ1, RQ2, and RQ3.Source: §3.2 · p. 9

a. Task Design

  • For triggering testcase identification, the model receives buggy functions and relevant developer-written tests, predicts the triggering subset, and generates a repair patch.Relevant tests reach at least one class modified by the ground-truth patch, and triggering tests are those that fail on the original buggy version.Source: a. Task Design · pp. 9–10, b. Automatic Evaluation · p. 10

b. Automatic Evaluation

  • Automatic patch evaluation classifies outputs as Pass, Not Pass, or Uncompilable, treating the latter two as repair hallucinations while recognizing that passing tests does not guarantee intended semantics.Passing patches are therefore supplemented with human comparison against developer patches and execution evidence.Source: b. Automatic Evaluation · pp. 10–11

3.4 Line Coverage Prediction

  • Line coverage prediction is evaluated by comparing predicted and observed executed-line sets for both buggy and compilable model-patched programs using precision, recall, and F1.The evaluation filters non-informative lines and uses F1 as the primary metric because it balances omitted and incorrectly predicted execution lines.Source: §3.4 · pp. 11–12

3.5 Additional Testcase Generation

  • Additional testcases are valid only when they fail on the original buggy program and pass on the developer-written fixed program, with the fixed program serving as the external behavioral oracle.Executing each testcase on the model-patched program additionally checks consistency with the model’s repair, while human annotation later explains hallucination manifestations and causes.Source: §3.5 · pp. 12–14, §3.6 · p. 14

a. Data Sampling

The study samples outputs across three APR tasks and three LLMs, stratifying cases into nine task–model groups to assess repair and understanding hallucination jointly.

Data Sampling

  • Samples are stratified into nine task–model groups, with each group meeting the minimum sample size for 95% confidence and a 10% margin of error.The groups cover three APR tasks and three LLMs.Source: a. Data Sampling · p. 14
  • Each sampled case contains both its generated patch and corresponding intermediate artifact, enabling paired analysis of repair and understanding hallucination.Detailed population and sample sizes are reported in Section 4.3.Source: a. Data Sampling · pp. 14–15

b. Independent Open Coding

Independent open coding characterizes hallucination by comparing generated patches and intermediate artifacts against developer-written or execution-grounded evidence. Annotators record concrete failure manifestations, supporting execution evidence, and likely causes without relying on a predefined taxonomy.

b. Independent Open Coding

  • Two experienced Java programmers independently inspect sampled cases, assigning descriptive codes for failure manifestations, execution evidence, and likely causes.The coding process uses no predefined taxonomy.Source: b. Independent Open Coding · p. 15
  • Patch analysis compares compilation, intended repair semantics, modified locations, failing tests, and test overfitting across uncompilable, failing, and passing patches.Passing patches are checked against developer patches to distinguish intended repairs from patches that merely fit available tests.Source: b. Independent Open Coding · p. 15
  • Intermediate-artifact analysis compares model outputs with execution-grounded oracles, using set relations, mismatched coverage regions, and testcase compilation, paths, assertions, and outcomes.Triggering-testcase discrepancies are characterized automatically, whereas line-coverage and additional-testcase errors receive targeted manual inspection.Source: b. Independent Open Coding · p. 15

c. Codebook Construction … 4 Experimental Setup

The study builds a finalized hallucination taxonomy through coded annotation and conflict resolution, then evaluates GPT-5, DeepSeek-R1, and Claude Sonnet 4.5 on 832 filtered Defects4J bugs across three APR tasks.

c. Codebook Construction

  • After independent open coding, researchers consolidate descriptive codes into a preliminary codebook defining categories, labeling criteria, required evidence, and representative corner cases.The preliminary codebook supports label comparison and conflict resolution.Source: c. Codebook Construction · p. 15

d. Conflict Resolution

  • κ = 0.82 for repair hallucination annotations and κ = 0.49 for understanding hallucination annotations in initial inter-annotator agreement.Agreement was measured separately using Cohen’s kappa, which accounts for chance agreement.Source: d. Conflict Resolution · p. 15
  • Conflicting labels are resolved in face-to-face reviews by two annotators and two experienced authors, with final labels assigned only after agreement.The additional authors have over nine years and over fifteen years of software engineering research experience, respectively.Source: d. Conflict Resolution · p. 16

e. Taxonomy Construction

  • The resolved labels produce a finalized hallucination taxonomy applied to all sampled cases to characterize category distributions across tasks and models.These taxonomy results support analysis of hallucination forms and potential contributing factors.Source: e. Taxonomy Construction · p. 16

4 Experimental Setup

  • The experimental setup also specifies generated-output and sampled-case annotation counts, model API settings, execution tools, timeouts, and the Java environment.These configuration details complete the study’s reproducible evaluation setup.Source: §4 · p. 16

4.1 Studied Models

  • The study evaluates GPT-5, DeepSeek-R1, and Claude Sonnet 4.5 because they provide strong code generation, reasoning, and instruction-following capabilities.These models were competitive and widely accessible through provider APIs when experiments began in November 2025.Source: §4.1 · p. 16

4.2 Dataset

  • The filtered Defects4J dataset contains 832 bugs with compilable buggy and fixed versions, at least one triggering testcase, and function-localized developer modifications.Extracted buggy functions, relevant test methods, and triggering testcases instantiate inputs for the three APR tasks.Source: §4.2 · p. 17

4.3 Implementation

  • The implementation pipeline constructs task inputs, queries models, parses outputs, and organizes artifacts for automatic evaluation and human annotation across three task-specific APR configurations.Triggering-testcase identification uses slices of at most 300 relevant methods; line-coverage prediction uses all triggering testcases; additional-testcase generation evaluates each generated testcase independently.Source: §4.3 · p. 18

d. Baseline Repair · e. Model Querying and Execution Environment

The study includes a task-independent baseline repair setting and evaluates all models under standardized zero-shot querying and isolated, time-bounded execution environments. These procedures use developer-written tests, coverage collection, and project-specific build configurations for consistent assessment.

d. Baseline Repair

  • The baseline repair setting supplies only complete developer-modified functions and asks the model to generate a minimal correct fix.Source: d. Baseline Repair · p. 19
  • Each baseline patch is applied to the buggy program and evaluated with the developer-written test suite and the same patch-evaluation procedure as task-specific settings.Source: d. Baseline Repair · p. 19

e. Model Querying and Execution Environment

f. Population and Sample Size · 5 Results · a. Repair Hallucination

This section examines hallucinations across final repairs and intermediate artifacts in APR, including their prevalence, manifestations, and contributing factors across artifact-generation tasks. It also considers how repair plausibility and intermediate understanding relate to repair outcomes and distinguishes execution-visible from semantically detectable hallucinations.

5.1 Prevalence of Repair and Understanding Hallucinations

  • The study defines hallucinations at both final-patch and intermediate-artifact levels, covering triggering testcase identification, line coverage prediction, and additional testcase generation.Population sizes use sliced instances, bug-level instances, and generated additional-testcase instances, respectively, with random sampling within task-model groups for annotation.Source: §5.1 · p. 19, f. Population and Sample Size · p. 19

a. Repair Hallucination

  • 21.0%–55.9% of repair attempts produce plausible patches across models and settings, with GPT-5 highest overall and line coverage prediction highest among tasks.Baseline plausible-patch rates are 21.0%–26.3%, while line coverage reaches 55.9% for GPT-5, 48.4% for DeepSeek, and 37.0% for Claude.Source: a. Repair Hallucination · pp. 20–21
  • Only 96 Claude, 107 DeepSeek, and 169 GPT-5 bugs receive plausible patches under all four settings, indicating limited robustness across task formulations and evidence.Many other bugs are plausibly patched only under specific settings or combinations of settings.Source: a. Repair Hallucination · p. 21
  • Passing repairs achieve higher triggering-testcase identification and line-coverage scores, including GPT-5 F1 scores of 0.607 versus 0.214 and buggy-program coverage F1 of 0.872.Claude and DeepSeek also show higher identification scores for passing repairs, while passing groups outperform unsuccessful groups in coverage prediction.Source: a. Repair Hallucination · p. 22, c. Line Coverage Prediction · p. 23
  • Generated triggering testcases are valid for only 30.5% of Claude, 34.6% of DeepSeek, and 48.4% of GPT-5 outputs.More than half of generated testcases are invalid for every model, and validity does not guarantee a plausible repair.Source: c. Line Coverage Prediction · p. 24

5.1 Prevalence of Repair and Understanding Hallucinations

  • Successful repairs generally have more accurate triggering-testcase identification, line-coverage predictions, and generated testcases, but these associations are not deterministic.Passing repairs can still contain inaccurate artifacts, while some failed repairs contain accurate ones; therefore, intermediate understanding alone does not determine repair success.Source: a. Repair Hallucination · p. 22, c. Line Coverage Prediction · pp. 23–24

5.2 Hallucination Types and Distributions

  • Repair hallucinations are categorized into 10 recurring patch types plus Others, ranging from compilation failures to test-failing, overfitting, and semantically incorrect passing patches.The taxonomy includes non-existent symbols, missing dependencies, structural errors, incorrect localization or strategy, API misuse, partial repairs, extraneous logic, boundary errors, and test-specific heuristics.Source: §5.2 · pp. 25, 26, 28
  • Passing all developer-written tests does not ensure semantic correctness, so overfitting patches require manual comparison with the developer-intended repair.Test-specific heuristic overfitting can avoid observed failures while deviating from intended repair semantics.Source: §5.2 · pp. 26, 29

a. Repair Hallucination

  • 72.7% of manually inspected repairs exhibit hallucinations, dominated by incorrect causal localization and inappropriate repair strategies, while compilation errors and overfitting remain non-negligible.Overfitting accounts for 16.38% of repair hallucinations and can pass developer-written tests despite violating intended repair semantics.Source: §5.2 · pp. 29–30

b. Triggering Testcase Identification

  • More than half of bugs are misidentified even by GPT-5, with mixed omission-and-spurious-addition errors the dominant triggering-testcase failure pattern.Models often capture some bug-related signal but struggle to distinguish true triggering tests from unrelated tests.Source: Table 6 · p. 30, §5.2 · p. 31
  • 71, 91, and 113 bugs show trigger-insensitive success for Claude, DeepSeek, and GPT-5, respectively, while trigger availability benefits DeepSeek and GPT-5 more than Claude.Trigger-dependent success exceeds trigger-absent-only success for DeepSeek and GPT-5, whereas Claude shows the reverse pattern.Source: Table 7 · p. 31, §5.2 · p. 32

c. Line Coverage Prediction

  • Branch-related hallucinations dominate low-scoring line-coverage predictions, appearing in 25 of 28 Claude, 11 of 11 DeepSeek, and 3 of 5 GPT-5 predictions.The main difficulty is identifying the correct conditional execution path under triggering testcases; exception, return, and assignment errors are less frequent.Source: Table 8 · p. 32, §5.2 · p. 33

d. Additional Testcase Generation

  • 46 of 185 additional-testcase hallucinations (24.9%) involve incomplete bug-triggering conditions, making this the most prevalent category.Models often omit boundary values, configurations, or input combinations needed to expose the faulty behavior.Source: Table 9 · p. 34, d. Additional Testcase Generation · p. 37
  • 64 of 185 additional-testcase hallucinations (34.6%) are compilation-related, while incorrect output or exception expectations reveal failures to encode developer-intended behavior.Errors include incompatible project symbols, dependencies, test frameworks, language features, malformed source encoding, and invalid test structure.Source: d. Additional Testcase Generation · pp. 35, 37
  • 56 generated testcases pass on both the buggy and developer-fixed programs, so they do not expose the target bug despite potentially serving as regression tests.This demonstrates that executable tests can lack discriminative power even when they satisfy basic runtime checks.Source: d. Additional Testcase Generation · p. 37
  • Shared non-existent-symbol and missing-dependency errors indicate insufficient project-context grounding across patches and generated testcases, while testcase generation adds bug-trigger and oracle-specific failure modes.These additional modes include incomplete triggering conditions and incorrect output or exception expectations.Source: d. Additional Testcase Generation · p. 37

5.2 Hallucination Types and Distributions

  • Exact triggering-testcase identification ranges from 23.0% to 40.9%, mixed misidentification affects 33.3%–47.6% of bugs, branch-related structures appear in 88.6% of inspected low-scoring coverage predictions, and incomplete bug-triggering conditions comprise 24.9% of observed additional-testcase hallucinations.These findings cover triggering testcase identification, line coverage prediction, and additional testcase generation, respectively.Source: d. Additional Testcase Generation · p. 38

5.3 Contributing Factors to Hallucinations

  • The study investigates factors that may explain hallucination emergence across final patches and intermediate APR artifacts, with Table 10 summarizing three possible causes.The examined artifacts include triggering testcase identification, line coverage prediction, and additional testcase generation.Source: §5.3 · p. 38

a. Insufficient Project-Context Grounding

Insufficient project-context grounding contributes to hallucinations when models lack repair-critical project entities, producing plausible but incompatible patches and testcases. The findings motivate selective context retrieval that preserves relevant project information without unnecessary token overhead.

Insufficient Project-Context Grounding

  • Missing project-specific symbols, dependencies, declarations, or testing conventions frequently produce uncompilable patches and testcases, while incorrect API usage reflects misinterpreted project semantics.These failures can arise even when generated code is plausible in a generic programming context.Source: a. Insufficient Project-Context… · pp. 38–39
  • Repair-critical entities absent from the provided context can cause models to generate semantically plausible but project-incompatible references.In Closure-1, the intended fix introduced removeGlobals, whereas the generated patch referenced undefined removeGlobal, causing compilation failure.Source: a. Insufficient Project-Context… · p. 39
  • Existing APR agents mitigate context-grounding failures through repository exploration, structure-aware retrieval, fault localization, and iterative compilation- or test-based validation, while newer methods add repository history or hierarchical documentation.The passage identifies context-selection strategies as future work because indiscriminate expansion can increase token consumption and irrelevant information.Source: a. Insufficient Project-Context… · p. 39

b. Imprecise Causal Localization

Imprecise causal localization drives hallucinations across triggering testcase identification, patch generation, and additional testcase generation by confusing failure-related behavior with the actual bug cause.

Imprecise Causal Localization

  • Incorrect Causal Localization is the most prevalent repair hallucination, arising when models modify code related to an observed failure without addressing its root cause.Related failures include Spurious Identification and Mixed Misidentification in triggering testcase identification, and Faulty Code Location Not Reached in additional testcase generation.Source: b. Imprecise Causal Localization · p. 39

c. Unstable Execution-Path Reasoning

Unstable execution-path reasoning is a major source of hallucination in LLM-based APR, especially when models must determine which branching path a triggering testcase exercises. Branch-related structures dominate low-scoring predictions, revealing difficulty reasoning about concrete execution behavior.

Unstable Execution-Path Reasoning

  • 39 of 44 manually inspected low-scoring predictions (88.6%) involved branch-related structures.This substantially exceeded errors involving exception-flow, return-statement, and assignment-statement structures.Source: c. Unstable Execution-Path Reasoning · p. 40
  • Models often predict an incorrect branch or omit the branch exercised by the triggering testcase.They also make errors determining whether exception-handling paths execute.Source: c. Unstable Execution-Path Reasoning · p. 40
  • Execution-path hallucinations reflect broader failures in grounding artifacts, localizing causes, and reasoning about concrete program paths.These limitations are identified as contributing factors to hallucinations in LLM-based APR.Source: c. Unstable Execution-Path Reasoning · p. 40

6 Discussion

The discussion argues that LLM-based APR should validate intermediate artifacts as potentially hallucinated behavioral claims and strengthen evidence grounding, while acknowledging threats from nondeterministic outputs, human annotation, and implementation errors.

6.1 Implications

  • APR evaluation should assess intermediate artifacts alongside final test-suite outcomes, using their correctness to estimate patch confidence and diagnose failures by repair stage.Relevant artifacts include triggering testcases, predicted line coverage, and generated additional testcases; they may reveal testcase-understanding, execution-path, or repair-synthesis failures.Source: §6.1 · p. 40
  • APR systems should combine repository-aware retrieval, program analysis, and repair evidence to improve project grounding, causal localization, and execution-path reasoning.Useful evidence includes symbols, dependencies, API usages, testing conventions, requirements, specifications, issue descriptions, change history, coverage, slicing, and execution traces.Source: §6.1 · pp. 40–41
  • Because branch-dependent behavior frequently causes execution hallucinations, systems should reason explicitly about predicates, inputs, program states, and configuration settings that determine executed paths.Dynamic coverage, program slicing, and execution traces can help distinguish causal locations and testcases from merely related evidence.Source: §6.1 · p. 41
  • Intermediate artifacts require independent verification because generated triggering testcases, execution paths, and additional testcases may hallucinate even when patches pass developer-written tests.They should be treated as verifiable behavioral claims rather than inherently reliable explanations.Source: §6.1 · p. 41
  • Hallucination categories support failure-specific validation pipelines, pairing compilation and symbol checks with dynamic analysis, additional testing, and semantic comparison as appropriate.The taxonomy may also guide targeted training data aimed at reducing specific hallucination forms rather than merely increasing general repair data.Source: §6.1 · p. 41
  • Generated testcases should be classified as bug-revealing, regression-preserving, patch-specific, or invalid because each provides different evidence about repair behavior.Tests passing on both buggy and fixed versions may preserve regression behavior, whereas tests passing only on a model patch may expose overfitting.Source: §6.1 · p. 41

a. Internal validity.

  • Internal validity is threatened by nondeterministic LLM outputs, evolving human judgments, and possible implementation or parsing errors.Mitigations include fixed prompts and settings, independent annotation with agreement measurement and discussion, retained logs, consistent pipelines, isolated execution, and clean checkouts.Source: a. Internal validity · p. 42

b. Construct validity.

The study’s validity measures rely on execution-based criteria and manual inspection, but passing tests or revealing bugs do not fully establish semantic correctness or testcase usefulness. Manual findings are therefore treated as potential contributors rather than proven causal mechanisms.

Patch validity

  • Developer-written tests may misclassify semantically incorrect or overfitted patches as passing, so sampled Pass patches are manually compared with developer repairs.Pass, Not Pass, and Uncompilable labels do not constitute a complete specification of intended repair behavior.Source: b. Construct validity · p. 42

Testcase validity

  • A generated testcase is valid only if it fails on the buggy version and passes on the fixed version, measuring bug revelation rather than overall testcase usefulness.Testcases passing on both versions may still provide regression-preserving value, motivating manual analysis of failure patterns.Source: b. Construct validity · p. 42

Interpretation of findings

  • Manual-analysis factors are synthesized across hallucination categories and APR tasks and presented as empirically grounded potential contributors, not established causal mechanisms.This framing reduces overinterpretation from alternative explanations.Source: b. Construct validity · p. 42

c. External validity.

The evaluation’s external validity is limited by its reliance on one Java benchmark and three LLMs, although the framework is designed for broader benchmarks, languages, and models.

External validity

  • Results may differ across datasets, programming languages, project ecosystems, bug types, models, and future model versions because evaluation used only Defects4J and GPT-5, DeepSeek-R1, and Claude Sonnet 4.5.The models were selected from different providers and evaluated with identical task definitions and procedures to reduce model-specific dependence.Source: c. External validity · p. 43
  • The framework can be instantiated for other languages and benchmarks using suitable test infrastructure and execution instrumentation, because it targets general APR artifacts and execution outcomes.Its model-agnostic tasks can also be applied to newly released models by collecting intermediate artifacts and evaluating them against execution-grounded oracles.Source: c. External validity · p. 43

d. Conclusion validity. … Informed consent

The study finds hallucination prevalent in both final patches and intermediate APR artifacts, while cautioning that artifact quality is correlationally associated with repair success rather than proven causal. Its conclusions motivate stronger project grounding and execution evidence, with funding acknowledged from Singapore’s Ministry of Education.

d. Conclusion validity.

  • Reported hallucination distributions come from samples rather than exhaustive annotation, mitigated through stratified sampling across every task–model combination and a common annotation procedure.This limitation concerns the validity of generalizing the observed distributions beyond the sampled artifacts.Source: d. Conclusion validity · p. 43
  • The observed association between intermediate-artifact quality and repair outcomes is correlational, so the study does not establish that improving an artifact will improve repair success.Conclusions are limited to reporting that more accurate intermediate artifacts are generally associated with successful repairs, rather than claiming direct causation.Source: d. Conclusion validity · p. 43

7 Conclusion

  • The study evaluates repair hallucination in final patches and understanding hallucination in intermediate artifacts across triggering-testcase identification, line coverage prediction, and additional testcase generation on 832 Defects4J bugs.These findings motivate evaluating both final patches and intermediate artifacts while strengthening APR systems with better project grounding and execution evidence.Source: §7 · pp. 43–44
  • 21.0%–55.9% of generated patches pass the developer-written test suite, while exact triggering-testcase identification reaches 23.0%–40.9% and valid additional testcases reach 30.5%–48.4%.GPT-5 generally exhibits the lowest tendency toward hallucination, although accurate intermediate artifacts do not always accompany successful repairs.Source: §7 · p. 43
  • Manual analysis identifies repair hallucinations in 590 of 812 outputs (72.7%), with incorrect causal localization and incorrect repair strategy accounting for 45.9% and 18.5%, respectively.Insufficient project-context grounding, imprecise causal localization, and unstable execution-path reasoning are identified as potential contributing factors.Source: §7 · pp. 43–44

Funding

  • The research is supported by Singapore’s Ministry of Education under Academic Research Fund Tier 3 Award ID MOET32020-0004, and the authors state that the findings do not represent the Ministry’s views.Source: Funding · p. 44

Related papers

Loading 2609.04909v1…