PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications

Boyang Yang, Zijian Cai, Fengling Liu, Bach Le, Lingming Zhang, Tegawendé F. Bissyandé, Yang Liu, Haoye Tian

arXiv:2506.23749v3cs.SE

TL;DR

Prior reviews lack a unified taxonomy of LLM repair paradigms and protocol-level analysis of reported results. This survey classifies 66 systems by repair capability and control, and its synthesis shows that design choices and evaluation protocols jointly shape reported repair success.

  • Problem

    Prior reviews lack a unified taxonomy of repair paradigms and protocol-level analysis of reported results, limiting evidence for cross-paradigm and result comparability.

    Source: §1 · p. 3

  • Method

    The survey bottom-up codes 66 repair systems into four primary paradigms based on repair capability and control, while recording auxiliary evidence as separate fields.

    Source: §3.4 · p. 10

  • Results

    The synthesis maps 66 representative systems and shows that design choices and evaluation protocols jointly shape reported repair success.

    Source: §12 · pp. 44–45

  • Takeaways & Limitations

    The synthesis identifies stronger benchmarks, more comparable reporting, repository-aware retrieval and analysis, and cost-aware planning as priorities for dependable repair infrastructure.

    Source: §12 · p. 45

    Because it preserves published scores without normalizing protocol differences or jointly reevaluating systems, the survey cannot establish broad cross-paper comparability.

    Source: §11.2 · p. 44

Abstract

from arXiv · show

Large language models (LLMs) are reshaping automated program repair. We present a reproducible hierarchical taxonomy that organizes 66 LLM-based repair systems according to where repair capability and control logic principally reside: task-adapted parameters, prompt and context design, designer-specified workflows, or LLM-directed runtime control. Adaptation, generation pattern, runtime control, and auxiliary evidence are preserved as separate coded dimensions. This representation exposes control distinctions hidden by utilization labels and supports cross-paradigm analysis of system design and evaluation evidence. To the best of our knowledge, it is the first publicly available LLM-based software repair survey to operationalize Fine-Tuning, Prompting, Procedural, and Agentic as one corpus-wide primary classification. Our hierarchy complements prior surveys through one corpus-wide primary decision rule, while the result-level protocol audit bounds which reported results are defensibly comparable. To analyze how repair systems use benchmarks, we record benchmark variants, metrics, base models, and fault-localization assumptions for each system's primary reported result, identifying protocol-aligned comparison windows and benchmark-family fragments. We clarify paradigm trade-offs in task alignment, deployment cost, controllability, and multi-hunk or cross-file repair. We outline open challenges and research directions. Our artifacts and scripted survey pipeline are publicly available at https://github.com/GLEAM-Lab/ProgramRepair.

1 Introduction

The survey proposes a corpus-wide taxonomy of LLM-based repair systems and a protocol-aware analysis of their reported evaluations, alongside scenario-specific design guidance and reproducible artifacts.

1 Introduction

  • Prior surveys organize repair by utilization, scenario, security scope, tuning method, evaluation role, or broad AI-assisted repair, while the paper identifies gaps in unified paradigms, auxiliary-evidence layers, and protocol-level comparison.The introduction also notes that repair research has expanded from zero-shot, single-function prompting to parameter adaptation, domain-knowledge-guided vulnerability repair, and repository-level tool-augmented workflows.Source: §1 · pp. 2–3
  • The survey classifies 66 systems published or released from January 2022 to April 2026 by primary repair locus into Fine-Tuning, Prompting, Procedural, and Agentic.Adaptation, generation pattern, runtime control, and auxiliary evidence remain separate coded dimensions; auxiliary evidence can support any paradigm without determining it.Source: Figure 1 · p. 2, §1 · p. 3
  • For each system, the survey records benchmark variant, metric, base model, and fault-localization assumptions, separating protocol-aligned comparison windows from non-comparable benchmark-family fragments.This result-level audit addresses why shared benchmark names may conceal differences in variants, pass@k budgets, fault-localization assumptions, or oracle strength.Source: §1 · p. 3
  • The survey analyzes paradigm behavior across function-level benchmarks, repository repair, vulnerability patching, educational tutoring, and industrial continuous integration, deriving guidelines tied to defect scope, data availability, latency, and compute budgets.Source: §1 · p. 4
  • It releases a living, versioned bibliography, extracted metadata, and an end-to-end scripted pipeline for reproducing literature retrieval, screening, and coding.The artifacts are publicly available at the paper's GitHub repository.Source: §1 · p. 4

2 Related Work

Earlier surveys leave LLM-centered repair control and protocol comparability underexamined; this survey classifies 66 systems across separate design dimensions and analyzes results through protocol-aligned comparison windows.

2 Related Work

  • Earlier surveys often predated LLM repair or subsumed it: Huang et al. [49] relegated LLMs to future work, Zhang et al. predated large-scale LLMs, and Dikici et al. folded GPT-style systems into deep learning.Winter et al. focused on developer perceptions and usability rather than a technical taxonomy.Source: §2 · p. 4
  • Recent reviews cover narrower slices: Anand et al. [5] survey few repair papers without a dedicated taxonomy, Haque et al. [40] focus on fine-tuning, and He et al. [41] examine LLM-as-judge evaluation.Zhou et al. limit their review to security-related bugs and treat retrieval-augmented and agentic methods mainly as future directions.Source: §2 · p. 5
  • Zhang et al. classify repair studies by LLM utilization modes and scenarios, while Abdollahi et al. survey code-intelligence benchmarks and Zhang et al. [146] examine prompt design and contamination risks.The present survey uses benchmark context to assess when repair-system results are comparable under compatible protocols.Source: §2 · p. 5
  • Existing comparison surveys do not independently align adaptation, runtime control, auxiliary evidence, and system-level evaluation protocols, obscuring orchestration and comparability across LLM-based repair systems.This survey codes 66 systems by taxonomy, defect scope, and deployment scenario, and analyzes results through protocol-aligned comparison windows rather than benchmark names alone.Source: §2 · p. 5
  • The survey’s four-way classification preceded Zhang et al.’s later Agent category and Procedural/Agentic split, with the two surveys converging independently on control distinctions while remaining complementary.Zhang et al. v2 (12 May 2024) used fine-tuning, few-shot, and zero-shot strategies; this survey’s first arXiv version (30 June 2025) operationalized Fine-Tuning, Prompting, Procedural, and Agentic; Zhang et al.’s v3 followed on 21 October 2025. Zhang et al. retain utilization modes and scenario-specific organization, while this survey assigns one corpus-wide primary locus and preserves orthogonal coded dimensions.Source: §2 · p. 5

3 Survey Methodology

The methodology builds a representative, criteria-bounded corpus and a reliability-audited framework for analyzing repair-system designs and evaluation evidence. It combines systematic searching, screening, coding, and protocol-focused questions to make comparison more reproducible.

3 Survey Methodology

  • Using the Goal Question Metric approach and systematic-review and snowballing guidance, the survey maps a representative corpus to expose design, evaluation, and open-problem patterns.LLM-based software repair is defined as systems whose inference-time pipeline centrally invokes a pretrained neural code model or LLM.Source: §3 · pp. 5–6

3.1 Research Questions

  • The survey asks how systems are coded and combined across scopes, which results are protocol-comparable, and what recurring failures support an evidence-based research agenda.Its four questions cover taxonomy reliability, design trade-offs, benchmark-specific comparability, and research directions for semantic correctness, repository-scale repair, and multi-hunk repair.Source: §3.1 · p. 7

3.2 Search Strategy

  • Searches combined repair, model, benchmark, and tool terms across digital libraries, then applied high-recall repair-, LLM-, and benchmark-related filters before snowballing and manual screening.The search was restricted to 2022 onward, with the last full query run on 30 April 2026; a fixed-seed audit of 100 benchmark-filter rejections found 0 potential false negatives.Source: §3.2 · pp. 7–9

3.3 Eligibility Criteria

  • Eligible studies targeted a concrete repair task, centrally used an inference-time LLM, reported quantitative results on a reproducible benchmark, and met publication, access, and reporting requirements.ArXiv-only systems were retained only with concrete repair pipelines and reproducible benchmark evidence when no in-scope archival version was found by the search date.Source: §3.3 · pp. 9–10

3.4 Taxonomy Construction and Reliability Audit

  • The taxonomy was developed through bottom-up coding, then assigns one primary paradigm from adaptation and runtime control while preserving auxiliary evidence, repair scope, and scenario tags.The coding dimensions also record generation pattern and external evidence, allowing hybrid behavior without making evidence-augmented repair a mutually exclusive top-level category.Source: §3.4 · pp. 10–11
  • 65/66 primary-paradigm labels agreed (Cohen’s κ=0.98), 64/66 control-subtype labels agreed (κ=0.97), and retrieval, analysis, and scenario tags had 66/66 raw agreement.Two independent coders recoded the final 66-system corpus; boundary cases were adjudicated using benchmark scope, workflow evidence, and full-text descriptions.Source: §3.4 · p. 11

3.5 Screening Flow

  • The screening process reduced 2,317 initial hits to 66 representative works; two coders agreed on 467 of 474 candidates before adjudicating seven disagreements.The 66 studies form a criteria-bounded corpus, not a complete census; the candidate pool included four archival-status records synchronized during publication-status updates.Source: §3.5 · pp. 11–12

4 Selected Works Overview and Taxonomy Distribution

The survey classifies 66 repair systems by where repair capability and control logic principally reside, while separately recording adaptation, generation, auxiliary evidence, scope, and scenario. Its corpus-level findings describe system scope, publication venues, and year-by-paradigm distribution, with high agreement in independent recoding.

4.1 Taxonomy at a Glance

  • The primary paradigm identifies where repair capability and control logic principally reside, while adaptation, prompting, orchestration, and auxiliary evidence remain separately coded.The primary label supports corpus-level aggregation but does not fully describe hybrid system behavior.Source: §4.2 · p. 13, §4.1 · p. 13

4.2 Classification rules

  • Fine-Tuning requires task-specific adaptation to carry the principal repair capability; otherwise, Prompting, Procedural, or Agentic classification depends on whether prompts, designer-fixed workflows, or LLM-directed control carry the main repair logic.Zero- or few-shot systems such as Agentless, ChatRepair, and Repilot are Procedural when designer-specified orchestration carries their principal repair logic.Source: §4.2 · pp. 13–15
  • The taxonomy refines its four paradigms with control subtypes and non-exclusive auxiliary-evidence tags, preserving hybrid properties without turning them into additional top-level categories.Subtypes include tuning methods, prompting settings, procedural workflows, and agentic systems; auxiliary tags include retrieval, analysis, tests, human feedback, and domain knowledge.Source: §4.2 · p. 15
  • Of the 66 systems, 55 target general program repair only, 10 focus exclusively on vulnerability repair, and one addresses both.Source: Figure 4 · p. 15, §4.2 · p. 16

4.3 Venues of selected works

  • The corpus contains nine arXiv-only preprints and 57 archival publications, with archival systems appearing across software engineering, AI, and security venues.The arXiv bar denotes publication status, not a venue.Source: Figure 5 · p. 16, §4.3 · p. 16

4.4 Temporal Distribution within the Criteria-Bounded Corpus

  • In the criteria-bounded corpus, 2023 includes many fine-tuning and prompting studies, 2024 adds procedural and agentic systems, and 2025 has the most systems across paradigms.The 2026 row covers January through April; these counts describe this corpus, not a complete field-wide trend. Independent coders agreed on 65/66 primary-paradigm labels (κ=0.98), 64/66 control-subtype labels (κ=0.97), and all retrieval, analysis, and scenario tags.Source: §4.4 · pp. 16–17, Table 8 · p. 17

5 Fine-Tuning Approaches

Fine-tuning adapts repair models by updating weights with bug-fix data, through full or parameter-efficient supervision, distillation, reward-based tuning, and auxiliary evidence. It can improve repair accuracy but trades generality and training cost for specialization; PEFT, distillation, and auxiliary evidence can reduce costs, while RL and preference tuning depend on reward design and test availability.

5.1 Full Fine-Tuning

  • VulMaster raises exact-match from 10.2% to 20.0%, BLEU from 21.3% to 29.3%, and CodeBLEU from 32.5% to 40.9%.It fine-tunes CodeT5 on a security-focused corpus and uses ChatGPT for missing context; the study reports high GPU-memory demand and weaker gains on ultra-long files. Other full-tuning studies report improvement across benchmarks, alongside GPU demand and overfitting risks [56].Source: §5.1 · p. 18

5.2 Parameter-Efficient Fine-Tuning

  • RepairLLaMA fixes 144 Defects4J v2 bugs, 109 HumanEval-Java tasks, and 20 GitBug-Java issues using rank-8 LoRA on a single GPU.Its adapters use 0.06% of parameters, but limited capacity and static fault localization reduce accuracy on multi-hunk or cross-file defects. MORepair reports the most fixes versus standard fine-tuning on EvalRepair-Java and C++; federated LoRA matches centralized training within two percentage points while exchanging megabytes of weights per round [80].Source: §5.2 · pp. 18–19

5.3 Knowledge Distillation

  • DistiLRR nearly doubles pass@1 on low-resource Perl and Golang and matches its teacher on high-resource languages.KNOD reports 72 Defects4J v1.2 and 50 v2.0 repairs through grammar-constrained decoding, while NARRepair reports 5.4–18.6 times faster inference than autoregressive baselines; its evaluations are Java-only [57] [138].Source: §5.3 · pp. 19–20

5.4 Reinforcement Learning Fine-Tuning

  • SWE-RL achieves a 41.0% solve rate on SWE-bench Verified, while AdaPatcher reports 62% consistency and 67.57% accuracy on ACPR [118] [16].SWE-RL uses patch diff similarity instead of execution; AdaPatcher combines supervised fault localization with preference optimization. RL tuning can incur test-execution costs and sparse rewards, and weak oracles or rule-based rewards may misvalue patches [118, 16, 8d7975, s18.p8#bae120].Source: §5.4 · pp. 20–21

5.5 Auxiliary-Evidence Fine-Tuned Repair

  • TraceFixer repairs 13–20% more bugs than a code-only baseline by conditioning fine-tuning on partial execution traces [9].PyTy’s top-50 candidates remove target errors in 85.4% of 281 held-out cases and exactly match developer fixes in 54.4%, versus 26.4% for GPT-4 [15]. Auxiliary context can improve repair but adds collection and encoding costs and can strain context windows [s19.p1#693bc0, s19.p7#850bcd].Source: §5.5 · pp. 21–22

6 Prompting Approaches

Prompt-based repair keeps the model frozen and generates a patch in one turn, while external prompt design supplies examples or auxiliary evidence. Across these approaches, focused context can help, but irrelevant or verbose context may hurt and context length and cost remain constraints.

6.1 Zero-shot Prompting

  • Zero-shot repair needs no additional training and can be competitive under reported protocols, but careful wording matters and results remain protocol-specific.AlphaRepair reports up to 3.3 times more repaired bugs than its study baselines on Defects4J-v1.2/2.0; Codex repaired 23 Python and 14 Java QuixBugs bugs with its highest-performing prompt variant.Source: §6.1 · p. 23
  • Zero-shot prompting can repair bugs without training or exemplars, but remains prompt-sensitive and its reliability on larger multi-hunk patches is unclear.ChatGPT fixed 84% of student-assignment tasks at TOP-5 and averaged 60% correct among five completions, rivaling Refactory; the verbose assignment description reduced accuracy.Source: §6.1 · p. 23

6.2 Few-shot Prompting

  • Few-shot repair uses curated or retrieved bug-fix examples to guide frozen models, but exemplar relevance, context-window limits, and sampling costs constrain the approach.Four similarity-retrieved exemplars more than doubled exact-match accuracy versus random examples and tripled it versus zero-shot [32]; explanation-bearing demonstrations with majority voting raised accuracy by roughly ten percentage points [3].Source: §6.2 · p. 24

6.3 Retrieval-Augmented Generation Enhanced Prompting

  • Retrieval-augmented prompting grounds one-shot patches in repository evidence and can support cross-file repair without extra inference rounds or model updates.RLCE’s project map retrieves relevant out-of-file definitions and usages [13], raising GPT-3.5’s top-1 fix rate from 22.6% to 56.0% and GPT-4’s from 41.1% to 81.5% on RepoBugs.Source: §6.3 · p. 25
  • Retrieval can inject layered project knowledge or API facts, while vulnerability repair can convert retrieved fixes into patterns that constrain the expected edit.Layered knowledge injection improves BugsInPy fix rates incrementally across Bug, Repository, and Project Knowledge [25]; PailGen reaches 23.23% exact match on Big-Vul_CVEfixes versus 5.07% for its baseline prompt.Source: §6.3 · pp. 24–25

6.4 Program-Analysis Evidence in Prompting

  • Compact runtime traces improve GPT-4’s correct-program accuracy by up to 6% over error-only prompts, but the benefit vanishes for GPT-3.5 and traces exceeding 200 lines [39].TracePrompt adds a structured execution trace to buggy code and a failing test; other analysis-enriched systems use contract-specific evidence or API-misuse labels to guide repairs.Source: §6.4 · p. 26
  • Analysis-enriched prompts use failing tests, semantic slices, traces, or domain-specific diagnostics to focus patches, while balancing useful evidence against prompt length and bias.APPATCH’s F1 reaches 36.46% on zero-day vulnerabilities versus 28.41% for the best prompting baseline, and 73.86% on ExtractFix versus 68.54% [84]; it requires a known vulnerability location.Source: §6.4 · pp. 25–26

6 Prompting Approaches

  • Prompt-based repair benefits most from focused auxiliary context, while irrelevant or verbose additions can hurt; adaptive context selection under token and cost limits remains an open problem.Auxiliary context includes similar fixes, repository or API snippets, failing tests, and compact execution traces.Source: §6 · p. 22, §6.4 · p. 27

7 Procedural Approaches

Procedural repair keeps the LLM frozen within designer-scripted workflows, varying the feedback source—tests, people, retrieval, or program analysis—while fixing control flow to support predictable costs and reproducibility. Deeper feedback can improve repair, but adds resource costs that workflows must manage.

7.1 Test-Feedback Loop Pipelines

  • REx reduces LLM calls roughly 2- to 5-fold and solves 73.7% of Nonlinear Loop Invariant tasks within 300 GPT-4 queries, versus a 60.5% specialized-solver baseline.Test-feedback loops use scripted iteration: ChatRepair feeds failing assertions back to the model, while ContrastRepair pairs failing and passing tests and ThinkRepair reuses reasoning traces. Richer feedback and adaptive candidate selection can boost success, but often require more tokens, calls, or engineering effort.Source: §7.1 · pp. 27–28

7.2 Human-in-the-Loop Pipelines

  • Human feedback turns frozen LLMs into iterative repair partners: CREF scores 76.6% AVG-5 on TutorCode, while HULA reports 82% plan approval and 56 merged pull requests.HULA also reports 86% file-recall and 45% LLM-based code-similarity on SWE-bench; DRCodePilot produces 109 and 18 exact-match patches on Flink and Solr, respectively, with CodeBLEU gains of 5.4% and 3.9%. Human guidance can improve workflows, though correctness beyond unit tests and context preparation remain challenges.Source: §7.2 · pp. 28–29

7.3 Retrieval-Enriched Scripted Pipelines

  • Scripted retrieval supplies targeted repository context: Agentless repairs 32.0% of SWE-bench Lite issues for about $0.70 per bug, while KGCompass reports 58.30% pass@1 for roughly $0.20 per bug.PATCH combines dependence context, commit messages, and retrieved fixes with tester, developer, and reviewer roles; on BFP it reports pass@1 33.97%, pass@3 37.08%, and pass@5 39.81%, versus GPT-4's 19.96%, 24.42%, and 26.00%, respectively. Retrieval can keep token costs low, but graph preprocessing, dynamic behavior coverage, and fixed retrieval depth remain challenges.Source: §7.3 · pp. 29–30

7.4 Analysis-Enriched Scripted Pipelines

  • Analysis-enriched loops use diagnostics beyond tests: SAN2PATCH repairs 31 of 39 VulnLoc vulnerabilities with GPT-4o, while Repilot fixes 27% and 47% more bugs than prior tools on Defects4J 1.2 and 2.0, respectively.These workflows feed sanitizer logs, static constraints, analyzer alerts, or execution outcomes into scripted regeneration. Repilot's fine-grained pruning can improve validity but has unresolved multi-file and cross-language scalability; continuous analyzer feedback can reduce hallucinations and overfitting, at the cost of longer runs and larger prompts, so deeper analysis should be triggered selectively.Source: §7.4 · pp. 30–31

8 Agentic Approaches

Agentic repair delegates runtime choices to LLMs, ranging from tool use in fixed loops to model-controlled planning and termination. Across these designs, greater autonomy supports repository-level repair but brings costs in tokens, latency, coordination, and reliability.

8 Agentic Approaches

  • Tool-augmented, LLM-gated, and self-controlled systems form three autonomy tiers: fixed outer loops, LLM judgments that gate decisions, and LLM authority over planning, delegation, and termination.A critic is agentic only when its output gates acceptance, branching, regeneration, or stopping; an offline score alone is auxiliary evaluation.Source: §8 · p. 31, §8.2 · p. 33

8.1 Tool-Augmented Agents

  • SWE-Agent fixes 12.47% of 2,294 SWE-bench issues without fine-tuning, using model-selected tools within a ReAct loop whose outer control remains static.Each reasoning step adds token and runtime cost, and the agent can select only from its predefined toolkit.Source: §8.1 · pp. 31–33

8.2 LLM-Gated Review

  • TSAPR repairs 201 of 835 bugs on Defects4J under a pass@32 budget and 27 of 79 vulnerabilities on VUL4J, combining tree search with an LLM that generates and gates patches.The authors compare these results against APR and AVR baselines with different patch budgets; dense test coverage can support a Test-as-Judge fallback.Source: §8.2 · p. 33
  • On Google-codebase bugs, abstention and validation raise filtered fail-to-pass@1 on human bugs from 11.29% to 53% and improve filtered accept@1 by up to 24pp on NPE bugs.The bug-abstention model skips low-probability repairs, while a patch validator scores candidate patches and trajectories even after build and regression tests pass.Source: §8.2 · p. 33

8.3 Self-Controlled System

  • 31% solve rate is reported for GPT-4o with SWE-Search on SWE-bench Lite, while Claude 3.5 Sonnet reaches 39% fixes at 2.7 issues per dollar.SWE-Search uses Monte Carlo Tree Search and multi-agent planning and debate; deeper search raises inference-time computation and cost.Source: §8.3 · p. 34
  • 60.0% pass@1 is reported for Claude 3.5 Sonnet on SWE-bench with Learn-by-Interact, which also improves up to 12.2pp over the baseline under the reported protocol snapshot.The framework distills interaction trajectories into reusable instructions; sparse or changing documentation can reduce trajectory quality and add computational overhead.Source: §8.3 · p. 34
  • Self-controlled agents benefit from role separation or self-generated sub-goals and stable feedback, but token budgets, tool latency, and coordination overhead can offset accuracy gains.The surveyed studies identify adaptive stopping, incremental context caching, and lightweight self-reflection as needed for reliable throughput.Source: §8.3 · p. 35
  • Agentic repair expands repository-level autonomy, but token cost, tool latency, context drift, and unstable control loops limit gains; practical systems need stopping rules, context caching, and lightweight safety checks.Self-controlled systems can tackle multi-file or cross-module defects, but incur higher coordination costs and safety risks.Source: §8.3 · pp. 34–35

9 Benchmark Protocol Profiles and Reported Metrics

This section treats published repair scores as protocol snapshots, identifying narrow windows where bounded comparisons are defensible while preserving heterogeneous results as benchmark-family evidence. Its corpus-wide audit finds too little protocol-aligned evidence for a paradigm leaderboard.

Comparison protocol

  • A comparison window requires the same benchmark variant and metric; model-controlled windows additionally require comparable base-model family and repair-scope assumptions.Other differences, including tool interface, search budget, retrieval, validation, training data, and model snapshot, remain confounders; unmatched rows stay in benchmark-family snapshots rather than a leaderboard.Source: §9 · p. 36, Table 10 · p. 37

Reported metrics

  • pass@1 appears in 21 of 66 rows, rising from 1 of 12 systems in 2023 to 5 of 16 in 2024 and 13 of 30 in 2025.pass@𝑘 counts a task solved when any of up to 𝑘 candidates passes; accept@1 and task-specific metrics remain distinct. The 11 security/API/crash-repair rows mix pass@1/pass@5 with exact match/EM, F1, CodeBLEU, correct repairs, success rate, and repair accuracy.Source: §9 · p. 36, Table 12 · p. 38

Defects4J

  • Defects4J rows span pass@𝑘 budgets from 1 to 5000 and differ in localization, evaluated subset, and repair scope, so they support protocol analysis rather than global score ordering.The groups separate localized/Perfect-FL, single-function, tool/search, and single-hunk settings.Source: §9 · p. 37, Table 11 · p. 38, Table 13 · p. 39

HumanEval-Java and EvalRepair-Java

  • EvalRepair-Java has a two-row model-controlled window using CodeLlama 13B and pass@10 without a fault-location prompt; other Java groups retain protocol differences.HumanEval-Java group H1 controls benchmark and pass@10 but varies in model and localization reporting; H3’s pass@100, pass@40, and accuracy rows remain snapshots.Source: §9 · p. 37, Table 14 · p. 39

Fragmented benchmark families

  • Security, API-misuse, crash-repair, and task-specific datasets remain fragmented, while smaller benchmark-family groups are not leaderboards.QuixBugs, BugsInPy, RepoBugs, DS-1000, and GitBug-Java appear in same-family or singleton groups, making their scores harder to compare than the narrow SWE-bench Lite, HumanEval-Java, and EvalRepair-Java windows.Source: §9 · pp. 37–38

SWE-bench

  • 18.00% to 58.30% is the reported SWE-bench Lite pass@1 range, reflecting whole-system stacks rather than isolated paradigm effects.Base models, search depth, cost budgets, and validation policies differ; SWE-bench Verified pass@1 is less uniformly reported and serves as repository-level context, not another aligned window.Source: §9 · p. 37, Table 16 · p. 39, Table 17 · p. 40

Protocol comparability audit

  • Of 2,145 system pairs, 100 share a benchmark label, 32 fall within bounded windows, four share a normalized base model, and only one is model-controlled.The sole model-controlled pair, MORepair versus Luo et al. on EvalRepair-Java, is within Fine-Tuning; no model-controlled cross-paradigm comparison exists. The assurance coding finds 36/66 systems discuss leakage or contamination, 28/66 report active controls, and 39/66 use validation beyond ordinary public tests.Source: §9 · p. 41

10 Open Challenges and Practical Implications

The section identifies five evidence-backed bottlenecks in LLM-based repair and proposes benchmark, cost-control, and patch-acceptance practices to address them. These challenges arise from surveyed systems and evaluation protocols, not generic expectations about LLMs.

Training and deployment cost

  • 19/21 fine-tuning systems use open or academic models, while 37/45 prompting, procedural, and agentic systems use proprietary/API models, making cost and reproducibility central deployment concerns.For 28/66 procedural or agentic systems, loop design, tool use, test feedback, or model-controlled stopping affect inference cost; reporting should include loop budgets and stopping criteria, not accuracy alone.Source: §10 · p. 41

Evaluation reliability

  • Evaluation remains difficult to interpret because localization assumptions, data leakage, oracle strength, and test-suite scope vary, and passing available tests can still leave hidden patch defects.36/66 systems discuss leakage or contamination risks, 28/66 report mitigation or diagnostic control, and 39/66 use validation beyond ordinary public tests; heterogeneous checks are protocol attributes, not directly comparable scores.Source: §10 · pp. 41–42

Workflow robustness

  • Repository-scale and agentic workflows can produce non-repeatable repairs, invalid commands, excessive loops, and unpredictable running times, making outcomes and costs hard to bound [6] [8] [106].Prompt wording, sampling temperature, or random seeds can change patches; randomness may accumulate across agent steps and alter tool sequences.Source: §10 · p. 42

Retrieval and context management

  • Retrieval can miss relevant files or functions, while excessive context can exceed model limits; analysis-derived structured data is often serialized as prompt text [134].A-MEM stores interactions as structured notes, links them in a graph, and supplies compact top-k context to improve recall within a token budget [131].Source: §10 · pp. 42–43

Patch assurance and deployment trust

  • Patch assurance evidence is fragmented across security, API-misuse, and crash-repair datasets and metrics, while only 10 systems expose recorded test, human, or LLM-gated review controls as acceptance gates.The remaining 56 systems do not expose such a gate as their primary control mechanism; deployment readiness needs comparable datasets, explicit oracles, and lightweight static or dynamic checks.Source: §10 · p. 43

Open challenges and practical implications

  • The section recommends multi-hunk benchmarks with stronger oracles, cost-aware repair policies, and independent validation signals so patch acceptance rests on explicit evidence.Proposed measures include contamination-aware splits and marked localization assumptions; model provenance, token budgets, reused analysis, capped calls, and simpler fallbacks; and tests, analysis, sanitizer or exploit reproduction, human review, or calibrated LLM-gated checks.Source: §10 · p. 43

11 Threats to Validity

The survey identifies threats to internal, external, and construct validity and mitigates them through a reproducible screening and coding pipeline, explicit corpus boundaries, and protocol-aware reporting. Its findings describe a criteria-bounded evidence base, not the full field, and heterogeneous benchmark protocols limit cross-paper comparability.

11.1 Internal validity

  • A structured pipeline mitigates internal-validity threats through multi-library searches, explicit screening criteria, independent screening and coding checks, and adjudication of borderline cases.Released scripts and results enable inspection and recomputation using the same notebooks.Source: §11.1 · p. 43

11.2 External validity

  • The 66-system corpus is criteria-bounded, excluding records outside venue or version criteria and systems without extractable benchmark evidence, so counts are not prevalence estimates.Prometheus [14] is logged as a boundary case, not included, because it adds no new protocol-aligned comparison window.Source: §11.2 · p. 44
  • Published benchmark scores remain incomparable across differing k values, model snapshots, retrieval and wall-clock budgets, and fault-localization assumptions; the survey preserves these protocols rather than normalizing them.It mitigates this limitation by separating protocol-aligned windows from benchmark-family snapshots and annotating benchmark scope, metrics, oracle strength, and fault-localization assumptions.Source: §11.2 · p. 44

11.3 Construct validity

  • A stable codebook operationalizes multiple repair dimensions, assigning primary paradigms by parameter adaptation and runtime control while recording auxiliary evidence separately.Released coding sheets and discussion of borderline cases make decisions auditable, but assess reproducibility rather than community consensus; broader assessment remains future work.Source: §11.3 · p. 44

12 Conclusion

The survey organizes 66 LLM-based repair systems by control authority and parameter adaptation, while separately recording auxiliary evidence and evaluation assumptions. Its synthesis clarifies paradigm tradeoffs and identifies evaluation, retrieval, and workflow-cost priorities for future research.

12 Conclusion

  • The survey classifies 66 representative systems by control authority and parameter adaptation, recording retrieval and analysis as separate evidence layers.It also consolidates benchmarks, evaluation assumptions, and pass@𝑘 definitions into common tables.Source: §12 · p. 44
  • Design choices and evaluation protocols jointly shape reported repair success across finetuning, prompting, procedural pipelines, and agentic frameworks.The synthesis clarifies tradeoffs among these paradigms.Source: §12 · p. 45
  • Future work should strengthen less-contaminated benchmarks, improve pass@𝑘 and resource-budget reporting, refine repository-aware retrieval and analysis, and make multi-step planning cost-aware.The survey identifies these directions as central to dependable everyday software-engineering infrastructure.Source: §12 · p. 45

Related papers

Loading 2506.23749v3…