PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications

Boyang Yang, Zijian Cai, Fengling Liu, Bach Le, Lingming Zhang, Tegawendé F. Bissyandé, Yang Liu, Haoye Tian

arXiv:2506.23749v3cs.SE

TL;DR

LLM-based repair lacks a unified way to distinguish adaptation, prompting, workflows, runtime control, and auxiliary evidence, while benchmark labels alone do not ensure comparable results. This survey codes 66 systems with a hierarchical taxonomy and audits reported evaluation protocols, finding that comparison requires protocol-aligned windows rather than shared benchmark names alone.

  • Problem

    Prior reviews do not unify repair paradigms and auxiliary evidence or analyze protocol differences that make shared benchmark scores unreliable.

  • Method

    The survey bottom-up codes 66 systems using a primary-locus taxonomy and records adaptation, runtime control, evidence, benchmark variants, metrics, models, and fault-localization assumptions.

  • Results

    The taxonomy operationalizes Fine-Tuning, Prompting, Procedural, and Agentic as corpus-wide paradigms and separates protocol-aligned comparison windows from benchmark-family fragments.

  • Takeaways & Limitations

    The synthesis clarifies paradigm trade-offs and identifies stronger benchmarks, transparent reporting, selective retrieval and analysis, and cost-aware planning as future priorities.

  • Takeaways & Limitations

    The survey preserves published scores but does not normalize k values, model snapshots, retrieval budgets, localization assumptions, or wall-clock budgets, and performs no joint reevaluation.

Abstract

from arXiv · show

Large language models (LLMs) are reshaping automated program repair. We present a reproducible hierarchical taxonomy that organizes 66 LLM-based repair systems according to where repair capability and control logic principally reside: task-adapted parameters, prompt and context design, designer-specified workflows, or LLM-directed runtime control. Adaptation, generation pattern, runtime control, and auxiliary evidence are preserved as separate coded dimensions. This representation exposes control distinctions hidden by utilization labels and supports cross-paradigm analysis of system design and evaluation evidence. To the best of our knowledge, it is the first publicly available LLM-based software repair survey to operationalize Fine-Tuning, Prompting, Procedural, and Agentic as one corpus-wide primary classification. Our hierarchy complements prior surveys through one corpus-wide primary decision rule, while the result-level protocol audit bounds which reported results are defensibly comparable. To analyze how repair systems use benchmarks, we record benchmark variants, metrics, base models, and fault-localization assumptions for each system's primary reported result, identifying protocol-aligned comparison windows and benchmark-family fragments. We clarify paradigm trade-offs in task alignment, deployment cost, controllability, and multi-hunk or cross-file repair. We outline open challenges and research directions. Our artifacts and scripted survey pipeline are publicly available at https://github.com/GLEAM-Lab/ProgramRepair.

1 Introduction

The survey proposes a hierarchical taxonomy that classifies 66 LLM-based repair systems by where repair capability and control logic reside, while separately coding auxiliary evidence and evaluation protocols. It uses this framework to expose design distinctions and support more defensible cross-system comparisons.

  • LLM-based repair has progressed from zero-shot single-function prompting toward parameter adaptation, domain guidance, and tool-augmented repository-level or vulnerability repair.
  • Prior surveys organize repair by utilization, scenario, security, tuning, or evaluation roles, but do not jointly model runtime control authority and parameter adaptation.
  • The survey identifies three gaps: no unified four-paradigm taxonomy, no separate auxiliary-evidence layers, and no protocol-level analysis of reported results.
  • A hierarchical primary-locus rule assigns Fine-Tuning, Prompting, Procedural, or Agentic according to where capability and control principally reside, while evidence sources remain non-exclusive fields.
  • The result-level audit records benchmark variants, metrics, base models, and fault-localization assumptions, separating protocol-aligned comparisons from non-comparable benchmark fragments.
  • The survey analyzes paradigm behavior across function-level benchmarks, repository-level repair, vulnerability patching, tutoring, and continuous integration, and releases a scripted, auditable survey pipeline.

2 Related Work

Earlier surveys either predate LLM-centered repair or organize it around narrower dimensions such as utilization, tuning, security, or evaluation. The surveyed work positions its taxonomy as a way to separate adaptation, runtime control, and auxiliary evidence.

  • Earlier APR surveys use search, constraint, template, learning, or localization-generation-ranking-validation views, while treating LLMs briefly or not as central control structures.
  • Recent surveys cover LLM repair, code generation, parameter-efficient tuning, or LLM-as-judge evaluation, but provide limited coverage or narrower workflow taxonomies.
  • Across prior studies, taxonomies generally do not independently separate parameter adaptation, runtime control authority, and auxiliary evidence.
  • The public version history records this survey’s earlier four-way operationalization before a later related taxonomy added an Agent category or Procedural/Agentic split.

3 Survey Methodology

The survey builds a representative corpus through reproducible search, filtering, screening, and snowballing, then codes systems bottom-up with independently audited taxonomy decisions.

  • Study design: The study uses GQM, systematic-review guidance, and snowballing to define research questions, search procedures, and coding of LLM-based repair systems.Its scope covers design paradigms, augmentation layers, evaluation protocols, benchmarks, defect scopes, and correctness or scalability assumptions.
  • Search strategy: Searches combine software-repair, LLM, benchmark, and tool terms across ACM DL, IEEE Xplore, DBLP, and arXiv.The final full query was run on 30 April 2026, with exact templates archived in the replication package.
  • Search strategy: Automated filters retain records mentioning repair tasks, LLMs, and reproducible benchmark-based empirical evaluation before manual screening and citation snowballing.The filtering pipeline includes repair-related, LLM-related, and benchmark-related rules, followed by backward citation chasing.
  • Scope and validity: The corpus is criteria-bounded rather than an exhaustive census, with venue-family choices and representative arXiv inclusion defining its external-validity boundary.The survey mitigates this boundary through multiple libraries, citation snowballing, topical screening, and separate reporting of peripheral archival coverage.
  • Taxonomy construction and reliability: Taxonomy construction uses bottom-up coding of included systems, separating parameter adaptation, runtime control, generation, evidence, repair scope, and deployment scenario.Two coders independently recoded the final corpus and recorded evidence locations before reconciliation.
  • Taxonomy construction and reliability: Coders agreed on 65 of 66 primary-paradigm labels (Cohen’s κ=0.98), 64 of 66 control-subtype labels (κ=0.97), and all retrieval, analysis, and deployment-scenario tags (66/66).These figures quantify reliability for the final 66-system corpus.

4 Selected Works Overview and Taxonomy Distribution

The selected-works overview classifies systems by the principal location of repair capability and runtime control while retaining adaptation, evidence, scope, and scenario as separate dimensions.

  • Primary taxonomy: The taxonomy assigns one primary paradigm using parameter adaptation and runtime-control authority, while preserving auxiliary evidence, scope, and scenario fields for hybrids.This prevents retrieval or analysis augmentation from becoming mutually exclusive top-level categories.
  • Fine-Tuning: Fine-Tuning is assigned when task-specific parameter adaptation carries the principal repair capability and headline result.Using an adapted checkpoint inside a larger workflow is insufficient unless adaptation remains the principal source of the reported result.
  • Prompting: Prompting covers cases where prompts and supplied context provide the main repair mechanism, including zero-shot, few-shot, and reasoning prompts.Multiple independent samples used only to estimate pass@k are evaluation replications rather than workflow steps.
  • Procedural: Procedural systems use designer-specified stage order, tools, retry policy, and stopping budget as the principal runtime control.Agentless, ChatRepair, and Repilot are classified as Procedural despite zero- or few-shot utilization labels.
  • Agentic: Agentic systems place action selection, branching, regeneration, or termination under LLM-directed runtime control.LLM-Gated Review is Agentic only when learned judgment controls acceptance, branching, regeneration, or stopping.
  • Defect scope: Within the 66-system corpus, 55 systems target general program repair, 10 target vulnerability repair exclusively, and 1 addresses both.These figures summarize defect-scope distribution rather than paradigm performance.
  • Publication and temporal distribution: The corpus contains nine arXiv-only preprints and 57 archival publications, while 2025 has the largest number of systems across paradigms.The annual counts are descriptive corpus context, not evidence of a complete field-wide trend.

5 Fine-Tuning Approaches

Fine-tuning improves repair accuracy through supervised, parameter-efficient, reinforcement, distillation, and auxiliary-evidence strategies, but specialization increases dependence on data, rewards, compute, and context quality.

  • Full Fine-Tuning: Full-parameter supervised fine-tuning achieves strong repair accuracy, but large studies report heavy GPU demand, overfitting, and generalization concerns.Performance improvements vary across models and benchmarks, while input/output length limits can prevent correct patch generation.
  • Full Fine-Tuning: VulMaster raises exact-match from 10.2% to 20.0%, BLEU from 21.3% to 29.3%, and CodeBLEU from 32.5% to 40.9% on vulnerable C/C++ functions.Its multi-source inputs and ChatGPT-generated contextual hints support the reported gains, but training requires approximately 30 GB of GPU memory.
  • Full Fine-Tuning: RepairCAT shows that a fully fine-tuned 1B-parameter StarCoder can address specialized AI-generated coding faults at modest computational cost.Its narrow data distribution and limited capacity reduce effectiveness on complex or conventional defects.
  • Parameter-Efficient Fine-Tuning: Specialized PEFT variants can improve repair outcomes, but their effectiveness depends on supervision quality, data diversity, and client or model conditions.QLoRA with rationale-aware optimization performs best against standard fine-tuning in reported EvalRepair settings, while federated LoRA matches centralized training within two percentage points.
  • Parameter-Efficient Fine-Tuning: PEFT reduces adaptation cost by updating compact task-specific parameters while preserving competitive repair performance and reusability of the frozen base model.RepairLLaMA uses rank-8 LoRA covering 0.06% of parameters and, with single-GPU training, fixes 144 Defects4J v2, 109 HumanEval-Java, and 20 GitBug-Java tasks.
  • Auxiliary-Evidence Fine-Tuned Repair: Auxiliary evidence such as execution traces improves repair when defects manifest at runtime, although gathering and encoding richer context challenges scalability.TraceFixer repairs 13−20% more bugs than a code-only baseline, while broader evidence motivates selective retrieval, compression, and context-aware PEFT.
  • Reinforcement Learning Fine-Tuning: Reinforcement and preference-based fine-tuning can steer models with execution, security, or textual rewards, but remain sensitive to reward design and test availability.SWE-RL reports a 41.0% solve rate on SWE-bench Verified using diff-similarity rewards, whereas dynamic execution can increase training costs.
  • Overall Findings: Across fine-tuning approaches, accuracy gains trade generality and training cost for specialization, with PEFT, distillation, and auxiliary evidence reducing some costs.Distillation can shorten decoding time, but fine-tuned systems remain exposed to catastrophic forgetting, narrow data, overfitting, and long-context limitations.

6 Prompting Approaches

Prompting keeps the language model frozen and obtains repairs through zero-shot, few-shot, retrieval-enriched, and analysis-enriched context. Focused examples and auxiliary evidence can improve one-step repair, but prompt length, relevance, sensitivity, and sampling costs remain important constraints.

  • 6 Prompting Approaches: Prompt-based repair treats the LLM as a frozen component that generates a complete patch in one query, while retrieval, curation, and validation remain external.This separates prompting from workflows where the model controls runtime orchestration.
  • 6.1 Zero-shot Prompting: Under one QuixBugs protocol, the highest-repair Codex prompt repaired 23 Python and 14 Java bugs from 40 tasks.The study varied only the static text preceding the buggy function and generated ten candidates per bug.
  • 6.1 Zero-shot Prompting: ChatGPT fixed 84% of student-assignment tasks at TOP-5, while verbose assignment descriptions reduced accuracy relative to concise prompts.The benchmark contained 1,783 incorrect assignments across five classic problems, and no retries or analysis feedback were used.
  • 6.1 Zero-shot Prompting: Zero-shot prompting can repair substantial bug subsets without fine-tuning, retrieval, or multi-turn interaction, but remains fragile for larger multi-hunk patches and imprecise context.AlphaRepair reports up to 3.3 times more repaired bugs than its study baselines on Defects4J.
  • 6.2 Few-shot Prompting: Few-shot prompting stabilizes one-pass generation by placing bug-fix exemplars before the target, but exemplar selection must respect context-window limits.Similarity-ranked, explanation-rich, and retrieval-selected demonstrations require careful curation.
  • 6.2 Few-shot Prompting: Lexically similar exemplars more than doubled exact-match accuracy relative to random exemplars and tripled it relative to zero-shot in one Codex study.The prompts used exactly four retrieved bug-fix pairs copied verbatim before each target.
  • 6.3 Retrieval-Augmented Generation Enhanced Prompting: Retrieval-enriched prompting can reach 76.55% exact-match accuracy while retaining a frozen model, and can support domain, repository, or security-specific repair.CEDAR retrieves four demonstrations automatically, while other approaches use knowledge graphs, repository context, or templates.
  • 6.4 Program-Analysis Evidence in Prompting: Analysis-enriched prompting uses focused slices, failing tests, execution traces, and domain knowledge to guide repairs without parameter fine-tuning.Its central challenge is adaptive context selection because irrelevant or verbose context can hurt performance under prompt-length and cost limits.

7 Procedural Approaches

Procedural approaches keep the LLM frozen within designer-specified repair loops, improving reproducibility and cost control through testing, human feedback, retrieval, or analysis. Richer feedback and context can improve repair effectiveness, but increase prompt, preprocessing, runtime, or engineering overhead.

  • 7 Procedural Approaches: Procedural repair fixes search, retrieval, validation, and iteration logic in advance while invoking an unchanged language model at scripted points.This design yields reproducible outcomes and predictable costs.
  • 7.1 Test-Feedback Loop Pipelines: ChatRepair combines failing assertions with buggy lines and averages 21.86 regeneration rounds, fixing 162/337 Defects4J bugs at roughly $0.42 per bug.The lightweight loop requires no auxiliary analysis.
  • 7.1 Test-Feedback Loop Pipelines: REx reduces LLM calls by roughly 2- to 5-fold and solves 73.7% of Nonlinear Loop Invariant tasks within 300 GPT-4 queries, versus 60.5% for the reported specialized-solver baseline.It uses Thompson sampling over partial patches, with test-pass proportions as rewards.
  • 7.2 Human-in-the-Loop Pipelines: Human feedback turns frozen LLMs into repair partners across tutoring, planning, and review workflows while minimizing computational cost and workflow complexity.CREF achieves an AVG-5 score of 76.6% on TutorCode, while HULA reports 82% plan approval and 56 merged pull requests in its online study.
  • 7.3 Retrieval-Enriched Scripted Pipelines: Retrieval-enriched loops provide targeted dependence context, intent, demonstrations, or repository evidence, but face preprocessing overhead, dynamic-behavior coverage, and token-economy trade-offs.PATCH’s multi-stage, multi-role interaction illustrates how retrieval is paired with explanation, reasoning, and review rather than single-pass generation.
  • 7.4 Analysis-Enriched Scripted Pipelines: Procedural workflows improve predictability through controllable cost and reproducibility, but deeper analysis must be triggered selectively to avoid unnecessary overhead.Continuous analyzer feedback can filter hallucinations and improve localization while increasing runtime and prompt size.

8 Agentic Approaches

Agentic approaches let the LLM choose repair actions and coordinate tools, planning, feedback, and review rather than following a fully prescribed sequence. They expand autonomy and can improve reported repair outcomes, but token cost, latency, context drift, calibration, and control-loop stability constrain deployment.

  • 8 Agentic Approaches: Agentic frameworks are distinguished by model-driven selection of the next repair action instead of handwritten logic selecting a fixed sequence.Their autonomy ranges from tool-augmented loops to self-controlled agents that decide every outer-loop step.
  • 8.1 Tool-Augmented Agents: Tool-augmented agents expose repository, build, test, browsing, or debugging utilities through perceive-think-act loops, with observations conditioning subsequent repair decisions.ReAct describes this interaction pattern, not the repair objective or validation mechanism.
  • 8.1 Tool-Augmented Agents: SWE-Agent fixed 12.47% of 2294 SWE-bench issues without fine-tuning, gaining adaptability by dynamically selecting informative tools at the cost of additional reasoning calls.SWE-Agent M resolved 12.2% of 517 SWE-bench Multimodal tasks, compared with 12.0% for SWE-Agent Base and 9.2% for SWE-Agent JS.
  • 8.2 LLM-Gated Review: LLM-gated review improves solve rates, suppresses patches that exploit weak tests, and provides textual justifications, but requires calibration and stronger-oracle alignment.The survey recommends treating critics as conservative gatekeepers rather than standalone authorities.
  • 8.3 Self-Controlled System: Learn-by-Interact lets the LLM explore environments, plan and execute actions, then distill trajectories into reusable instructions; its reported Claude 3.5 Sonnet result is 60.0% pass@1.The approach depends on accurate, current environment documentation, whose absence can degrade synthesized trajectories and increase computational overhead.
  • 8.3 Self-Controlled System: Structured sub-goals, stable feedback signals, and role separation narrow autonomous search, while token budgets, tool latency, and coordination overhead can offset accuracy gains.The surveyed studies motivate adaptive stopping rules, context caching, and lightweight self-reflection for continuous-integration demands.
  • 8.3 Self-Controlled System: Retrieval is descriptively associated with repository-level or industrial targets: 15 of 29 retrieval-tagged systems versus 1 of 37 without retrieval, odds ratio 38.57.The association is non-causal because paradigm, scenario, task, and publication year are confounded.

9 Benchmark Protocol Profiles and Reported Metrics

The survey treats reported repair results as protocol snapshots, using narrow comparison windows only when benchmark, metric, and key assumptions align. Heterogeneous benchmark variants, budgets, localization assumptions, and metrics therefore limit global score ordering.

  • Comparability is assessed at the reported-result level, separating protocol-aligned windows from benchmark-family snapshots rather than treating shared names as sufficient.The protocol records benchmark variants, metrics, base models, fault localization, and other assumptions.
  • pass@k denotes up to k sampled or considered patches, while exact match, F1, CodeBLEU, and other task-specific metrics remain separate.The survey does not convert task-specific metrics into pass@k.
  • 21 of 66 rows report pass@1, increasing from 1 of 12 systems in 2023 to 13 of 30 in 2025.The security, API, and crash-repair rows use mixed metrics rather than one reporting convention.
  • Defects4J rows differ in pass@k from 1 to 5000, localization assumptions, evaluated subsets, and repair scope, so they support protocol analysis rather than global ranking.The survey separates localized, single-function, tool/search, and single-hunk groups.
  • The narrowest model-controlled Java window uses EvalRepair-Java, CodeLlama 13B, pass@10, and no fault-location prompt.Other HumanEval-Java and EvalRepair-Java rows remain protocol snapshots when their metrics or budgets differ.
  • SWE-bench Lite pass@1 is the most populated comparison window, with reported scores ranging from 18.00% to 58.30% as whole-system results.Base models, search depth, cost budgets, and validation policies differ within the window.
  • Among 2,145 unordered system pairs, 100 share a benchmark label, 32 fall within bounded windows, and only one is model-controlled.The model-controlled pair also matches benchmark variant, metric, candidate budget, base model, localization, and oracle assumptions.

10 Open Challenges and Practical Implications

The survey identifies five recurring bottlenecks: unreliable evaluation, escalating workflow cost, proprietary-model dependence, imperfect context management, and limited patch-acceptance evidence. It recommends stronger benchmarks, cost-aware policies, and explicit assurance mechanisms.

  • Training and deployment cost: 19 of 21 fine-tuning systems use open or academic models, whereas 37 of 45 prompting, procedural, and agentic systems use proprietary/API models.This split makes deployment cost and reproducibility central concerns.
  • Evaluation reliability: Perfect localization, narrow scopes, leakage, and weak tests can make reported repair success an incomplete measure of end-to-end correctness.The corpus records heterogeneous validation strength and reporting formats.
  • Workflow robustness: Repository-scale and agentic workflows can produce unstable tool sequences, invalid commands, loops, and unpredictable running times.Prompt wording, sampling temperature, and random seeds can also change generated patches.
  • Retrieval and context management: Seven retrieval-enriched systems may miss key files or exceed context limits, while analysis-enriched systems often serialize structured diagnostics as prompt text.These limitations make context quality a cross-cutting bottleneck.
  • Patch assurance and deployment trust: Only 10 systems expose test feedback, human feedback, or LLM-gated review as recorded control subtypes, leaving explicit acceptance gates uncommon.The survey calls for comparable datasets, explicit oracles, and lightweight static or dynamic checks before deployment.
  • Five bottlenecks span semantic correctness, repository-scale repair, inference cost, model access, context quality, and patch assurance.They include weak evaluation settings, growing multi-step budgets, proprietary/API dependence, noisy retrieval, and limited acceptance gates.

11 Threats to Validity

The survey addresses threats to internal, external, and construct validity through structured screening, explicit coding, boundary-case logging, and protocol-aware interpretation. Its corpus and comparisons remain bounded by selection criteria and heterogeneous published evaluations.

  • Structured searches, explicit eligibility criteria, independent checks, adjudication, and released scripts mitigate identification, selection, and coding threats.The pipeline enables inspection and recomputation of survey decisions.
  • The 66-system corpus is criteria-bounded rather than exhaustive, so its counts should not be interpreted as prevalence estimates for all LLM-based repair research.Venue-family choices and ranking perspectives can change the peripheral archival set.
  • Published scores are not normalized across k values, model snapshots, retrieval budgets, localization assumptions, or wall-clock budgets, and no joint reevaluation is performed.Protocol-aligned windows and benchmark-family snapshots mitigate but do not remove this comparability limitation.
  • The taxonomy codes parameter adaptation, runtime control, generation pattern, auxiliary evidence, and repair scope using an ordered primary-locus rule.This operationalization addresses ambiguity in definitions and measurements.

12 Conclusion

The survey organizes 66 LLM-based repair systems by control authority and parameter adaptation while treating retrieval and analysis as separate evidence layers. Its synthesis links design choices and evaluation protocols to reported repair success and highlights stronger benchmarks, transparent budgets, and selective repository-aware tooling as priorities.

  • The taxonomy classifies 66 representative systems by control authority and parameter adaptation, with retrieval and analysis recorded separately.The framework covers fine-tuning, prompting, procedural pipelines, and agentic systems.
  • The synthesis shows that design choices and evaluation protocols jointly shape reported repair success across fine-tuning, prompting, procedural, and agentic frameworks.It frames the paradigms as involving distinct trade-offs.
  • Future directions emphasize stronger benchmarks, less contaminated evaluation, transparent pass@k and resource reporting, and selective repository-aware retrieval and analysis.These directions follow from the survey's synthesis of taxonomy and evaluation evidence.
Loading 2506.23749v3…