PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Learning-based Automated Program Repair

Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, Zhenyu Chen

arXiv:2301.03270v3cs.SE

TL;DR

Manual software debugging is costly, while learning-based APR studies have become difficult to navigate because of varied techniques, datasets, metrics, and design choices. This paper systematically surveys the field, its workflow, evidence, applications, and challenges, reporting a comprehensive review of 112 studies and 53 datasets. It also identifies practical limitations and directions for future learning-based APR research.

  • Problem

    The growing and heterogeneous learning-based APR literature makes it difficult to understand the state of the art and improve existing techniques.

  • Method

    The paper systematically surveys learning-based APR techniques, workflows, datasets, metrics, empirical studies, applications, and open-science issues.

  • Results

    The survey analyzes 112 relevant studies and 53 collected datasets while organizing learning-based APR around its major repair and evaluation components.

  • Takeaways & Limitations

    The survey provides researchers with a comprehensive view of learning-based APR’s progress, advantages, limitations, challenges, and future directions.

  • Takeaways & Limitations

    Learning-based APR faces reproducibility constraints from substantial machine-resource requirements, while test-based validation and match-based metrics have important limitations.

Abstract

from arXiv · show

Automated program repair (APR) aims to fix software bugs automatically and plays a crucial role in software development and maintenance. With the recent advances in deep learning (DL), an increasing number of APR techniques have been proposed to leverage neural networks to learn bug-fixing patterns from massive open-source code repositories. Such learning-based techniques usually treat APR as a neural machine translation (NMT) task, where buggy code snippets (i.e., source language) are translated into fixed code snippets (i.e., target language) automatically. Benefiting from the powerful capability of DL to learn hidden relationships from previous bug-fixing datasets, learning-based APR techniques have achieved remarkable performance. In this paper, we provide a systematic survey to summarize the current state-of-the-art research in the learning-based APR community. We illustrate the general workflow of learning-based APR techniques and detail the crucial components, including fault localization, patch generation, patch ranking, patch validation, and patch correctness phases. We then discuss the widely-adopted datasets and evaluation metrics and outline existing empirical studies. We discuss several critical aspects of learning-based APR techniques, such as repair domains, industrial deployment, and the open science issue. We highlight several practical guidelines on applying DL techniques for future APR studies, such as exploring explainable patch generation and utilizing code features. Overall, our paper can help researchers gain a comprehensive understanding about the achievements of the existing learning-based APR techniques and promote the practical application of these techniques. Our artifacts are publicly available at \url{https://github.com/QuanjunZhang/AwesomeLearningAPR}.

1 INTRODUCTION

Learning-based APR has emerged to address the cost and difficulty of manual bug fixing by learning bug-fixing patterns from code corpora. This survey organizes the field’s techniques, workflows, datasets, metrics, empirical studies, applications, and challenges.

  • Motivation: Manual debugging is costly and error-prone, motivating APR techniques that automatically fix software bugs.Software debugging accounts for over 50% of software development cost.
  • Learning-based APR: Learning-based APR uses deep learning to automatically learn bug-fixing patterns from large source-code corpora.These techniques can support multiple languages and multi-hunk repairs when trained on buggy–fixed code pairs.
  • Outlook: The paper aims to help researchers understand the field’s advantages and limitations and advance learning-based APR practice.The authors also identify current challenges and suggest future research directions.
  • Survey scope: The survey reviews learning-based APR workflows, techniques, datasets, metrics, empirical studies, industrial applications, pre-trained models, and open-science challenges.It analyzes 112 studies and 53 datasets, and covers fault localization, preprocessing, patch generation, ranking, validation, and correctness.
  • Positioning: Unlike earlier surveys centered mainly on traditional repair, this survey focuses on deep-learning integration, repair domains, challenges, and studies through November 2022.The covered domains include vulnerability and syntax-error repair.

2 SURVEY METHODOLOGY

The survey combines database searching, keyword filtering, manual screening, deduplication, and snowballing to identify relevant learning-based APR studies. This process produced a final collection of 112 papers and revealed rapidly increasing publication activity since 2020.

  • Search process: The search covered Google Scholar, ACM Digital Library, and IEEE Xplore using separate APR-related and deep-learning-related keyword groups.The resulting search string combined program-repair terms with neural, machine-learning, transformer, transfer, and supervised-learning terms.
  • Study selection: Filtering removed papers published before 2016, papers shorter than seven pages, and duplicates, leaving 283 papers for manual review.The 2016 cutoff reflects the first learning-based APR study identified by the survey.
  • Study selection: Manual relevance screening followed by snowballing produced a final survey corpus of 112 papers.Snowballing recovered relevant studies missed by the keyword search, including a syntax-error repair paper.
  • Trend observation: The collected learning-based APR literature increased rapidly after 2020.The survey attributes this trend partly to a plateau in traditional APR and deep learning’s success in related tasks such as translation.

3 BACKGROUND AND CONCEPTS

APR searches for a small program change that satisfies available specifications, usually test suites, while learning-based APR formulates repair as translating buggy code into fixed code. The background also highlights overfitting risks and traditional search-based repair strategies.

  • Automated Program Repair: APR seeks a program variant that satisfies a specification while remaining within a permitted edit distance from the buggy program.The formalization uses a buggy program, specification, transformation operators, and maximum edit distance.
  • Automated Program Repair: Most APR techniques use test suites as specifications, but incomplete tests make correctness of plausible patches difficult to ensure.A patch that passes the available tests is plausible, yet it may overfit those tests and fail to generalize.
  • Automated Program Repair: Traditional APR commonly localizes suspicious code, generates candidate patches through transformations, and validates candidates against the original tests.A plausible patch passes the tests, while a correct patch is also semantically equivalent to the intended repair.
  • Traditional APR: Traditional APR includes heuristic-based techniques that search repair spaces constructed from prior patches or similar code snippets.SimFix intersects abstract and concrete search spaces before applying heuristic search.
  • Neural Machine Translation: Learning-based APR treats repair as an NMT task that maps a buggy code sequence X to a fixed code sequence Y.Seq2Seq models learn this mapping by maximizing the conditional probability of the target sequence given the source sequence.

4 LEARNING-BASED APR

This section presents the workflow of learning-based automated program repair tools and introduces popular techniques through examples.

  • The section explains the workflow followed by learning-based APR tools.
  • It introduces several popular learning-based APR techniques.
  • Examples are used to illustrate the techniques discussed.

4.1 Overall Workflow

Learning-based APR processes buggy programs through localization, preprocessing, neural patch generation, ranking, validation, and correctness assessment. Each phase transforms inputs or filters candidates before deployment.

  • Fault localization and preprocessing: The workflow localizes suspicious code elements before preprocessing buggy snippets into model-ready tokens.Localization returns statements or methods; preprocessing applies context, abstraction, and tokenization.
  • Patch generation: Patch generation embeds processed tokens, encodes them, and decodes fixed-code candidates from vocabulary probabilities.The decoder uses encoder states and previously generated tokens to produce probability distributions.
  • Patch ranking: Beam search ranks candidate patches by iteratively selecting high-scoring tokens according to estimated likelihood.The ranking stage prioritizes several candidate patches from the model’s prediction distribution.
  • Validation and correctness: Patch validation checks generated candidates against program specifications such as functional tests or static analysis tools.Only candidates passing the available specification proceed to correctness assessment.
  • Validation and correctness: Correctness assessment estimates whether plausible patches overfit the existing specification before developers manually check them for deployment.Plausible patches pass the specification but may still be incorrect.

4.2 Fault Localization

Fault localization identifies suspicious program elements and directly affects repair performance. Learning-based APR uses spectrum-based and other localization methods, while perfect localization can isolate model evaluation from localization noise.

  • Localization techniques: Fault localization ranks program elements by suspiciousness scores computed from static analysis or dynamic execution information.Common families include spectrum-based, mutation-based, and slicing-based techniques.
  • Localization techniques: Learning-based APR commonly adopts spectrum-based fault localization, such as Ochiai, to identify buggy lines and related code structures.DLFix extracts AST nodes related to the localized line, while Recoder also uses Ochiai.
  • Localization assumptions: Perfect-based localization assumes the genuine buggy element is known, enabling evaluation independent of localization techniques.Several systems manually extract changed statements or otherwise use ground-truth locations.
  • Localization techniques: Some APR systems perform localization themselves by ranking potentially erroneous lines or candidate lines to modify.DeepFix uses beam search, while Prophet analyzes dynamic execution traces.
  • Localization assumptions: Fault localization can bias APR evaluation, and a wrong suspicious element prevents a correct patch from being generated.The survey identifies localization as a preceding step with significant impact on patch-generation performance.

4.3 Data Pre-processing

Data preprocessing converts buggy code into tokens suitable for neural repair models through context selection, code abstraction, and tokenization. These choices trade information coverage, vocabulary size, and model complexity.

  • Preprocessing components: Preprocessing analyzes buggy snippets and returns processed tokens using code context, abstraction, and tokenization.These three components define how code enters training and inference.
  • Code context: Broader context supplies more fix ingredients but can introduce vocabulary noise and long-term dependencies, whereas narrow context may omit necessary semantics.The survey identifies a trade-off between vocabulary size and context size.
  • Code context: Context granularity ranges from context-free statements to class-level inputs that provide richer information.Class-level context can preserve important surrounding code while reducing input complexity through abstraction.
  • Code abstraction: Code abstraction renames natural code elements to predefined tokens, reducing vocabulary size and filtering irrelevant information.Identifiers and literals may be abstracted while semantic idioms are retained.
  • Code abstraction: Some systems retain raw source code because identifiers can carry semantic information, creating a trade-off with abstraction.Function names may communicate intended behavior, while abstraction emphasizes common patterns.
  • Code tokenization: Tokenization converts source code into model identifiers using character-, word-, or subword-level schemes.Subword methods such as BPE, BBPE, and SentencePiece reduce vocabulary size; camel-case handling supports source reconstruction.

4.4 Patch Generation

Patch generation is the central learning-based APR phase, combining code representation with neural architectures to learn transformations from buggy to fixed code. Approaches use sequence, tree, and graph structures, with representations ranging from direct translation to prompts and masks.

  • Overview: Patch generation represents source code and selects an encoder-decoder architecture to learn buggy-to-correct transformations.The survey organizes techniques by code representation and model architecture.
  • Sequence-based generation: Sequence-based methods treat APR as token-to-token translation, commonly feeding buggy code directly into sequence-to-sequence models.These models may overlook programming-language structure and generate syntactically invalid patches.
  • Input representations: Context, prompt, and mask representations distinguish surrounding code, formulate repair as prompted completion, or fill masked buggy lines.Prompt representations concatenate components with labels, while mask representations support cloze-style prediction.
  • Model architectures: Pre-trained models are self-supervised on large unlabeled corpora and then fine-tuned on limited labeled repair data.CodeBERT is one example used for learning-based APR.
  • Tree-based generation: Tree-based methods parse code into ASTs and use structure-aware models to capture syntactic information during patch generation.DLFix encodes context and changed subtrees with tree-based recurrent models.
  • Graph-based generation: Graph-based representations model AST nodes and their relations, including child, sibling, data-dependence, and control-dependence edges.Recoder uses directional AST graphs, while other approaches use data and control dependence graphs.

4.5 Patch Ranking

Patch ranking prioritizes likely-correct candidates within an enormous search space, balancing coverage, retrieval speed, and code-informed prioritization.

  • The decoder can generate V^l candidate patches, making it impractical to validate the full combinatorial search space.Ranking reduces the number of candidates requiring inspection or validation.
  • Patch ranking prioritizes candidate patches with a high probability of correctness to improve inference efficiency and reduce manual validation effort.
  • Beam search is the most common strategy, retaining k probable tokens at each iteration and ranking them using likelihood estimates over d prediction steps.
  • Vanilla beam search can produce uncompilable patches because token likelihoods ignore code information such as valid variables.
  • Learning-based APR also uses feature-based and masked-language-model ranking methods, including Prophet and AlphaRepair.
  • Effective ranking should provide sufficient search coverage, retrieve candidates efficiently, and prioritize likely-correct patches using syntactic and semantic code features.

4.6 Patch Validation

Patch validation filters ranked candidates by compiling and testing them, but dynamic execution becomes costly at the scale of learning-based APR.

  • Patch validation takes ranked NMT-generated candidates and returns plausible patches for deployment.
  • Recompiling programs and executing available test suites can automatically filter hundreds or thousands of candidates, including 1,000 candidates per bug in CIRCLE.
  • Validation cost can become substantial: CURE generates 10,000 candidates per bug and validates its top 5,000, while AlphaRepair returns at most 5,000 candidates per bug.
  • On-the-fly validation frameworks such as UniAPR and self-boosted prioritization tools such as SeAPR seek to accelerate dynamic execution.
  • Dynamic execution is standard for checking compilability and test-suite behavior, but its cost motivates validation optimizations and learning-based methods tailored to APR.

4.7 Patch Correctness

Patch correctness assessment addresses overfitting after test-based validation by using static, dynamic, semantic, and learned evidence to distinguish correct patches from plausible ones.

  • Test suites are incomplete program specifications, so patches that pass available tests may not generalize and can overfit.
  • Despite encouraging progress, patch overfitting still limits practical deployment and motivates patch-aware features and stronger pretrained models.
  • Traditional APCA studies use execution-trace similarity, while learning-based approaches apply feature extractors and classifiers to predict correctness.
  • Table 1 compares learning-based APCA techniques by publication time, language, features, datasets, and public repositories.
  • Static representation methods can filter incorrect patches; one QuixBugs study removed 45% (16/35) of incorrect patches, although it used one BERT model on 40 one-line bugs.
  • ODS uses 202 AST-derived static features and gradient boosting, achieving 71.9% accuracy in detecting overfitting patches across 26 projects.
  • Later methods incorporate unchanged-code context, failing-test specifications, execution semantics, bug-report similarity, and combined static-dynamic measures.

4.8 State-of-the-Arts

Learning-based APR has progressed from sequence-based translation toward tree-, graph-, context-, and reward-aware models that address vocabulary, code structure, multi-hunk repair, and patch quality. Representative systems demonstrate broader repair coverage and improved repair outcomes across diverse benchmarks.

  • Sequence-based Approaches: 36% of pull requests received the same patch from an NMT model, demonstrating that sequence models can learn meaningful code changes.The study mined method pairs before and after pull requests and used code abstraction to reduce vocabulary size.
  • Sequence-based Approaches: CoCoNut fixed 509 bugs across six benchmarks and four programming languages, including 309 bugs not fixed by earlier APR tools.Its context-aware architecture separates the buggy line from method context, while ensemble learning captures diverse fixes.
  • Sequence-based Approaches: CURE fixed 57 Defects4J bugs and 26 QuixBugs bugs by combining pre-training, code-aware beam search, and sub-word tokenization.These components target developer-like code learning, uncompilable patch avoidance, and out-of-vocabulary words.
  • Sequence-based Approaches: RewardRepair fixed 207 bugs across four benchmarks and reached a 45.3% compilable rate among Top-30 candidates, exceeding CURE's 39%.Its mixed loss uses compilation and test-execution feedback rather than token similarity alone.
  • Tree-based Approaches: Tree-based approaches model AST structure for code transformations, while DEAR extends DLFix to multi-hunk, multi-statement repair.DEAR fixed 164 more bugs than DLFix across Defects4J, BigFix, and CPatMiner, including 61 multi-hunk or multi-statement bugs.
  • Graph-based Approaches: Graph-based approaches represent programs with AST or feedback graphs and learn graph transformations, edits, or syntax-guided expansions.Examples include HOPPITY, DrRepair, GRAPHIX, and Recoder, which incorporate graph structure, diagnostic feedback, pre-training, or AST-aware decoding.

5 PRE-TRAINED MODEL-BASED REPAIR

Pre-trained models for APR are organized around universal models adapted to repair and specific APR techniques, including fine-tuning, zero-shot generation, and multimodal repair. The surveyed approaches span several architectures and demonstrate expanding practical and empirical use of pre-training.

  • 5. PRE-TRAINED MODEL-BASED REPAIR: Pre-trained APR techniques are commonly divided into universal models supporting multiple code tasks and specific models designed only for program repair.Universal models are transferred across downstream tasks, whereas specific techniques focus on APR.
  • 5.1 Universal Pre-trained Model-based APR Techniques: T5-like encoder-decoder architectures are widely used because they naturally formulate program repair as code generation.Encoder-only and decoder-only architectures also appear, but encoder-only models require an additional decoder for generation.
  • 5.1 Universal Pre-trained Model-based APR Techniques: CodeT5 incorporates identifier-aware objectives to capture code semantics while supporting code understanding and generation tasks.Its unified framework also supports multi-task learning.
  • 5.2 Specific Pre-trained Model-based APR Techniques: Specific APR techniques use pre-trained models through fine-tuning, zero-shot generation, multimodal inputs, and staged syntax-repair workflows.Examples include TFix, SynShine, MODIT, AlphaRepair, and VulRepair.
  • 5.2 Specific Pre-trained Model-based APR Techniques: SynShine outperforms DeepFix and SequenceR on Blackbox token ranges and is integrated with VSCode for public use.Its workflow combines BlockFix, LineFix, and UnkFix with RoBERTa-based pre-training and compiler diagnostics.
  • 5.2 Specific Pre-trained Model-based APR Techniques: The survey identifies a growing body of empirical studies evaluating how pre-trained models perform across repair scenarios.These studies complement newly proposed pre-trained APR techniques.

6 EMPIRICAL EVALUATION

The survey examines datasets, evaluation metrics, and empirical studies that shape learning-based APR assessment. It highlights the separation between training and evaluation data, the tension between execution- and match-based metrics, and evidence for industrial and cross-scenario applicability.

  • 6.1 Dataset: Learning-based APR commonly uses supervised training on large labeled datasets followed by evaluation on smaller selected datasets.Training data generally contains bug-fixing pairs, while evaluation data may additionally include test suites for patch validation.
  • 6.1 Dataset: Defects4J remains the most widely adopted benchmark, while newer datasets broaden coverage across languages and repair settings.The collected datasets mainly cover Java, JavaScript, Python, C, and C++; Java remains the most targeted language.
  • 6.1 Dataset: High-quality training datasets remain important because NMT-based APR depends on bug-fixing pairs and their creation requires extensive manual effort.Large-scale datasets were curated to support more persuasive experiments and model learning.
  • 6.2.2 Match-based Metrics.: Accuracy and BLEU efficiently compare candidate patches with reference patches, but token matching can overlook equivalent code semantics and important syntax.CodeBLEU is adopted as a code-oriented variant of BLEU to address these concerns.
  • 6.2 Evaluation Metrics: Learning-based APR evaluation combines execution-based metrics such as plausible patches with match-based metrics such as BLEU.The survey describes this combination as a characteristic difference from traditional APR evaluation.
  • 6.3 Empirical Study: NMT models fixed buggy methods in 9%–50% of cases on BFP-small and BFP-medium, supporting further APR research with neural translation.The result comes from an empirical study that trained models to translate buggy methods into corresponding correct methods.
  • 6.3 Empirical Study: Industrial transfer learning improved TFix performance on Samsung Kotlin defects, fixing 289 more defects after fine-tuning than the original model.The original model fixed 94 of 1,961 defects, while additional transfers fixed 211 more defects than transfer using only defect-fixing knowledge.
  • 6.3 Empirical Study: GraphCodeBERT with data-flow information performed significantly better than CodeBERT for C/C++ vulnerability repair across five CWE types.The comparison highlights the relevance of code dependencies in pre-trained vulnerability-repair models.

7 APPLICATION AND DISCUSSION

Learning-based APR has expanded from end-to-end patch generation into industrial deployment and hybrid systems that enhance traditional repair components. The survey also identifies reproducibility, artifact availability, and rapidly evolving pre-trained-model research as important application concerns.

  • Industrial Deployment: Industrial deployments such as Getafix, Google’s hybrid models, and BUGLAB demonstrate learning-based APR in practical bug-fixing and detection settings.Getafix ranks fixes from historical repairs; Google’s sandwich and GREAT models combine structural or global information; BUGLAB uses self-supervised learning.
  • Industrial Deployment: The survey reports that learning-based APR has gained industrial adoption through tools associated with Meta, Fujitsu, Bloomberg, Alibaba, GitHub, and Microsoft.The reported applications emphasize collaboration between developers and automated systems in producing reliable software.
  • DL for Traditional APR: Learning-based APR research increasingly combines neural models with traditional repair techniques instead of relying only on end-to-end patch generation.Deep learning can improve components such as repair-template selection, while ACS uses corpus frequencies to infer predicates.
  • Open Science: Reproducing learning-based APR is difficult because training requires substantial computation, specialized environments, hyperparameters, and random seeds.The survey links these requirements to expensive equipment and resource demands, while also noting that only a few studies publicly release source code.
  • Latest Advancements: The survey covers advances through November 2022, while acknowledging that substantial 2023 developments fall outside its comprehensive analysis.It highlights increasing use of pre-trained models, diverse bug types, traditional-APR combinations, and improvements to code context and patch validation.
  • Latest Advancements: Recent APR studies increasingly use pre-trained models, including type-aware and domain-rule-informed architectures for generating structured patches.Tare incorporates type information into graph-based repair, while KNOD uses tree decoding and domain-rule distillation.

8 IMPLICATION AND GUIDELINES

The survey proposes guidelines for improving learning-based APR through better representations, validation, datasets, explainability, and integration with traditional repair. It emphasizes broader repair domains and interactive processes while identifying overfitting and evaluation weaknesses as continuing challenges.

  • Code Representation: Future APR models should systematically evaluate code representations across model configurations and tailor representations to specific repair scenarios.The survey specifically encourages studying representations such as structural and dynamic features rather than assuming one representation is optimal.
  • Patch Validation: Patch validation should distinguish correct patches from overfitting by combining semantic understanding with evaluation methods targeted to learning-based repairs.The survey recommends studying differences between traditional and learning-based patches and developing predictive validation techniques.
  • Training Dataset Construction: Training datasets require standardization because automated mining introduces noise and existing datasets vary substantially in sample size.The survey notes duplicated samples and contrasts 3,241,966 samples in CoCoNut with 2,000 in DLFix, which can bias comparisons.
  • Evaluation: More practical evaluation should address the limits of test-based plausibility, static matching, and manual inspection when assessing patch correctness.The survey proposes hybrid metrics combining dynamic execution and static matching, while noting that manual inspection is resource-intensive and error-prone.
  • Repair Process: A unified repair process could let fault localization and patch generation exchange feedback, improving both tasks through interaction.This recommendation differs from treating localization and generation as separate stages.
  • Traditional APR Integration: Deep learning can enhance traditional APR components, while predictive technique selection can match a buggy project with an appropriate repair approach.The survey gives donor-code retrieval in TBar as an example where pre-trained models could improve template-based repair.
  • Repair Domains and Explainability: Future work should extend learning-based APR to domains such as test, concurrency, and API-misuse repair and improve explainability through attention or input perturbation.The survey frames attention as a white-box attribution mechanism and input perturbation as a black-box explanation mechanism.
  • Pre-trained Models: Pre-trained models remain a major research direction, but general-purpose pre-training may mismatch APR’s need to learn bug-fixing patterns from code pairs.The survey distinguishes snippet-oriented pre-training objectives from APR’s paired-input transformation objective.

9 CONCLUSION

Automated program repair addresses the costly task of fixing software bugs and can reduce manual debugging effort. The survey concludes that learning-based APR has shown promising results while documenting its frameworks, evidence, and remaining challenges.

  • Conclusion: Automated program repair targets automatic software-bug fixing and can alleviate manual debugging effort.The paper connects APR with software testing, validation, and debugging practices.
  • Conclusion: The survey covers learning-based repair frameworks, datasets, metrics, empirical studies, and strategies for fault localization, generation, ranking, validation, and correctness.It presents these components as the basis for understanding existing learning-based APR techniques.
Loading 2301.03270v3…