Source-linked AI summary
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, Denys Poshyvanyk
TL;DR
The paper asks whether developer-written bug fixes mined from project histories can provide enough evidence for NMT to learn code transformations. It mines and abstracts GitHub bug-fix pairs, trains encoder-decoder models to translate buggy code into fixed code, and finds that the models predict developer patches in 9-50% of cases while generating candidates in a split second. The study remains an empirical feasibility investigation rather than an end-to-end repair tool.
Problem
Automated repair struggles with programmer-acceptable patches and test-case overfitting, motivating study of whether mined bug-fix histories can support learning meaningful repairs.
Method
The study mines and abstracts method-level bug-fix pairs from GitHub, then trains NMT encoder-decoder models to translate buggy code into fixed code.
Results
9-50% of small bug-fix pairs are fixed in the test set, up to 2,927 unique fixed bugs, depending on the required number of candidate patches.
Takeaways & Limitations
NMT models can emulate developer AST operations, generate syntactically correct candidate patches in more than 82% of cases, and produce tens of candidates in a split second.
Takeaways & Limitations
The study does not compare NMT with state-of-the-art repair techniques and does not yet implement an end-to-end tool, including automatic patch implementation or test-case execution.
Abstract
from arXiv · showhide
Millions of open-source projects with numerous bug fixes are available in code repositories. This proliferation of software development histories can be leveraged to learn how to fix common programming bugs. To explore such a potential, we perform an empirical study to assess the feasibility of using Neural Machine Translation techniques for learning bug-fixing patches for real defects. First, we mine millions of bug-fixes from the change histories of projects hosted on GitHub, in order to extract meaningful examples of such bug-fixes. Next, we abstract the buggy and corresponding fixed code, and use them to train an Encoder-Decoder model able to translate buggy code into its fixed version. In our empirical investigation we found that such a model is able to fix thousands of unique buggy methods in the wild. Overall, this model is capable of predicting fixed patches generated by developers in 9-50% of the cases, depending on the number of candidate patches we allow it to generate. Also, the model is able to emulate a variety of different Abstract Syntax Tree operations and generate candidate patches in a split second.
1 INTRODUCTION
The study investigates whether Neural Machine Translation can learn developer-like bug-fixing patches from software histories mined in the wild. It evaluates patch prediction, AST-operation emulation, and inference efficiency.
- Motivation: Automated repair approaches struggle to produce programmer-acceptable patches and can overfit test cases.Prior techniques also rely on relatively limited, manually crafted transformations or fixing patterns.
- Motivation: GitHub’s extensive change histories provide bug-fixing examples that machine learning can use to learn bug-fixing activities in the wild.The study builds on the idea that project history can help identify meaningful repair patches.
- Approach: The work evaluates an NMT approach that translates buggy code into fixed code, aiming to emulate real patches written by developers.The approach is an empirical expansion of the authors’ earlier idea of learning bug-fixes.
- Findings: Multiple candidate patches can be produced in less than a second, while test-case execution remains necessary in practice for assessing recommended patches.The NMT approach does not execute test cases during generation, unlike some generate-and-validate approaches.
- Approach: ∼787k bug-fixing commits yield ∼2.3M method-level bug-fix pairs after extracting multiple differences per commit and abstracting their code for NMT training.The process mines commits, extracts method-level AST edits, independently considers method differences, and abstracts the resulting pairs.
- Findings: 9-50% of cases are successfully predicted, depending on the number of candidate patches generated.Over 82% of generated candidates are syntactically correct, and models emulate 27-64% of developers’ AST operations.
2 APPROACH
The approach mines bug-fixing commits, extracts method-level buggy–fixed pairs and edit actions, abstracts code while preserving idioms and mappings, and trains encoder-decoder models on size-specific datasets.
- Data preparation: Bug-fixing commits are mined from thousands of GitHub repositories to create method-level buggy and corresponding fixed code pairs.These bug-fix pairs are the examples used to learn the transformation from buggy to fixed code.
- Data preparation: GumTree identifies the edit actions performed between each buggy and fixed method.The extracted action list represents the method-level differences in each bug-fix pair.
- Abstraction: A Java lexer and parser abstract the bug-fix pairs while retaining frequent identifiers and literals as idioms.The abstraction produces representations better suited for learning.
- Abstraction: The abstraction outputs abstracted pairs and a mapping that enables reconstruction of the original source code.The mapping connects abstract identifiers and literals to their original forms.
- Modeling: Separate datasets group fixes for small and medium methods before encoder-decoder models are applied to each set.The passage describes size-specific datasets followed by model training for each group.
2.1 Bug-Fixes Mining
The authors identify GitHub bug-fixing commits using commit-message patterns, manually validate a sample, and filter changes to obtain focused Java bug-fix data.
- Commit identification: ∼10M (10,056,052) bug-fixing commits are identified by the commit-message search.The search uses the specified combinations of repair-related terms.
- Validation: 97.6% of a statistically significant sample of identified commits are judged to be true positive bug fixes.Two authors independently analyzed 384 commits at a 95% confidence level with a ±5% confidence interval.
- Filtering: Commits involving non-Java files, newly created files, or more than five Java files are discarded to focus on learnable bug-fixes.The filtering ensures that buggy pre-commit code exists and excludes changes spread across many files.
- Filtering: 787,178 bug-fixing commits remain after the mining and filtering process.The result contains buggy and fixed code for the retained commits.
2.2 Bug-Fix Pairs Analysis
The study constructs method-level bug-fix pairs from mined commits, abstracts their code to control vocabulary, and filters examples for learnable transformations.
- A BFP pairs a buggy code component with its corresponding fixed code for learning bug-to-fix translation.
- AST differencing identifies edit actions that transform buggy code into fixed code, including updates, insertions, deletions, and moves.
- Method-level fragments provide a balance between meaningful context and manageable transformation variability for learning fixes.
- ∼2.3M BFPs were extracted by retaining modified mapped methods and excluding methods created or deleted during fixes.
- The extraction omits changes outside methods and treats multiple methods changed in one fixing activity as independent bug fixes.
- Abstraction replaces identifiers and literals with positional IDs while preserving a mapping, yielding a vocabulary of ∼430 tokens and enabling reconstruction of source code.
- 272 frequent identifiers and literals are retained as idioms rather than replaced, helping preserve examples that require common code elements.
- 58k (58,350) and 65k (65,455) bug-fixes remain in BFPsmall and BFPmedium after filtering.
2.3 Learning Patches
The approach trains an RNN Encoder-Decoder on abstracted buggy and fixed methods, then uses beam search to generate multiple candidate patches.
- The models train on pairs of abstracted buggy and fixed code, without supplying AST edit actions to the learner.
- The RNN Encoder-Decoder maps an input token sequence to an output sequence through a learned conditional distribution.
- Attention computes each context vector as a weighted average of encoder states, allowing different input terms to receive different weights.
- The encoder and decoder are trained jointly end-to-end by minimizing negative log likelihood with stochastic gradient descent.
- Beam search retains k highest-probability hypotheses at each step, expanding them over the vocabulary until end-of-sequence tokens are reached.
- When k = 1, beam search is equivalent to greedy decoding; larger beams produce multiple candidate patches for one buggy method.
- A k = 3 visualization shows lower-ranked branches pruned while surviving hypotheses continue until completion.
- Hyperparameter search tests RNN cell types, layer counts, encoder-decoder units, and embedding sizes for both datasets.
3 EXPERIMENTAL DESIGN
The study evaluates whether NMT can learn buggy-to-fixed code transformations using mined bug-fix pairs, beam-search patch generation, and analyses of syntax and AST-operation coverage.
- The study asks whether NMT can learn code transformations from buggy to fixed states using internal and external datasets.
- The authors train and evaluate Encoder-Decoder models on BFPsmall and BFPmedium, selecting configurations on validation data before testing unseen instances.
- Beam Search Decoding generates k candidate patches for each buggy method, with beam sizes ranging from 1 to 50.
- CodRep provides an external dataset of source-code files from real commits across multiple open-source studies for evaluating generalization.
- The evaluation examines perfect fixes, syntactic correctness, AST-operation coverage, theoretical bug coverage, and generated-patch examples.
4 RESULTS
NMT models increasingly reproduce developer fixes as beam width grows, while also covering more operations and generalizing to CodRep. Their internal-dataset performance reaches 50.16% for small BFPs and 28.55% for medium BFPs.
- The best models used bidirectional Encoder and Attention Decoder configurations with 256 units, 512-dimensional embeddings, and LSTM cells.
- 9% and 3% of BFPs were perfectly predicted with one candidate, increasing to 50% and 28% with 50 candidates for small and medium methods, respectively.At beam size 15, perfect predictions reached 40% for small methods and 20% for medium methods.
- At beam size 50, Msmall fixed 2,927 of 5,835 bugs and Mmedium fixed 1,869 of 6,545 bugs exactly as developers did.
- The performance difference between small and medium methods may reflect more potential faulty locations in larger methods.
- On CodRep, beam size 50 produced fixes for 26.66% of small methods and 12.07% of medium methods, supporting evaluation on a heterogeneous external dataset.The CodRep percentages were slightly lower than those for BFP datasets, possibly because of dataset size and differing idioms.
- The RQ1 summary reports developer-inspired fixes for 9.22%-50.16% of small BFP bugs and 3.22%-28.55% of medium BFP bugs, depending on beam width.
4.2 RQ2: What types of operations are performed by the models?
The models learn diverse AST-level bug-fixing operations, produce highly syntactically correct patches, and cover many theoretically fixable bugs across qualitative examples.
- Syntactic correctness: 99% and 98% syntactic correctness occur when Msmall and Mmedium generate one prediction, respectively.Syntactic correctness decreases as the number of generated candidates increases.
- AST operation coverage: 28% and 16% operation coverage are achieved by Msmall and Mmedium with one generated candidate, respectively.Increasing beam size to 5 and 10 sharply increases operation coverage.
- Theoretical bug coverage: 94% and 84% theoretical bug coverage are suggested for Msmall and Mmedium when learned AST actions are combined in perfect fixes.These percentages represent theoretical coverage based on successfully learned and emulated operation combinations.
- Qualitative fix patterns: The models generate fixes involving statement order, missing switch defaults, try-catch scope, method calls, casts, and logic or boolean operators.The qualitative examples include changing || to &&, adding negation, and changing <= to <.
- Observed limitations: The model sometimes produces functionally correct but imperfect patches, and struggles with null checks, operator changes, and drastic developer modifications.Reported failures include incomplete refactoring, missed null checks, opposite boolean behavior, and failure to reproduce major computation changes.
- Summary: The study reports over 82% syntactic correctness and representative learned AST operations that could theoretically fix a large percentage of test-set bugs.Table 4 lists the top-10 operations successfully emulated in perfect predictions.
4.3 RQ3: What is the training and inference time of the models?
Training the models requires a one-time cost of up to 15 hours, while inference remains fast enough to generate many candidate patches for each bug in under a second.
- Training time: Six and 15 hours are required to train Msmall and Mmedium, respectively, using three consumer-level GPUs.The authors characterize this as an acceptable one-time cost for cross-project model construction.
- Inference time: 50 candidate patches can be generated for one bug in less than a second after training for less than 15 hours.This result summarizes the reported training and inference performance.
5 THREATS TO VALIDITY
The study’s validity is constrained by mined-data imprecision, possible hyperparameter effects, limited language and method-size scope, and the absence of direct repair-system comparison or end-to-end validation.
- Construct validity: The mined GitHub bug-fix dataset may contain imprecisions because it was chosen for scale rather than curated benchmark construction.The authors manually analyzed a sample of extracted commits to verify that they represented bug fixes.
- Internal validity: Model performance may depend on the selected hyperparameter configuration.The authors describe their hyperparameter search as mitigation for this internal-validity threat.
- External validity: The study does not compare NMT models with state-of-the-art automated repair techniques.The stated goal was a large-scale feasibility study rather than proposing a novel repair system.
- External validity: An end-to-end repair tool would still require automatic patch implementation and test-case execution to assess patch suitability.These steps are identified as future work.
- Scope: The experiments focus only on Java programs and small- and medium-sized methods.The authors state that the infrastructure could be adapted to other languages by replacing language-specific tooling.
6 RELATED WORK
The paper situates NMT-based patch learning within redundancy-based repair, code search, learned transformation templates, and machine translation for software engineering.
- Automated program repair: Automated program repair commonly relies on the redundancy assumption that programs contain seeds of their own repair.Redundancy-based techniques exploit repetition or existing code during repair.
- Redundancy in software history: Prior studies find that code changes often reuse existing fragments, including same-file temporal redundancy and single-line micro-clones.These findings support searching existing code or history for repair material.
- Repair techniques: Existing repair approaches include deterministic search, test-guided patch ranking, manually written patterns, semantic code search, and feature-based candidate ranking.Examples include GenProg, Prophet, PAR, and SearchRepair.
- Learned repair templates: Genesis learns code-transformation templates from human-written patches but targets three defect types: null pointers, out of bounds, and class casting.The paper contrasts its broader patch-learning aim with this narrower template-learning scope.
- Position of this study: This study tests whether NMT can learn bug fixes from large software corpora and extends earlier feasibility work with beam search, AST-operation analysis, and qualitative evaluation.The data source is real-world project history, including bug-fixing changes mined from repositories.
- Machine translation: NMT systems have replaced traditional statistical approaches as the state of the art in translation, but their data hunger challenges software-engineering applications.Prior work applied statistical machine translation to Java-to-C# method migration.
7 FUTURE WORK
Future work targets expanding the approach beyond short function-level patches and improving how it learns smaller changes within larger methods.
- Training analysis: Training-set scaling from 10% through 90% was deferred because repeated deep-network training and evaluation would require substantial time and computational resources.
- Granularity: The approach currently learns function-level patches limited to fewer than 100 tokens, motivating exploration of class- or package-level granularity.Larger granularity could provide more context for certain bug fixes.
- Segmentation: A proposed segmentation technique would preserve larger-method context while focusing learning on smaller changes.
- Segmentation: The planned methodology would combine deep learning with static analysis, abstracting changes suitable for static tools and retaining more complex fragments for the model.The goal is to balance meaningful context against overwhelming abstraction.
8 CONCLUSION
The study evaluates Neural Machine Translation for learning bug fixes from real-world bug-fix pairs mined and abstracted from open-source histories. Models translate buggy code into fixed code, fixing thousands of unique bugs with syntactic validity, AST-operation coverage, and rapid candidate generation varying across settings.
- Conclusion: The study mines, extracts, and abstracts real bug-fixes into method-level bug-fix pairs, then trains NMT models to translate buggy code into fixed code.The evaluation examines feasibility, predicted-patch types and quality, and training and inference time.
- Conclusion: More than 82% of generated patches were syntactically correct.
- Conclusion: Msmall emulated 28-64% of developer AST operations, while Mmedium achieved 16-52% coverage.
- Conclusion: The models generated tens of candidate patches in a split of a second.
- Conclusion: The study provides an empirical foundation for developing and evaluating program-repair techniques based on NMT.