Source-linked AI summary
SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, Martin Monperrus
TL;DR
Automated program repair remains focused largely on simple fixes, motivating a data-driven approach that learns to translate buggy code into fixed code. SEQUENCER combines sequence-to-sequence learning with a copy mechanism and evaluates the result on real commits and Defects4J. It correctly predicts many one-line fixes, while its current scope and evaluation depend on one-line repairs and fault localization.
Problem
Existing automated program repair techniques are limited mainly to simple one-line patches and rely heavily on top-down, domain-specific design.
Method
SEQUENCER uses sequence-to-sequence learning with an encoder/decoder architecture, copy mechanism, and abstract buggy context to generate one-line patches.
Results
950/4,711 testing samples were successfully predicted, and 61 Defects4J patches passed the full test suite.
Takeaways & Limitations
The approach demonstrates sequence-to-sequence learning and copying as a data-driven, generic basis for one-line program repair without domain-specific top-down design.
Takeaways & Limitations
The current approach targets one-line replacement patches and assumes fault localization can identify a buggy line inside a method.
Abstract
from arXiv · showhide
This paper presents a novel end-to-end approach to program repair based on sequence-to-sequence learning. We devise, implement, and evaluate a system, called SequenceR, for fixing bugs based on sequence-to-sequence learning on source code. This approach uses the copy mechanism to overcome the unlimited vocabulary problem that occurs with big code. Our system is data-driven; we train it on 35,578 samples, carefully curated from commits to open-source repositories. We evaluate it on 4,711 independent real bug fixes, as well on the Defects4J benchmark used in program repair research. SequenceR is able to perfectly predict the fixed line for 950/4711 testing samples, and find correct patches for 14 bugs in Defects4J. It captures a wide range of repair operators without any domain-specific top-down design.
1 INTRODUCTION
The paper frames program repair as a promising but limited application of machine learning, and presents SEQUENCER as an end-to-end, language-agnostic approach for one-line fixes. It combines curated commit data, sequence-to-sequence learning, and copying to address source-code vocabulary challenges, with evaluations on real commits and Defects4J.
- Motivation: Automated program repair modifies existing programs to fix bugs, while current techniques remain largely limited to simple, mostly one-line patches.The paper contrasts this scope with the broader ambition of machines writing complete software systems.
- Approach: Sequence-to-sequence learning translates buggy program-token sequences into fixed sequences, using curated commits as training data.The approach adapts a method commonly used for translating human-language sentence pairs.
- Evaluation: 950/4,711 (20%) testing samples were perfectly fixed, using real one-line commits from three open-source projects.The system predicts the fixed line exactly as originally crafted by the developer.
- Evaluation: 119% more samples were correctly patched by SEQUENCER than by the closest related work on the same buggy-input dataset.The paper presents this as a comparative novelty result.
- Approach: SEQUENCER uses a copy mechanism to address source code’s unlimited vocabulary problem and an abstract buggy context to provide long-range code dependencies.Its input is based on full classes while the predicted output is a fixed line.
- Evaluation: On 75 one-line Defects4J bugs, SEQUENCER found 2,321 patches, with 761 compiling, 61 plausible, and 18 semantically equivalent to human patches.The paper also qualitatively analyzes eight repair operators captured by the approach.
2 BACKGROUND ON NEURAL MACHINE TRANSLATION WITH SEQUENCE-TO-SEQUENCE LEARNING
Sequence-to-sequence learning models program repair as translation from buggy to fixed token sequences. Its copy mechanism addresses rare or unseen tokens, a particularly important issue for source code with extremely large vocabularies.
- Sequence-to-sequence learning: SEQUENCER treats buggy code as input and fixed code as output, analogous to neural machine translation between two languages.The model operates on sequences of program tokens rather than natural-language words.
- Sequence-to-sequence learning: An encoder reads input tokens, and a recurrent decoder generates output tokens sequentially, with training using the target outputs to learn next-token generation.A special <EOS> token marks the end of the input sequence.
- Sequence-to-sequence learning: Softmax converts decoder values into probabilities over a learned vocabulary, while recurrent weights capture long-range dependencies in the input.The hidden state after the input sequence encodes information used to begin and continue output generation.
- Vocabulary problem: A learned-vocabulary decoder cannot output tokens absent from its training set, creating a problem for rare words and identifiers.Natural-language proper names illustrate this limitation.
- Copy mechanism: The copy mechanism directly transfers rare or unknown tokens from the input sequence into the output sequence.This is especially useful when the needed token appears in the input but not in the learned vocabulary.
- Copy mechanism: Source code makes copying particularly relevant because developers freely create variable and method names, producing many rare tokens and an extremely large vocabulary.The paper reports that preliminary BPE experiments were less effective than copying for SEQUENCER.
3 APPROACH TO USING SEQ-TO-SEQ LEARNING FOR REPAIR
SEQUENCER formulates one-line program repair as translating an abstract buggy context into a fixed line. Its architecture combines context abstraction, an attention-based sequence-to-sequence network, and a copy mechanism for source-code tokens.
- Problem formulation: SEQUENCER generates a replacement fixed line for a bug localized to a specific source-code line.The system focuses on one-line patches and takes fault-localization data as input while attempting generation for each candidate location.
- Buggy context abstraction: The abstract buggy context organizes fault-localization data into a concise token sequence while retaining information needed for patch generation.It balances shorter sequences, which sequence-to-sequence models handle better, against preserving contextual information for predicting a fix.
- Buggy context abstraction: The context marks the buggy line and retains surrounding method and class information, including variables, initializers, signatures, and non-buggy methods.Non-suspicious method bodies are stripped, and long inputs may be truncated to a predetermined size.
- Sequence-to-sequence network: The sequence-to-sequence network encodes the abstract buggy context and decodes the target fixed line using a bidirectional LSTM encoder and attention.The source sequence X represents the abstract buggy context, while the target sequence Y represents the fixed line.
- Copy mechanism: The copy mechanism lets the decoder select tokens from the abstract buggy context when they are absent from the training vocabulary.It combines vocabulary generation with copying based on the decoder state and attention, addressing rare source-code identifiers and values.
- Patch preparation: Patch preparation converts token predictions into source-code patches because raw outputs may contain unresolved <unk> tokens and spaces between every token.The paper notes that copy-based replacement may not handle every unknown token and that token spacing can produce malformed source code.
4 EVALUATION
The paper evaluates SEQUENCER empirically as part of its assessment of sequence-to-sequence program repair.
- The evaluation section describes the empirical assessment of SEQUENCER.
4.1 Research Questions
The evaluation asks how well SEQUENCER predicts fixed lines, uses copying, repairs Defects4J bugs, and captures repair operators.
- RQ1 asks to what extent SEQUENCER can perfectly predict the fixed line.
- RQ2 examines how often the copy mechanism generates out-of-vocabulary patch tokens and which abstract-context parts supply copied tokens.
- RQ3 evaluates SEQUENCER’s effectiveness on the established Defects4J benchmark.
- RQ4 investigates which repair operators are captured by sequence-to-sequence learning.
4.2 Experimental Methodology
The methodology measures prediction quality on curated real commits, examines copying and context use, and evaluates generated patches on Defects4J under one-line repair assumptions.
- RQ1: Perplexity measures how well SEQUENCER predicts a sample, alongside training and validation accuracy during training.The source and true target sequences are denoted X and Y, and y_i is the i-th target token.
- RQ1: CodRep4Medium contains 1,116 CodRep4 samples whose buggy methods are limited to 100 tokens for comparison with Tufano et al.’s approach.
- RQ2: Copy effectiveness is assessed across CodRep4 by classifying vocabulary status for tokens in successfully predicted lines and locating copied out-of-vocabulary tokens.These locations are used to measure the importance of context, particularly the abstract buggy context.
- RQ3: SEQUENCER is evaluated on Defects4J, a collection of reproducible Java bugs used widely as a program-repair benchmark.
- RQ3: The Defects4J study first focuses on 75 bugs fixed by developers through replacement of a single line.The input is the actual buggy file and line number; generated patches are compiled and executed against the developer test suite.
- RQ3: Generated patches are classified as compilable, plausible, or correct, with correct patches passing tests and being semantically equivalent to the human patch.The categories form a strict inclusion structure: correct implies plausible, and plausible implies compilable.
- RQ4: RQ4 combines grounded theory with targeted analysis to characterize grammatically correct transformations captured by the token-based approach.The targeted analysis searches for cases using copying and cases involving specific programming constructs.
4.3 Training Data
The training data combines curated Java one-line fixes from open-source commit histories, with separate duplicate-free training, validation, and testing splits. Its source-code vocabulary has a long tail, motivating a 1,000-token vocabulary and 1,000-token context limit.
- CodRep and Bugs2Fix provide Java fixes mined from open-source project histories, without requiring test suites exposing buggy behavior.
- 15,548 of 92,849 Bugs2Fix commits are one-line patches within a method, defining the subset matching SequenceR’s problem domain.
- The data uses CodRep datasets 1, 2, 3, and 5 plus Bugs2Fix for training, CodRep4 for testing, and a random 95%/5% training-validation split with no duplicates.
- 35,578 samples comprise the training set, while 4,711 samples comprise the testing set.
- Only 14% of samples exceed the 1,000-token context limit and require truncation.
- The sequence-to-sequence task receives approximately 350 input tokens and produces approximately 6 output tokens on average.
- 567,304 distinct tokens follow a long-tailed distribution, so training is limited to the 1,000 most common tokens.
4.4 Experimental Results
On real one-line fixes, SequenceR exactly reproduces many developer patches and outperforms the compared approach on the restricted benchmark subset. Its copy mechanism is central to generating longer fixes and reusing tokens from broader code context.
- Perfect predictions: 950 of 4,711 CodRep4 tasks receive a predicted line exactly matching the developer’s fix.
- Method length: SequenceR accuracy decreases from 30.8% on CodRep4Medium to 20.2% on CodRep4 as buggy methods become longer.
- Copy mechanism: The longest successful generated patch is 68 tokens, compared with 27 tokens for the longest successful patch without copying.
- Copy mechanism: 641/950 (68%) successful patches copy all copied tokens from the buggy line, while others copy from the buggy method or class.
- Copy mechanism: A non-vocabulary identifier can be copied from the buggy method, enabling a correct patch such as replacing masterNode with nonMasterNode.
- Copy mechanism: 734/950 (77%) successful patches use copied tokens, while only 216/950 (23%) use vocabulary tokens exclusively.
4.5 Answer to RQ3: Defects4J Evaluation
On 75 one-line Defects4J bugs, SequenceR generates many candidate patches, including compilable, plausible, and semantically equivalent patches. The evaluation focuses on patch generation under perfect fault localization and excludes bugs outside methods.
- Evaluation scope: 2,321 patches are generated for 58 of 75 Defects4J bugs; 17 bugs remain unfixed because their faults are not localized inside methods.
- Patch outcomes: 761 generated patches compile, 61 are plausible across 19 bugs, and 18 are semantically equivalent to human patches.
- Patch outcomes: 18 of 61 plausible patches are correct, a ratio of 30%.
- Runtime: The estimated workflow totals 130 machine hours to find patches correctly fixing 9 bugs.
- Copy mechanism: In one Defects4J example, getPct is copied from the buggy class despite being absent from both the vocabulary and buggy method.
- Comparison: Compared with publicly available tools, Elixir, CapGen, and SimFix report 22, 19, and 17 correct patches respectively among the same 75 one-line bugs.
- Evaluation assumptions: SequenceR’s Defects4J evaluation assumes perfect fault localization to isolate patch generation from localization differences among repair systems.
4.6 Answer to RQ4: Qualitative Case Studies
Qualitative cases show SequenceR generating diverse one-line repairs across method calls, conditions, keywords, encapsulation, arithmetic, and token-level edits. The examples include target and call changes, argument addition, condition simplification, and off-by-one repair.
- Method-call repairs: The case studies include call replacement, call deletion, and argument addition in method invocations.
- Method-call repairs: SequenceR captures method-call target changes, such as replacing executorService with scheduledExecutorService.
- Condition repairs: SequenceR can simplify an if condition by removing two clauses from its boolean formula.
- Syntax repairs: SequenceR generates patches replacing programming-language keywords, providing examples described as clues of syntax understanding.
- Encapsulation repairs: The examples include changing direct field access from size to the encapsulating method size().
- Arithmetic and token operations: SequenceR repairs classical off-by-one errors and uses token deletion, addition, and replacement operations.
5 ABLATION STUDY
The ablation study evaluates how architectural, data, context, and training choices affect SequenceR, identifying copy, abstract buggy context, and parameter settings as important factors.
- Ablation protocol: The ablation study varies one parameter at a time against a greedily selected golden model and reports performance on the same testing dataset.Each configuration is run multiple times because learning is random; the study supports model selection and future replication.
- Core components: Copy and abstract buggy context improve the model beyond a single-line seq-to-seq baseline, with the results identifying copy as essential.Table 2 compares beam search, copy, and context variants.
- Vocabulary and pretraining: A 1K-token vocabulary performs better than smaller or larger alternatives, despite the golden model producing 38% raw patches containing <unk> tokens.The 700-token and 1400-token configurations produce 43% and 37% raw <unk> patches, respectively, but the 1000-token model is better optimized.
- Data and training: Combining CodRep and Bugs2Fix data improves model generalization, whereas unsupervised pretraining worsens performance in the tested configuration.Removing the encoder-decoder bridge changed the mean and standard deviation but yielded a lower best result than the golden model.
- Context and capacity: Reducing abstract buggy context to the buggy line lowers accuracy, while truncation is necessary to avoid out-of-memory failure from excessive sequence length.The study also reports that 500-token context hurts accuracy, presumably because it reduces opportunities for token copying.
- Fault localization: Removing <START_BUG> and <END_BUG> labels causes significant accuracy loss, although the network still creates 356 correct patches without test coverage or other fault-localization information.The labels help identify the buggy region and line-break positions needed for valid patch generation.
- Project overlap: Training and testing on the same projects yields a 12% improvement in the best model but requires more complete integration into a project regression system.The primary use case instead applies the golden model to projects absent from training.
6 RELATED WORK
Related work spans static and dynamic program repair, educational repair systems, code-learning methods, and neural patch generation; SequenceR differs in its larger context and global, language-agnostic patch generation.
- Educational systems: sk_p predicts replacement statements from neighboring statements and uses beam search, while other MOOC systems combine learning with sketch-based synthesis or predict bug types from traces.These systems focus on student submissions rather than the real programs used by SequenceR.
- Educational systems: Educational repair systems address compiler or student-program errors, whereas SequenceR targets logical bugs in larger, more complex real programs.DeepFix uses a 129-token vocabulary because compiler-error repair can operate on token types, unlike SequenceR’s setting.
- Learning-based repair: DeepRepair uses machine learning to select repair ingredients from similar code, whereas SequenceR uses learning to generate the actual patch.The distinction concerns learning’s role inside the repair process.
- Neural patch generation: Tufano et al. reduce vocabulary through source-code abstraction and study small and medium methods, while SequenceR addresses source-code vocabulary through copying.Parallel work by Hata et al. uses project-specific training, whereas SequenceR uses global training and evaluates cross-project applicability.
7 CONCLUSION
SequenceR combines sequence-to-sequence learning with copying for program repair and shows results on both held-out real fixes and Defects4J. The authors identify multi-line repair and broader dependency modeling as future directions.
- Contribution: SequenceR combines an encoder-decoder architecture with a copy mechanism to address the large vocabulary of source code.The system is presented as a sequence-to-sequence approach to program repair.
- Implication: The copy mechanism provides a way to alleviate source-code’s unlimited-vocabulary problem and is presented as novel evidence for its use in program repair.This is the paper’s stated contribution relative to its knowledge of prior work.
- Future directions: Future work includes extending SequenceR to multi-line patches, exploring tree-to-tree transformation learning, and testing dependencies beyond the class level.The current context abstraction captures class-level, long-range dependencies.