PaperlayerView exact revision on arXiv

Source-linked AI summary

Patching as Translation: the Data and the Metaphor

Yangruibo Ding, Baishakhi Ray, Premkumar Devanbu, Vincent J. Hellendoorn

arXiv:2008.10707v2cs.SEcs.LGcs.PL

TL;DR

The paper asks whether models transplanted from machine translation are well suited to program repair, where the repair search space may be expanded by translating whole buggy lines. It empirically studies translation-based and edit-based models, finding that edit-based enhancement fixes more bugs with its top prediction. The authors conclude that translation does not capture bug fixing well and that edit-generating models show promising improvements.

  • Problem

    Whether translating buggy programs into repairs is generally feasible, and whether translation expands the repair search space despite real-world fixes usually changing only a few tokens.

    Source: §2.2 · p. 3

  • Method

    The paper empirically analyzes repair characteristics and model outputs, comparing translation-based seq2seq models with models that predict code edits conditional on buggy context.

    Source: §3.3.1 · p. 4, §7 · p. 11

  • Results

    22 additional bugs were fixed on the test set by the edit-based enhancement over the baseline model using its top prediction alone.

    Source: §5.2 · p. 10

  • Takeaways & Limitations

    The translation paradigm does not capture bug fixing well, while generating edits rather than raw tokens offers a promising improvement in this study.

    Source: §7 · p. 11

    The study did not compare its models with state-of-the-art NMT-based repair tools because its goal was to evaluate the general translation idea, not present the best-performing models.

    Source: §6 · p. 11

Abstract

from arXiv · show

Machine Learning models from other fields, like Computational Linguistics, have been transplanted to Software Engineering tasks, often quite successfully. Yet a transplanted model's initial success at a given task does not necessarily mean it is well-suited for the task. In this work, we examine a common example of this phenomenon: the conceit that "software patching is like language translation". We demonstrate empirically that there are subtle, but critical distinctions between sequence-to-sequence models and translation model: while program repair benefits greatly from the former, general modeling architecture, it actually suffers from design decisions built into the latter, both in terms of translation accuracy and diversity. Given these findings, we demonstrate how a more principled approach to model design, based on our empirical findings and general knowledge of software development, can lead to better solutions. Our findings also lend strong support to the recent trend towards synthesizing edits of code conditional on the buggy context, to repair bugs. We implement such models ourselves as "proof-of-concept" tools and empirically confirm that they behave in a fundamentally different, more effective way than the studied translation-based architectures. Overall, our results demonstrate the merit of studying the intricacies of machine learned models in software engineering: not only can this help elucidate potential issues that may be overshadowed by increases in accuracy; it can also help innovate on these models to raise the state-of-the-art further. We will publicly release our replication data and materials at https://github.com/ARiSE-Lab/Patch-as-translation.

1 INTRODUCTION

The paper examines whether neural machine translation is an appropriate model for program repair, identifying mismatches in the information available, repair behavior, and training objective. It also tests edit-based models that predict changes relative to buggy code.

1 INTRODUCTION

  • Buggy fragments may lack information needed for repair, making fixes ambiguous without broad context that can include surrounding files.The paper argues that reliable repair may be impossible without this broader context for many real-world bugs.Source: §1 · p. 1
  • Repairs often preserve nearly all buggy-line tokens, so the few changed tokens are crucial, yet left-to-right translation models struggle to predict that transition.The large amount of copying distracts from and inflates the training quality signal.Source: §1 · p. 2
  • Token-probability losses and overlap measures such as BLEU poorly reflect repair quality, and models produce syntactically incorrect, repetitive patches that limit search-based repair.The paper links these issues to the mismatch between token overlap and repair quality, given few token changes and missing context.Source: §1 · p. 2
  • Edit-based models that predict insertions and deletions relative to buggy code generate correct patches more often and achieve better top-K accuracy than translation-based models.A generic context extension proved less effective, likely because existing models struggle to model large context windows.Source: §1 · p. 2

2 BACKGROUND

The background contrasts language translation with program repair and reviews models used in both areas. It frames the central question as whether translation assumptions fit repair, and introduces edit-based methods as an alternative.

2.1 Neural Machine Translation

  • Translation models encode an input and generate an equivalent output, commonly producing tokens autoregressively from left to right; the paper uses a Transformer model.NMT usually treats strongly overlapping translations as high quality, although context can matter and multiple translations may be valid.Source: §2.1 · pp. 2–3
  • Unlike translation, repair typically makes small changes that alter buggy code’s semantics, so seq2seq assumptions require empirical evaluation rather than automatic adoption.The paper distinguishes repair from tasks such as code summarization and migration, which aim to preserve input semantics.Source: §2.1 · p. 3

2.2 Automated Program Repair

  • APR includes generate-and-validate methods and synthesis-based methods, while learned systems also predict bug locations and edits rather than translating whole lines.Examples include DeepFix, VarMisuse repair [31], and Graph2Diff, which generates AST edits [29].Source: §2.2 · pp. 3–4
  • Prior repair systems translated buggy code and limited context into repaired code, but whether this is feasible, mischaracterizes fixes, or suits the NMT objective remained empirically open.The paper asks whether translation expands the search space despite real-world repairs usually changing only a few tokens.Source: §2.2 · p. 3
  • The authors implement a simple edit-based model to compare its distinct loss with token-overlap objectives and examine effects on training outcomes.Edit-based methods learn incorrect-token locations and changes to make, rather than maximizing overlap between the bug and repair.Source: §2.2 · p. 4

3 METHODOLOGY

The study evaluates a representative translation setup for single-line repair using a collected Java bug-fix dataset. It analyzes repair characteristics and generated patches, and trains models to examine context, similarity, and tokenization.

3 METHODOLOGY

  • The study analyzes how well deep-learning models developed for translation apply to software engineering, rather than aiming to produce a state-of-the-art repair tool.Its representative seq2seq approach is used to study limitations of model transplantation.Source: §3 · p. 4

3.1 Scope

  • The repair task assumes the bug is already localized to exactly one line, which the model translates into its repaired counterpart.This is presented as the most direct form of repair as translation.Source: §3.1 · p. 4

3.2 Data

  • The dataset contains about 60,000 candidate bug fixes from 8,644 Java projects, extracted from one-line changes in the 10,235 most-starred GitHub repositories as of March 30, 2020.Commit messages were filtered using a keyword heuristic, and manual checks found the vast majority of inspected samples were bug fixes.Source: §3.2 · p. 4

3.3 Experiment Setup

  • Experiments analyze real fixes and model-generated patches, including how varying amounts of code context affect disambiguation and the information available for repair.Context can include surrounding lines, a function, or a whole file or project.Source: §3.3 · p. 4, §3.3.1 · p. 4

3.3.2 Similarity analysis

  • The researchers compare real bugs and patches using Levenshtein edit distance, averaged 1- through 4-gram Jaccard similarity, and BLEU.BLEU measures prediction-reference n-gram overlap normalized by predicted sequence length; the three metrics assess different aspects of similarity.Source: §3.3.2 · p. 4

3.3.3 Model training and BPE

  • The Transformer is trained on about 55,000 training bugs, with a 90%:5%:5% train/validation/test split across organizations.The model is optimized on held-out validation performance before evaluation on the test set.Source: §3.3.3 · p. 4
  • Byte-pair encoding splits rare or long words into frequent sub-tokens, and the study applies it to model inputs and predictions.This addresses unfamiliar tokens and variable or method names formed from combinations of words.Source: §3.3.3 · pp. 4–5

4 ANALYSIS

The analysis examines real-world bug-fix characteristics and patches generated by the trained NMT model. It assesses translation adequacy, architectural concerns, and their impact on performance.

4 ANALYSIS

  • The analysis compares characteristics of real-world bugs and patches with those of patches generated by the trained NMT model.Source: §4 · p. 5
  • It scrutinizes whether translation is an adequate paradigm for repair and identifies architectural concerns.Source: §4 · p. 5
  • It quantifies the impact of the identified architectural concerns on model performance.Source: §4 · p. 5

4.1 Task Design

Program repair differs from translation because fixes may require information absent from buggy lines, and similar bugs do not reliably yield similar patches. The section examines how missing context and novel vocabulary challenge translation-based repair.

4.1 Task Design

  • Translation preserves meaning, but repair must change buggy behavior, and a learner may lack the developer’s reasoning behind the fix.A repair may introduce a new API call or other tokens absent from the buggy lines.Source: §4.1 · p. 5

4.1.1 Program repair needs (lots of) contextual information.

  • Nearly 90% of patches introduce vocabulary absent from the buggy source, and immediate context rarely supplies the unseen patch tokens.Some tokens occur elsewhere in the file, but often far from the bug; the table compares new-vocabulary ratios across context windows.Source: §4.1.1 · p. 5, Table 1 · p. 5

4.1.2 Without context, program repair is inherently ambiguous.

  • Similar bugs only partially correlate with similar patches; Figure 1 shows a smeared pattern rather than a pronounced diagonal.The authors report this pattern using both Jaccard and BLEU similarity scores.Source: §4.1.2 · p. 6, Figure 1 · p. 6
  • Even highly similar bugs often have dissimilar patches: patches score below 0.5 similarity in more than half of the cases.The authors conclude that transferring patch information from similar bugs is almost entirely ineffective.Source: §4.1.2 · p. 6, Table 2 · p. 6
  • A buggy line alone leaves repair inherently ambiguous because valid fixes can depend on project requirements and developer preferences not encoded in the code.Different candidate repairs can have distinct semantics.Source: §4.1.2 · p. 6

4.1.3 The challenges of new vocabulary.

  • 5.6% is the model’s prediction rate for required new vocabulary, and even its top-25 sampled patches anticipate only 14% of that vocabulary.The authors conclude that the lack of information in training data, vocabulary, and immediate context makes repair as translation largely infeasible.Source: §4.1.3 · pp. 6–7

4.2 Architectural Design

The section argues that autoregressive translation architectures are a poor fit for repairs that usually preserve most buggy-line tokens and make limited syntactic changes. These data motivate focusing repair models on the tokens that need modification rather than generating the entire line left to right.

4.2 Architectural Design

  • Autoregressive translation requires generating the entire repair, and this architectural mismatch may hinder training and test convergence without making repair infeasible.Source: §4.2 · p. 7

4.2.1 The patch preserves most of the tokens in the bug.

  • Median edit distance was 1; 51.1% of repairs changed one token and 63.6% changed at most two, while average edit distance was 3.29.Source: §4.2.1 · p. 7, Figure 2 · p. 8
  • The similarity findings apply to small, one-line bug fixes and may not hold for larger patches that reconstruct a buggy module.Source: §4.2.1 · p. 7
  • Because repairs retain most buggy-line tokens, the section proposes predicting which tokens to preserve and searching only for required modifications.Source: §4.2.1 · p. 7

4.2.2 The patch tends to make minor changes to the bug’s syntax.

  • Slightly more than half of patches preserve their bugs’ exact syntactic structure; among patches introducing new tokens, 56.2% do not change syntax.Source: §4.2.2 · p. 7

4.2.1 The patch preserves most of the tokens in the bug.

  • Average Jaccard similarity was 0.76 and average BLEU was 0.61, with 88.4% at Jaccard ≥0.6 and 72.5% at BLEU ≥0.5.Source: Figure 2 · p. 8

4.3 Program Repair via NMT (Objective)

The study tests whether NMT’s autoregressive objective suits program repair by examining token accuracy against complete-patch accuracy and whether beam search explores diverse repairs. It finds that token-level performance can mislead, beam search adds few alternatives, and NMT objectives are inappropriate for repair.

4.3 Program Repair via NMT (Objective)

  • The empirical test asks whether teacher-forced token accuracy tracks complete-patch quality and whether beam search efficiently explores repair alternatives.Source: §4.3 · p. 8
  • Teacher-forced token accuracy rises from about 83% to 85% after epoch 10, while complete-repair accuracy is only 4.5% after 10 epochs.The model initially minimizes loss by copying, then struggles to predict the required changes.Source: Figure 3 · p. 8, §4.3 · p. 9
  • Per-token and complete-sequence accuracy correlate strongly overall (ρ = 0.914), but less after epoch 10 (ρ = 0.863), making token accuracy misleading.The remaining 2% token-accuracy increase corresponds to about 7% complete-repair accuracy.Source: §4.3 · p. 9
  • Top-5 accuracy exceeds top-1 by at most about 5 percentage points, and the gap shrinks as top-1 improves, indicating few novel alternatives.Generated candidates are often similar and differ by only a few tokens.Source: §4.3 · p. 9
  • NMT objectives are inappropriate for program repair, reducing training efficacy on more appropriate metrics.Source: §4.3 · p. 9

5 SEQ2SEQ MODEL FOR PROGRAM REPAIR

The model redesign replaces whole-line generation with edit prediction and adds bug-file context. On the test set, the edit-based enhancement fixed 22 additional bugs with its top prediction, while the contextual enhancement did not improve performance.

5.1 Model Changes

  • The edit model represents a patch as a token-level diff, using insertion and deletion pointers and generating only newly required tokens.It identifies the longest shared prefix and suffix between bug and patch, then points to the segment to delete and emits tokens to insert.Source: §5.1 · p. 9

5.2 Results

  • The edit-based enhancement fixed 22 additional bugs on the test set with its top prediction, while the contextual enhancement did not seem to help.Table 4 reports repair accuracy on de-duplicated test data; the supplied passages do not give its cell values.Source: Table 4 · p. 10, §5.2 · p. 10
  • The edit model’s teacher-forced per-token accuracy increased more smoothly and tracked complete-repair prediction more closely than the baseline’s behavior.Figure 5 compares teacher-forced per-token accuracy, complete-sequence accuracy, and top-5 beam-search accuracy during training.Source: §5.2 · p. 10, Figure 5 · p. 10
  • The edit model predicted newly introduced patch vocabulary in 6.8% of top-1 samples and 15.7% of top-25 samples, versus 5.6% and 14% for the baseline.The edit model was stronger at top-5 sampling but weaker than the baseline at top-25; the authors attribute this to committing to insertion and deletion pointers first, which bounded subsequent sampling.Source: §5.2 · p. 10
  • Although the authors tried beam search over 25 likely start-and-end pointer combinations, the model tended to choose one pair with very high probability, limiting the set effectively explored.The authors identify this as a possible topic for future work.Source: §5.2 · p. 10
  • The contextual enhancement adds up to 500 tokens from the bug’s file and biases decoder attention toward buggy tokens, but its performance was slightly worse than the context-free model.The authors suggest that modeling large amounts of context may be challenging; the context models trained more slowly and generalized worse despite similar training accuracy.Source: §5.1 · p. 9, §5.2 · p. 10

6 THREATS TO VALIDITY

The study focuses on small, one-line fixes and does not claim that their empirical properties generalize to larger, more complex defects. It also does not compare its models with state-of-the-art NMT repair tools, because its goal is to evaluate the feasibility of the general approach.

6 THREATS TO VALIDITY

  • The findings are not claimed to generalize from small, one-line fixes to larger, more complex defects.The authors identify generalization to other defect types and model transplants into software engineering as external threats to validity.Source: §6 · p. 11
  • The study does not compare its models with state-of-the-art NMT-based repair tools.Its aim is to evaluate the feasibility of patching as translation using a representative setup, not to present the best-performing models.Source: §6 · p. 11

7 CONCLUSION

The paper empirically evaluates patching as translation and finds that translation-based modeling does not capture bug fixing well. Adapting sequence-to-sequence models to generate edits rather than raw tokens yields promising improvements.

7 CONCLUSION

  • The study finds that the translation paradigm does not capture bug fixing well for a range of reasons.It evaluates software patching as language translation as an example of transplanting models from neighboring fields into software engineering.Source: §7 · p. 11
  • Adapting sequence-to-sequence models to generate edits rather than raw tokens leads to promising improvements.Source: §7 · p. 11
  • The authors hope the study will inspire more empirically grounded research into model transplants for program repair and other software engineering applications.Source: §7 · p. 11

Related papers

Loading 2008.10707v2…