PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Bug-Fixes Using Pretrained Transformers

Dawn Drain, Chen Wu, Alexey Svyatkovskiy, Neel Sundaresan

arXiv:2104.07896v2cs.CLcs.PL

TL;DR

Exact Java bug repair asks a model to reproduce the developer’s fix from a buggy method. DeepDebug uses code pretraining followed by finetuning for this task, and its best model generates 75% more non-deletion fixes than the previous state of the art.

  • Problem

    Patch generation from a buggy Java method is evaluated against the exact token sequence of the developer’s fix.

    Source: §2 · p. 2

  • Method

    DeepDebug pretrains a sequence-to-sequence transformer on Java code, optionally warmstarted from an English checkpoint, then finetunes it to generate fixes from buggy methods.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    75% more non-deletion fixes than the previous state of the art are generated by DeepDebug’s best model.

    Source: §10 · p. 7

  • Takeaways & Limitations

    Code abstraction benefits smaller GRU models, but not larger transformers, which can use meaningful function and variable names.

    Source: §9 · p. 6, §10 · p. 7

    Bug fixing is often underspecified from the method alone, and the approach excludes test cases, example usages, and the class constructor.

    Source: §8 · p. 5

Abstract

from arXiv · show

Detecting and fixing bugs are two of the most important yet frustrating parts of the software development cycle. Existing bug detection tools are based mainly on static analyzers, which rely on mathematical logic and symbolic reasoning about the program execution to detect common types of bugs. Fixing bugs is typically left out to the developer. In this work we introduce DeepDebug: a data-driven program repair approach which learns to detect and fix bugs in Java methods mined from real-world GitHub repositories. We frame bug-patching as a sequence-to-sequence learning task consisting of two steps: (i) denoising pretraining, and (ii) supervised finetuning on the target translation task. We show that pretraining on source code programs improves the number of patches found by 33% as compared to supervised training from scratch, while domain-adaptive pretraining from natural language to code further improves the accuracy by another 32%. We refine the standard accuracy evaluation metric into non-deletion and deletion-only fixes, and show that our best model generates 75% more non-deletion fixes than the previous state of the art. In contrast to prior work, we attain our best results when generating raw code, as opposed to working with abstracted code that tends to only benefit smaller capacity models. Finally, we observe a subtle improvement from adding syntax embeddings along with the standard positional embeddings, as well as with adding an auxiliary task to predict each token's syntactic class. Despite focusing on Java, our approach is language agnostic, requiring only a general-purpose parser such as tree-sitter.

1 Introduction

Prior repair work used test-suite-guided patch generation, copying, or synthetic bugs, while DeepDebug applies pretrained sequence-to-sequence transformers to real-world Java repair. Its results highlight gains from pretraining and more constructive fixes, with raw code outperforming abstraction for its best models.

1 Introduction

  • Earlier automated patching generated candidates until one passed a test suite, while later work sought the developer’s exact fix on mined Java methods.Copying-based approaches on this dataset report large gains, and SequenceR’s added context gives a 15% relative boost.Source: §1 · p. 1
  • Prior synthetic-bug studies report high accuracy on synthetic examples, including over 90% for a pretrained BERT-style model finetuned on synthetic bugs.The cited RNN study reports 86% accuracy on synthetic bugs and 41% on real-life bugs.Source: §1 · pp. 1–2
  • Java-only pretraining raises patches found from 1049/12380 to 1392/12380, and a BART warmstart followed by Java pretraining raises them to 1839/12380.The pretraining corpus comprises 67k Java repositories; the target benchmark is Patches in the Wild.Source: §1 · p. 2
  • 75% more non-deletion fixes than the previous state of the art are generated by DeepDebug’s best model.The paper distinguishes non-deletion fixes from deletion-only fixes when evaluating patches.Source: §1 · p. 2
  • DeepDebug reports superior results on concrete code, especially after pretraining, while abstraction tends to benefit smaller models.The best results are reported for raw code rather than abstracted code.Source: §1 · p. 2
  • DeepDebug also explores syntax embeddings and an auxiliary task that predicts each token’s syntactic class.The paper describes the resulting improvement as subtle.Source: §1 · p. 2

2 Problem overview

The repair task maps a buggy Java method to the exact token sequence of the developer’s fix. During finetuning, the sequence-to-sequence model learns to generate that fix given the buggy method.

2 Problem overview

  • The task is to produce the exact developer-authored token sequence for a buggy Java method mined from a commit.The task is framed as patch generation rather than merely producing any plausible repair.Source: §2 · p. 2
  • The illustrated repair must capture the value returned by the getter method.Source: Figure 1 · p. 2
  • Finetuning minimizes the expected log-probability of generating the correct function given the buggy function.The passage defines sets of buggy and correct functions and model parameters for the objective.Source: §2 · p. 2

3 Data

The Java benchmark mines short methods from commits identified as bug-fixing, then divides them into small and medium method datasets. It includes both concrete and abstracted code, with concrete code yielding superior results, especially after pretraining.

3 Data

  • The dataset extracts modified Java methods of at most 100 tokens from commits whose messages contain bug-fixing terms, splitting them into under-50-token and up-to-100-token sets.The source repositories were public, non-fork GitHub repositories mined through 2016.Source: §3 · pp. 2–3
  • One illustrated fix normalizes minTime to 0 and maxTime to 1 by manipulating a return value, correcting reversed boundary values.Source: Figure 2 · p. 3
  • Abstracted code replaces many tokens with type-specific placeholders while preserving selected common tokens and frequent idioms.Comments, irrelevant whitespace, and indentation are also stripped during normalization.Source: §3 · p. 3
  • Concrete code produces superior results to abstracted code with idioms, especially after pretraining.Source: §3 · p. 3

4 Baseline

Earlier baselines report exact-match accuracy for short Java methods, while the paper cautions that commit-message-mined bug fixes are diverse and noisy. It speculates that reported performance may approach the practical top-1 ceiling for this task.

4 Baseline

  • Copy That! achieved 17.7% accuracy on small methods (<50 tokens) and 8.0% on medium methods (50–100 tokens), versus Seq2Seq’s 9.2% exact accuracy and 3.2% accuracy, respectively.Source: §4 · p. 3
  • The paper speculates that 17.7% and 8.0% may be near the best practical top-1 accuracy on this task.Source: §4 · p. 3
  • Commit-message-mined bug-fixing commits are diverse and noisy, unlike methods known to contain bugs, especially synthetic bugs from a small class.Source: §4 · p. 3

5 Model

DeepDebug uses a standard transformer with code-specific tokenization and syntax information, while applying copy attention selectively. Its experiments compare concrete and abstracted code across model capacities.

5 Model

  • Copy attention is used for 60-million-parameter models trained from scratch and for state-of-the-art replication, but omitted from full-scale pretraining experiments due primarily to technical limitations.The architecture is the standard transformer.Source: §5 · p. 3
  • A roughly 50k-token BPE vocabulary for concrete code incorporates whitespace tokens, increasing pretraining context length and processing speed by around 40%.The abstracted-code vocabulary contains all 433 unique tokens.Source: §5 · p. 3
  • Syntax embeddings supplement positional and vocabulary embeddings, helping distinguish keywords and easing the model’s parsing of formal grammar.The paper describes this as a way to encompass both raw-code and abstracted-code approaches.Source: §5 · pp. 3–4
  • Smaller models perform best on abstracted code, while higher-capacity models better use semantics associated with concrete code.The paper also evaluates raw code, fully abstracted code, and code abstracted with idioms.Source: Figure 3 · p. 4

6 Pretraining

Pretraining uses span masking on filtered Java repositories from GitHub. The paper reports the corpus scale and the compute used for pretraining.

6 Pretraining

  • Span masking replaces consecutive three-token spans with a single mask and trains the model to reproduce the masked tokens in order.The method masks 30% of tokens, which increases effective context length during pretraining by about 50%.Source: §6 · p. 4
  • The filtered pretraining corpus contains 67,000 repositories, 8 million files, 1.3 billion lines of code, 13 billion tokens, or 54GB.Repositories have at least 10 stars and were updated within the last five years; files are deduplicated and filtered and cleaned.Source: §6 · p. 4
  • Pretraining runs for ten epochs over one week on a DGX-2 with sixteen 32GB V100 GPUs, estimated at about 4% of the resources used for the public English BART checkpoint.The comparison is with BART’s 40 epochs on 160GB using a more compute-intensive objective.Source: §6 · p. 4

7 Experiments

The experiments compare code representations and pretraining strategies, then test syntax-related additions and small-method copying. They also illustrate the span-masking setup used for pretraining.

7 Experiments

  • A 60-million-parameter transformer with copy mechanism is used to compare concrete code with abstracted code containing the 433 most common tokens as idioms.The experiment is trained from scratch.Source: §7 · p. 4
  • Three pretraining strategies are considered: Java-only pretraining, direct finetuning from English-pretrained BART, and Java domain-adaptive pretraining warmstarted from BART.The authors hypothesize that English knowledge may help with programming concepts and self-documenting names.Source: §7 · p. 4
  • An auxiliary token-type labeling loss is tested alongside next-token prediction, using labels including METHOD, VARIABLE, STRING_LIT, and NUM_LIT.The paper motivates syntax highlighting as useful during early language learning, before formal grammar is internalized.Source: §7 · pp. 4–5
  • Adding token-type embeddings gives a small improvement when training from scratch, but no improvement when added only during finetuning.The embeddings accompany positional and vocabulary embeddings during encoding.Source: Figure 4 · p. 5, §7 · p. 5
  • Figure 5 illustrates span-masking input and output, with each mask covering exactly three tokens.Source: §7 · p. 5, Figure 5
  • The paper partially replicates and extends Copy That! by evaluating small methods on concrete and fully abstracted code without idioms using its specialized span-copying GRU model.Source: §7 · p. 5

8 Results

Many generated fixes are deletions or fall into a few simple patterns. The paper also identifies missing context and underspecified intended behavior as barriers to fixing some methods.

8 Results

  • Many fixes follow a few uninspiring patterns, including simple deletions, swapping “protected”, “private”, and “public”, or inserting “native”.Source: §8 · p. 5
  • Roughly two-thirds of first-attempt fixes are deletions; among non-deletion fixes, specified simple patterns account for 7% in the strongest models and up to 50% in the weakest.The patterns include access-modifier swaps and inserting “native”; results therefore report deletion and non-deletion counts separately.Source: §8 · p. 5
  • The authors report that missed fixes are often impossible to identify without more context.In practice, correct behavior may be underspecified because tests, example usages, and the class constructor are excluded.Source: §8 · p. 5
  • With only a buggy method, a pop() off-by-one error can leave it ambiguous whether the head index should be bounded below by zero or negative one.Source: §8 · p. 5

9 Analysis

The analysis finds that pretraining particularly helps medium-sized methods, while the best results use English pretraining followed by Java adaptation. Code representation and syntax-related training choices affect model sizes and training settings differently.

9 Analysis

  • Medium-method fixes increase by a factor of three when adding pretraining.The analysis also reports that medium methods are harder to fix than short methods.Source: §9 · pp. 5–6
  • The model uses the function name “backward” to infer that the operation should decrement rather than increment.The passage presents this as an example of the model drawing on meaningful function names.Source: Figure 7 · p. 6, §9 · p. 6
  • Smaller GRUs and transformers trained on abstract code produce seven times as many deletions as constructive fixes; smaller GRUs benefit from abstraction, unlike larger transformers.The analysis also reports that transformers benefit from concrete code, unlike smaller GRU-based models.Source: §9 · p. 6, Table 1 · p. 6, Table 3 · p. 7
  • English pretraining boosts repair performance, and the best results come from further pretraining the English model on Java.The English-only BART checkpoint outperforms the Java-only model, although the latter was trained for only 4% as long.Source: §9 · pp. 6–7
  • Adding an auxiliary token-type labeling task helps training from scratch but makes no difference when fine-tuning the pretrained model.The task predicts whether each subword belongs to a TYPE, METHOD, VARIABLE, STRING_LIT, NUM_LIT, or other category.Source: Figure 6 · p. 6, §9 · p. 7
  • Scaling from 60 million to 400 million parameters yields underwhelming returns when training from scratch, which the authors hypothesize reflects limited training data.Each training set contains around 50,000 examples and tens of megabytes of data.Source: §9 · p. 7
  • Syntax-type embeddings provide a small benefit during training from scratch but none during fine-tuning; the authors hypothesize that they may clash with pretrained representations.The reported benefit applies to adding embeddings during encoding.Source: §9 · p. 7

10 Conclusions and Future Work

DeepDebug uses a multistage pretraining pipeline for bug repair, warmstarting from an English checkpoint, further pretraining on Java, and fine-tuning on the target task. Its best model produces more non-deletion fixes than the prior state of the art; future work includes additional languages and iterative fixes.

10 Conclusions and Future Work

  • DeepDebug warmstarts from a public English checkpoint, further pretrains on code, and fine-tunes on the target repair task.The best workflow first pretrains on English and then on Java; Table 4 notes that English pretraining used twenty-five times as much compute as Java pretraining.Source: §10 · p. 7, Table 4 · p. 8
  • Future work will explore expanding to multiple languages and considering iterative fixes.Source: §10 · p. 7
  • On medium methods, the best result comes from English pretraining followed by Java; on small methods, further Java pretraining gives the best standard top-1 accuracy.The English BART baseline is described as very strong on small methods.Source: Table 4 · p. 8, Table 5 · p. 8
  • Syntax-type labeling and embeddings offer small benefits from scratch, but adding the labeling task during fine-tuning yields no improvement.The table also reports no benefit from adding syntax embeddings during fine-tuning.Source: Table 6 · p. 9
  • The best model generates 75% more non-deletion fixes than the previous state of the art.Table 7 reports that the prior copy-span model produces 57% as many non-deletion fixes as this approach.Source: §10 · p. 7, Table 7 · p. 9

Related papers

Loading 2104.07896v2…