PaperlayerView exact revision on arXiv

Source-linked AI summary

Fix Bugs with Transformer through a Neural-Symbolic Edit Grammar

Yaojie Hu, Xingjian Shi, Qiang Zhou, Lee Pike

arXiv:2204.06643v1cs.LGcs.AIcs.PLcs.SE

TL;DR

Code repair is difficult because bugs require precise syntactic and semantic understanding, often hinging on small code changes. NSEdit predicts executable edit sequences from buggy code using a Transformer, an edit grammar, location pointers, and rerankers. It reaches 24.04% accuracy on the Tufano CodeXGLUE dataset and reports robust performance across package variation.

  • Problem

    Code repair remains challenging because a model must understand buggy programs syntactically and semantically, and fixes can depend on small, precise changes.

    Source: §1 · p. 1

  • Method

    NSEdit uses a Transformer to predict executable editing sequences from buggy code alone, guided by a regular-language edit grammar and pointer-based location selection.

    Source: §1 · p. 1

  • Results

    NSEdit achieved 24.04% on the Tufano CodeXGLUE code-repair dataset, a reported state-of-the-art result, and edit prediction doubled top-1 accuracy over fixed-code prediction for concrete code with separate packages (9.04% vs. 3.98%).

    Source: §4.1 · p. 7, §4.3 · p. 8

  • Takeaways & Limitations

    The paper presents an edit grammar as a neural-symbolic interface through which a Transformer generates executable scripts for code changes.

    Source: §1 · p. 2, §5 · p. 10

    NSEdit is formulated to use buggy code without auxiliary program-analysis information.

    Source: §1 · p. 2

Abstract

from arXiv · show

We introduce NSEdit (neural-symbolic edit), a novel Transformer-based code repair method. Given only the source code that contains bugs, NSEdit predicts an editing sequence that can fix the bugs. The edit grammar is formulated as a regular language, and the Transformer uses it as a neural-symbolic scripting interface to generate editing programs. We modify the Transformer and add a pointer network to select the edit locations. An ensemble of rerankers are trained to re-rank the editing sequences generated by beam search. We fine-tune the rerankers on the validation set to reduce over-fitting. NSEdit is evaluated on various code repair datasets and achieved a new state-of-the-art accuracy ($24.04\%$) on the Tufano small dataset of the CodeXGLUE benchmark. NSEdit performs robustly when programs vary from packages to packages and when buggy programs are concrete. We conduct detailed analysis on our methods and demonstrate the effectiveness of each component.

1 INTRODUCTION

Code repair remains difficult because bugs require precise syntactic and semantic understanding, while small differences can separate buggy code from its fix. NSEdit predicts executable edit sequences from buggy code alone, using an edit grammar, location pointers, and reranking.

1 INTRODUCTION

  • The regular-language edit grammar lets a Transformer generate executable edit scripts, with delete and insert actions and a two-mode decoder for words/actions and edit locations.A pointer network selects locations by comparing decoder output with encoder memory.Source: §1 · pp. 1–2, Figure 1 · p. 3
  • NSEdit uses only buggy code, avoiding reliance on program-analysis graphs that may be unavailable when bugs cause syntax errors.The authors also note that program-analysis tools can impose language-dependent restrictions.Source: §1 · p. 2
  • NSEdit achieved state-of-the-art performance on CodeXGLUE, and edit prediction performed better than fixed-code prediction even across package variation.The method uses only buggy code, without auxiliary program-analysis representations.Source: §1 · p. 2
  • The introduction positions NSEdit as a simple edit representation whose grammar and finite-state machine constrain generated scripts to follow the edit language.The paper presents this as a neural-symbolic API for executable code edits.Source: §1 · p. 2
  • An ensemble of rerankers reorders beam-search edit sequences, improves exact-match accuracy, and can trade recall for higher precision.Fine-tuning rerankers on the validation set is reported to reduce overfitting.Source: §1 · p. 3

2 RELATED WORK

The related work covers code-repair systems that use fix patterns, pointer networks, graph transformations, and program-analysis information. It also situates Transformer among models used beyond language processing.

2 RELATED WORK

  • Code-repair approaches include clustering fix patterns, pointer-based bug localization, graph-edit prediction, and backtranslation tuned with test-suite analysis.The cited examples include Getafix, Vasic et al., Dinella et al., and DeepDebug.Source: §2 · p. 3
  • Transformer and derivatives such as BERT and GPT are described as prominent in machine translation and deep NLP.The passage also notes scaling with model size.Source: §2 · p. 3
  • Transformer has also been adapted to image classification and protein-structure modeling, illustrating use beyond natural-language tasks.Source: §2 · p. 3

3 METHODS

NSEdit maps buggy code to edit scripts using a finite-state grammar, a location pointer, beam search, and reranking; it achieves state-of-the-art top-1 accuracy on the Tufano small abstract dataset.

3 METHODS

  • The task is to predict edits that transform buggy source code into its fixed version.Training tokenizes buggy-fixed pairs, fine-tunes pretrained CodeBERT and CodeGPT with teacher forcing, then generates top-5 hypotheses for reranking.Source: §3 · pp. 3–4

3.1 TOKENIZATION AND EDITING SEQUENCE GENERATION

  • Tokenized code diffs yield delete(i, j) and insert(i, s) actions, with a finite-state grammar organizing action, word, and location tokens.The grammar’s state machine starts at BOS and accepts at EOS.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 TRAINING TRANSFORMER TO PREDICT BUG FIX EDITING SEQUENCES WITH TEACHER FORCING

  • CodeBERT and CodeGPT support teacher-forced decoding of edit actions and words or pointer-selected locations; location inputs reuse encoder-memory embeddings, and both modes share cross-entropy loss.The pointer scores edit locations against encoder memory, producing a probability distribution over positions.Source: §3.2 · pp. 4–5

3.3 GENERATING BEAM SEARCH HYPOTHESES DURING INFERENCE

  • Beam search generates five hypotheses, while the finite-state machine determines the next token type and masks invalid tokens to enforce grammar syntax.During inference, predicted edit locations select the corresponding encoder-memory embeddings.Source: §3.3 · p. 5

3.4 RERANKING THE BEAM SEARCH HYPOTHESES

  • Because correct edits can appear below rank one, two rerankers score beam hypotheses and combine with beam probability to select final edits.The rerankers use different architectures; validation-set fine-tuning for one epoch mitigates overfitting.Source: §3.4 · pp. 5–6
  • NSEdit achieves state-of-the-art top-1 exact-match accuracy on the Tufano small abstract dataset in CodeXGLUE.The dataset normalizes variable, method, and type names.Source: Table 1 · p. 6

4 EXPERIMENTS AND RESULTS

NSEdit achieved state-of-the-art accuracy on the CodeXGLUE Tufano dataset, and experiments compare edit prediction with fixed-code prediction across datasets, package settings, backbones, and reranking configurations. Reranking also supports a more efficient precision–recall trade-off than beam-search confidence scores.

4.1 NSEDIT ACHIEVES SOTA PERFORMANCE ON CODEXGLUE CODE REPAIR BENCHMARK

  • 24.04% accuracy gave NSEdit state-of-the-art performance on the CodeXGLUE Tufano code repair benchmark.NSEdit was the only compared method predicting edits rather than fixed programs.Source: §4.1 · p. 7

4.2 PREDICTING EDITING SEQUENCES PERFORMS BETTER THAN PREDICTING FIXED CODE

  • On Tufano abstract data, predicting editing sequences performed better than predicting fixed code.The comparison used Baseline models without CodeGPT or rerankers to isolate the translation target.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 NSEDIT PERFORMS ROBUSTLY AGAINST PACKAGE-TO-PACKAGE VARIATIONS

  • With packages separated and concrete code, edit prediction reached 9.04% test top-1 accuracy versus 3.98% for fixed-code prediction.The authors report that edit prediction overfits less when packages are separate or input programs retain original names; mixed-package results show larger validation-to-test gaps.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 NSEDIT IS A GENERAL BUG FIX METHOD IN DIFFERENT LANGUAGES AND SETTINGS

  • NSEdit was evaluated on three ETH Py150 code repair tasks, supporting its use across languages and settings.The tasks were variable-misuse classification, wrong binary operator, and swapped operand.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 PRE-TRAINED ENCODER AND DECODER CAN BE FINE-TUNED TOGETHER ON EDITING SEQUENCES

  • Fine-tuning CodeBERT and CodeGPT backbones improved NSEdit performance on Tufano data, including when the decoder was not pretrained on editing sequences.The Table 5 comparison excluded reranking and fine-tuned the backbones together.Source: §4.5 · pp. 8–9, Table 5 · p. 9

4.6 RERANKERS BRING SIGNIFICANT PERFORMANCE IMPROVEMENTS

  • The fine-tuned reranker ensemble achieved the best accuracy among the tested settings, while ensemble use and validation-set fine-tuning each improved performance.A Transformer reranker alone had lower accuracy than using no reranker.Source: §4.6 · p. 9, Table 6 · p. 9

4.7 ENSEMBLE RERANKER EFFICIENTLY TRADES OFF PRECISION AND RECALL

  • At 50% precision, reranking retained 38.4% recall versus 22.4% with beam-search confidence scores.The reranking score provided higher recall at every precision level in the Tufano small abstract dataset comparison.Source: §4.7 · pp. 9–10, Figure 3 · p. 10

5 DISCUSSIONS

NSEdit achieves state-of-the-art performance on CodeXGLUE code repair, and the paper presents its edit grammar as an example of a neural-symbolic interface built from a DSL.

5 DISCUSSIONS

  • NSEdit achieves state-of-the-art performance on the CodeXGLUE code repair task.The paper also concludes that predicting editing sequences is more effective than predicting fixed code.Source: §5 · p. 10
  • The edit grammar illustrates a broader approach: train a Transformer to use a domain-specific language as a programmable interface to traditional symbolic systems.Source: §5 · p. 10

A BEAM SEARCH FORMAL DEFINITION

Beam search assigns each generated editing sequence a probability from its token probabilities, then expands and retains top-scoring candidate beams until termination.

A BEAM SEARCH FORMAL DEFINITION

  • A beam's sequence probability is the product of the probabilities of its generated tokens.The passage also describes taking the logarithm of this product.Source: A BEAM SEARCH FORMAL DEFINITION · p. 14
  • At each step, beam search selects the top-K tokens using the log probability score to form the next set of beams.Source: A BEAM SEARCH FORMAL DEFINITION · p. 14
  • Search continues until an EOS token is predicted or the maximum length is reached; multiple new beams may extend one prior beam.A previous beam is not guaranteed to remain as a subsequence in the next beam set.Source: A BEAM SEARCH FORMAL DEFINITION · p. 14

B RERANKER ABLATION EXPERIMENTS DETAILS

The ablations compare beam-search scoring with Transformer and ensemble rerankers, including versions fine-tuned on the validation set.

B RERANKER ABLATION EXPERIMENTS DETAILS

C TRAINING AND MODEL HYPERPARAMETERS

The appendix specifies the model's implementation, training schedule, beam-search constraint handling, reranker training, and ensemble hyperparameter search.

C TRAINING AND MODEL HYPERPARAMETERS

  • The implementation uses PyTorch, with pretrained CodeBERT and CodeGPT loaded from HuggingFace, and AdamW with a triangular learning-rate schedule.The learning rate is 1e-4 times the GPU count; for CodeGPT, pretrained parameters use half that rate and randomly initialized parameters use four times that rate.Source: C TRAINING AND MODEL HYPERPARAMETERS · p. 14
  • NSEdit trains for at most 60 epochs with early stopping when accuracy does not improve; training with rerankers on Tufano datasets takes about a day on four V100 GPUs.Automatic mixed precision is enabled.Source: C TRAINING AND MODEL HYPERPARAMETERS · p. 15
  • Beam search uses a length-penalized score and assigns zero probability to tokens invalid under the edit grammar's finite-state machine.Source: C TRAINING AND MODEL HYPERPARAMETERS · p. 15
  • Rerankers train for 12 epochs on buggy programs whose beam search includes at least one correct editing sequence, followed by one validation-tuning epoch.Validation tuning uses one GPU.Source: C TRAINING AND MODEL HYPERPARAMETERS · p. 15
  • The ensemble reranker is a linear model, with coefficients selected by grid search over logarithmic and then narrower linear ranges.The coefficient search uses [0.01, 100] and then [0.1, 2].Source: C TRAINING AND MODEL HYPERPARAMETERS · p. 15, Table 7 · p. 15

D TUFANO DATASET EDITING SEQUENCE STATISTICS

Tufano medium ground-truth edits are longer and more numerous than those in Tufano small by both mean and median. The authors suggest these sequences are harder to predict correctly.

D TUFANO DATASET EDITING SEQUENCE STATISTICS

E EXAMPLE BUG FIXES

The paper presents correctly produced NSEdit fixes spanning coding-style improvements, logging removal, missing annotations, logical errors, operator changes, and spelling errors.

E EXAMPLE BUG FIXES

Loading 2204.06643v1…