PaperlayerView exact revision on arXiv

Source-linked AI summary

CURE: Code-Aware Neural Machine Translation for Automatic Program Repair

Nan Jiang, Thibaud Lutellier, Lin Tan

arXiv:2103.00073v4cs.SEcs.AIcs.LG

TL;DR

Existing NMT-based APR can omit correct fixes and generate many patches that violate code constraints. CURE combines PL pre-training, code-aware search, and subword tokenization; it fixes 57 Defects4J and 26 QuixBugs bugs, outperforming existing APR tools. Its evaluation uses Java benchmarks and perfect fault localization, limiting the demonstrated scope.

  • Problem

    NMT-based APR often generates uncompilable or compilable-but-incorrect patches, while limited vocabularies can exclude correct fixes.

  • Method

    CURE combines a code-pre-trained PL model, valid-identifier and length-controlled beam search, and BPE subword tokenization for APR.

  • Results

    CURE fixes 57 Defects4J bugs and 26 QuixBugs bugs, outperforming all existing APR tools on both benchmarks.

  • Takeaways & Limitations

    CURE demonstrates that code-aware NMT can combine developer-like code modeling, constrained search, and a smaller search space for automatic program repair.

  • Takeaways & Limitations

    The evaluation covers two Java benchmarks and uses perfect fault localization for comparison with prior APR techniques.

Abstract

from arXiv · show

Automatic program repair (APR) is crucial to improve software reliability. Recently, neural machine translation (NMT) techniques have been used to fix software bugs automatically. While promising, these approaches have two major limitations. Their search space often does not contain the correct fix, and their search strategy ignores software knowledge such as strict code syntax. Due to these limitations, existing NMT-based techniques underperform the best template-based approaches. We propose CURE, a new NMT-based APR technique with three major novelties. First, CURE pre-trains a programming language (PL) model on a large software codebase to learn developer-like source code before the APR task. Second, CURE designs a new code-aware search strategy that finds more correct fixes by focusing on compilable patches and patches that are close in length to the buggy code. Finally, CURE uses a subword tokenization technique to generate a smaller search space that contains more correct fixes. Our evaluation on two widely-used benchmarks shows that CURE correctly fixes 57 Defects4J bugs and 26 QuixBugs bugs, outperforming all existing APR techniques on both benchmarks.

I. INTRODUCTION

NMT-based APR is limited by search spaces that may omit correct fixes and search strategies that ignore code syntax and structure. CURE addresses these limitations with code-aware modeling, tokenization, and beam search.

  • Correct repair requires both including the correct fix in the search space and finding it efficiently.
  • Source code’s strict syntax, semantics, and unbounded identifiers create challenges that distinguish code repair from natural-language translation.
  • Up to 67%–97% of patches from state-of-the-art NMT-based APR models are uncompilable.Generated patches can call methods with wrong parameters, invoke undeclared variables, or contain mismatched parentheses.
  • Programming language models: CURE pre-trains a programming-language model on software code to learn correct syntax and developer-like code before fine-tuning for APR.The PL model predicts next code tokens and is combined with an NMT model for APR.
  • Code-aware search strategy: CURE’s code-aware beam search restricts generation to valid identifiers and penalizes patches whose lengths differ substantially from buggy lines.These strategies target uncompilable candidates and very short or long sequences produced by large-beam search.
  • Subword tokenization: BPE tokenization addresses out-of-vocabulary compound and rare identifiers while reducing the search space and retaining more correct patches.

B. Contributions

CURE combines large-scale PL pre-training, an NMT architecture, code-aware beam search, and subword tokenization. On Defects4J and QuixBugs, it fixes 57 and 26 bugs, respectively, outperforming existing APR tools.

  • CURE pre-trains a PL model on 4.04 million methods from 1,700 open-source projects to capture code syntax and developer-like source code.
  • CURE combines a pre-trained PL model with NMT architectures to learn both code syntax and repair patterns.
  • CURE’s code-aware beam search uses valid-identifier checking and length control to find more correct fixes.
  • CURE applies subword tokenization to address the out-of-vocabulary problem in automatic program repair.
  • 57 Defects4J bugs and 26 QuixBugs bugs are correctly fixed by CURE, the highest totals among evaluated APR tools.The evaluation uses the two widely used benchmarks Defects4J and QuixBugs.

II. BACKGROUND

CURE defines repair outcomes and its component architectures, then organizes repair into training, inference, and validation stages. Its design combines subword tokenization, PL modeling, and code-aware search.

  • Candidate patches are generated patches, plausible patches pass validation, and correct patches match or are semantically equivalent to developers’ patches.
  • Pre-training learns a general task from a very large dataset, after which fine-tuning adapts the model to a specific task with fewer data.
  • CoNuT provides CURE’s NMT architecture through buggy-line and context encoders, a merger, decoder, attention module, and token-generation module.
  • CURE’s three techniques target distinct functions: subword tokenization improves the search space, PL modeling learns developer-like code, and code-aware beam search improves patch ranking.
  • Training: Training extracts and tokenizes methods from open-source projects, then uses them to train a PL model that learns developer-like code with correct syntax.
  • Inference: During inference, CURE tokenizes buggy and context lines, extracts in-scope identifiers, and generates candidate patches with code-aware beam search.
  • Validation: Validation compiles patched projects and executes their test suites before outputting plausible patches for developer examination.

B. Data Extraction

CURE uses open-source Java methods for PL pre-training and APR-specific data for fine-tuning, while subword tokenization reduces vocabulary size and addresses out-of-vocabulary identifiers. This representation can enable correct patches that word-level tokenization misses.

  • CURE trains its PL model on millions of methods from open-source Java projects and fine-tunes it using buggy lines, context, and correct fixes.
  • Enhanced word-level tokenization separates code using spaces, camel letters, underscores, strings, and numbers.
  • 2% of test-set tokens remain out of vocabulary after word-level tokenization, and excluding rare project-specific tokens can hinder bug fixing.
  • BPE reduces vocabulary size and mitigates out-of-vocabulary problems by iteratively merging frequent byte pairs.
  • Subword tokenization splits the OOV identifier “charno” into “char@@” and “no”, enabling CURE to generate the correct patch.
  • CURE forms a smaller search space that contains more correct patches through subword tokenization.

D. Programming Language Model

CURE pre-trains GPT on complete source-code methods, then combines it with CoNuT and fine-tunes the combined APR model on buggy lines, context, and fixes. An auxiliary language-model objective helps preserve code knowledge during APR fine-tuning.

  • D. Programming Language Model: CURE trains a GPT programming-language model on open-source programs to predict next tokens and learn developer-like code.
  • D. Programming Language Model: Pre-training separates programming-language learning from patch learning and supplies syntax knowledge before APR fine-tuning.
  • E. Fine-Tuning for APR with a PL Model: The APR model combines the pre-trained GPT module with CoNuT and is fine-tuned using buggy lines, context, and correct fixes.
  • E. Fine-Tuning for APR with a PL Model: The combined objective adds an auxiliary language-model likelihood to the NMT objective, weighted by λ, to retain pre-training information.
  • E. Fine-Tuning for APR with a PL Model: During inference, CURE encodes the buggy line and entire buggy method separately before merging the encoder outputs for patch generation.
  • E. Fine-Tuning for APR with a PL Model: CURE initializes decoding with the last context token before the buggy line rather than CoCoNuT’s “<START>” token.

F. Ensemble Learning

CURE uses ensemble learning across models with different hyperparameters and architectures to increase the number of bugs fixed.

  • Combining models with different hyperparameters and CoNuT or FConv architectures forms CURE’s ensemble.

G. Code-Aware Beam-Search Strategy and Patch Generation

CURE makes beam search code-aware by filtering invalid identifiers and controlling patch length, helping preserve promising paths to correct fixes. It then reconstructs and ranks generated patch sequences for repair.

  • Search motivation: Beam search retains only the n highest-scoring nodes at each step, but vanilla search uses model log-probability without code information.This can favor out-of-scope variables and produce uncompilable candidate patches.
  • Code-aware search: CURE combines valid-identifier checking and length control to make beam search code-aware.The two strategies incorporate identifier validity and patch-length information into candidate selection.
  • Valid-identifier check: Static analysis extracts valid identifiers, tokenizes them into subwords, and maps prefixes to valid succeeding tokens during decoding.For example, the mapping lets the search complete “max_ending_” with “here” when that identifier is valid.
  • Valid-identifier check: Vanilla beam search can discard the correct-fix subtree when a lower average log-likelihood path loses to higher-scoring alternatives.In the example, the correct “... Math .” path is dropped, whereas invalid “max CaMeL” is penalized and the correct subtree remains searchable.
  • Length control: 75% of bugs in the patch training data have fix-length differences of at most 5 tokens, motivating penalties for patches much shorter or longer than buggy lines.Length control adjusts the log-likelihood of <EOS>; a tolerance threshold of 5 provides flexibility.
  • Length control: Length control prevents very short candidates from consuming beam slots, allowing search to explore deeper paths when the buggy sequence is longer.In the example, the short “{ }” patch receives a large <EOS> penalty and is excluded from the top 1,000 nodes.
  • Length control: CURE focuses on fixes similar in length to buggy lines, while its length-control strategy can be adapted to longer patches by changing penalty weights.The paper suggests separating similar-length and other bugs as a possible decomposition of APR.

H. Patch Validation

After generation, CURE reconstructs token sequences into code statements, inserts ranked patches into buggy files, and validates them through compilation and tests.

  • Patch reconstruction: CURE reconstructs subwords into words and replaces abstracted numbers and strings using donor code from the buggy file.The resulting token sequences become code statements before patch insertion.
  • Patch validation: CURE ranks reconstructed statements by average token log-probability, inserts them at buggy lines, and compiles every patched project.Compilation filters out uncompilable patches before test-based validation.
  • Patch validation: A patch is accepted when it preserves passing tests and passes at least one test that failed on the buggy project.These are the validation criteria used in previous work.

IV. EXPERIMENTAL SETUP

The evaluation uses temporally realistic training and validation data, two established APR benchmarks, and comparisons against existing tools under explicit plausibility and correctness criteria.

  • Data and protocol: CURE avoids future-data leakage by using commits made before the first benchmark bug in 2006 for pre-training, fine-tuning, and validation.This design makes the evaluation realistic with respect to available historical data.
  • Data and protocol: The programming-language training set contains 4.04 million methods extracted from 1,700 open-source Java projects rolled back before 2006.Methods longer than 1,024 tokens and abstract methods were excluded; 30,000 instances were used for validation.
  • Data and protocol: Patch training data comes from 45,180 Java projects and excludes instances whose context or fixes exceed 1,024 tokens after subword tokenization.The source data is CoCoNuT’s shared Java patch-training corpus.
  • Model configuration: CURE uses BPE with a target vocabulary of 50,000 and trains a GPT model with eight transformer layers and six attention heads.The GPT model uses embedding size 384 and is trained for five epochs.
  • Model configuration: Hyperparameters are tuned with random search, training 100 APR models on a smaller patch-data subset before retaining the top five models for each GPT-based combination.The search covers convolution dimensions, kernel sizes, convolutional layers, dropout, and λ=0.3.
  • Evaluation benchmarks: Evaluation uses Defects4J v1.4.0 and QuixBugs, excluding duplicate Closure 63 and Closure 93 bugs.Plausible patches pass relevant tests, while two co-authors independently assess correctness as identical or semantically equivalent to developer patches.
  • Baselines: CURE is compared with 25 APR techniques, including pattern-based and NMT-based approaches, none of which combines BPE, a pre-trained PL model, and code-aware search.Table I reports correctly fixed and plausible bugs as x/y.

A. RQ1: How does CURE perform against state-of-the-art APR techniques?

CURE fixes more bugs than existing APR tools on both benchmarks, with gains over both NMT-based and pattern-based systems. The results also connect its unique fixes and compilable-patch behavior to its model and search components, within a single-hunk scope.

  • Overall performance: 57 Defects4J bugs and 26 QuixBugs bugs are correctly fixed by CURE, the highest totals reported for both benchmarks.CURE outperforms the best existing approach on each benchmark.
  • Overall performance: CURE fixes five more Defects4J bugs than TBar and 13 more than the best NMT-based approach, CoCoNuT.It also fixes twice as many QuixBugs bugs as CoCoNuT and fixes 12 bugs no evaluated QuixBugs tool fixes.
  • Unique fixes: Chart 17 is a Defects4J bug that only CURE fixes among the 25 compared approaches.Its correct patch ensures the second parameter is non-negative, a transformation captured by the PL model despite being rare in patch-training data.
  • Unique fixes: CURE fixes the QuixBugs KTH bug because its code-aware search avoids the uncompilable-patch overload that defeats CoCoNuT.Nopol, RSRepair, and Astor lack the required fix pattern for this bug.
  • NMT comparison: CURE fixes Closure 10 by extracting the valid identifier “anyResultsMatch” even though it is declared outside the buggy context.CoCoNuT prioritizes tokens in the buggy line and context and therefore fails to generate that identifier.
  • Pattern-based comparison: CURE fixes Math 41 while TBar does not because the repair requires coordinated changes to loop initialization and its condition.The paper attributes CURE’s success to learning similar transformations from patch-training data.
  • Compilability: CURE produces more compilable top-ranked patches than SequenceR and CoCoNuT across the reported top-k ranges.The ablation comparison indicates contributions from the model components and code-aware search.
  • Scope: CURE is designed for single-hunk bugs, where buggy lines and patches are single code segments with separate test cases.This is the stated applicability scope shared with many state-of-the-art generate-and-validate APR techniques.

B. RQ2: What are the contributions of CURE’s components?

CURE’s components improve repair by combining GPT pre-training, code-aware search, and subword tokenization. Ablation results show gains in bug fixing, compilability, patch ranking, and search-space coverage.

  • 1) Impact of the GPT PL model:: GPT pre-training helps fix six more bugs in each benchmark when comparing BPE+GPT+CoNuT+vanilla with BPE+CoNuT+vanilla.
  • 1) Impact of the GPT PL model:: GPT increases the average compilable rate by 2%–4% and improves correct-patch rank from 414 to 131.The reported average rank is 68% higher for BPE+GPT+CoNuT+vanilla than for BPE+CoNuT+vanilla.
  • 2) Impact of the code-aware beam-search strategy:: Code-aware beam search helps fix six more Defects4J bugs and four more QuixBugs bugs than BPE+GPT+CoNuT+vanilla.It also increases average compilable rate by 3%–7%.
  • 2) Impact of the code-aware beam-search strategy:: CURE’s code-aware beam search improves correct-patch rank from 131 to 101 before validation.The reported improvement is 21% higher than BPE+GPT+CoNuT+vanilla.
  • 2) Impact of the code-aware beam-search strategy:: Length control reduces the average candidate-to-correct-fix length difference from seven tokens to five tokens.This produces candidate patches more similar in length to correct fixes.
  • 3) Impact of subword tokenization:: Subword tokenization reduces vocabulary size from 139,423 to 50,057 tokens and helps cover more correct fixes.It helps fix four more bugs in one comparison and 10 more bugs in another.
  • 3) Impact of subword tokenization:: Subword tokenization separates out-of-vocabulary tokens in 14 bugs into more common tokens.The reported examples include “binsearch” and “charno”.

C. Execution Time

CURE incurs one-time costs for data extraction, pre-training, and APR fine-tuning, followed by per-bug generation and validation costs. The authors report that CURE uses fewer models and validates fewer patches than a state-of-the-art NMT approach.

  • Execution cost: CURE takes 2.5 minutes to generate 10,000 candidate patches and 16.5 minutes to validate one bug on average.Inference uses four GPUs.
  • Execution cost: Pre-training the GPT PL model takes ten days on four GPUs for five epochs, while fine-tuning the top-10 APR models takes 10.7 hours per model on average.The paper characterizes these training expenses as one-time costs.
  • Execution cost: CURE’s reported per-bug process uses fewer models and validates fewer patches than the state-of-the-art NMT-based approach.The authors report that CURE is faster and fixes more bugs in this comparison.
  • Scope: The evaluation uses two Java benchmarks, although the authors state that CURE is not tied to a specific programming language or benchmark.Generalization to other languages requires updating the programming-language parser.
Loading 2103.00073v4…