CURE: Code-Aware Neural Machine Translation for Automatic Program Repair

• Correct repair requires both including the correct fix in the search space and finding it efficiently.Source: §I · p. 1
• Source code’s strict syntax, semantics, and unbounded identifiers create challenges that distinguish code repair from natural-language translation.Source: §I · pp. 1–2
• Up to 67%–97% of patches from state-of-the-art NMT-based APR models are uncompilable.Generated patches can call methods with wrong parameters, invoke undeclared variables, or contain mismatched parentheses.Source: §I · pp. 1–2

Programming language models

• CURE pre-trains a programming-language model on software code to learn correct syntax and developer-like code before fine-tuning for APR.The PL model predicts next code tokens and is combined with an NMT model for APR.Source: A. Our approach · p. 2

Code-aware search strategy

• CURE’s code-aware beam search restricts generation to valid identifiers and penalizes patches whose lengths differ substantially from buggy lines.These strategies target uncompilable candidates and very short or long sequences produced by large-beam search.Source: A. Our approach · p. 2

Subword tokenization

• BPE tokenization addresses out-of-vocabulary compound and rare identifiers while reducing the search space and retaining more correct patches.Source: A. Our approach · p. 3

• CURE pre-trains a PL model on 4.04 million methods from 1,700 open-source projects to capture code syntax and developer-like source code.Source: B. Contributions · p. 3
• CURE combines a pre-trained PL model with NMT architectures to learn both code syntax and repair patterns.Source: B. Contributions · p. 3
• CURE’s code-aware beam search uses valid-identifier checking and length control to find more correct fixes.Source: B. Contributions · p. 3
• CURE applies subword tokenization to address the out-of-vocabulary problem in automatic program repair.Source: B. Contributions · p. 3
• 57 Defects4J bugs and 26 QuixBugs bugs are correctly fixed by CURE, the highest totals among evaluated APR tools.The evaluation uses the two widely used benchmarks Defects4J and QuixBugs.Source: B. Contributions · p. 3

• Candidate patches are generated patches, plausible patches pass validation, and correct patches match or are semantically equivalent to developers’ patches.Source: §II · p. 3
• Pre-training learns a general task from a very large dataset, after which fine-tuning adapts the model to a specific task with fewer data.Source: §II · p. 3
• CoNuT provides CURE’s NMT architecture through buggy-line and context encoders, a merger, decoder, attention module, and token-generation module.Source: §II · p. 3
• CURE’s three techniques target distinct functions: subword tokenization improves the search space, PL modeling learns developer-like code, and code-aware beam search improves patch ranking.Source: §III · p. 3

Training

• Training extracts and tokenizes methods from open-source projects, then uses them to train a PL model that learns developer-like code with correct syntax.Source: A. Overview · p. 3

Inference

• During inference, CURE tokenizes buggy and context lines, extracts in-scope identifiers, and generates candidate patches with code-aware beam search.Source: A. Overview · p. 3

Validation

• Validation compiles patched projects and executes their test suites before outputting plausible patches for developer examination.Source: A. Overview · p. 4

• CURE trains its PL model on millions of methods from open-source Java projects and fine-tunes it using buggy lines, context, and correct fixes.Source: B. Data Extraction · p. 4
• Enhanced word-level tokenization separates code using spaces, camel letters, underscores, strings, and numbers.Source: C. Code Representation and Tokenization · p. 4
• 2% of test-set tokens remain out of vocabulary after word-level tokenization, and excluding rare project-specific tokens can hinder bug fixing.Source: C. Code Representation and Tokenization · p. 4
• BPE reduces vocabulary size and mitigates out-of-vocabulary problems by iteratively merging frequent byte pairs.Source: C. Code Representation and Tokenization · p. 4
• Subword tokenization splits the OOV identifier “charno” into “char@@” and “no”, enabling CURE to generate the correct patch.Source: C. Code Representation and Tokenization · pp. 4–5
• CURE forms a smaller search space that contains more correct patches through subword tokenization.Source: C. Code Representation and Tokenization · p. 5

D. Programming Language Model

• CURE trains a GPT programming-language model on open-source programs to predict next tokens and learn developer-like code.Source: D. Programming Language Model · p. 5
• Pre-training separates programming-language learning from patch learning and supplies syntax knowledge before APR fine-tuning.Source: D. Programming Language Model · p. 5

E. Fine-Tuning for APR with a PL Model

• The APR model combines the pre-trained GPT module with CoNuT and is fine-tuned using buggy lines, context, and correct fixes.Source: E. Fine-Tuning for APR with a PL Model · pp. 5–6
• The combined objective adds an auxiliary language-model likelihood to the NMT objective, weighted by λ, to retain pre-training information.Source: E. Fine-Tuning for APR with a PL Model · p. 5
• During inference, CURE encodes the buggy line and entire buggy method separately before merging the encoder outputs for patch generation.Source: E. Fine-Tuning for APR with a PL Model · p. 6
• CURE initializes decoding with the last context token before the buggy line rather than CoCoNuT’s “<START>” token.Source: E. Fine-Tuning for APR with a PL Model · p. 6

• Combining models with different hyperparameters and CoNuT or FConv architectures forms CURE’s ensemble.Source: F. Ensemble Learning · p. 6

Search motivation

• Beam search retains only the n highest-scoring nodes at each step, but vanilla search uses model log-probability without code information.This can favor out-of-scope variables and produce uncompilable candidate patches.Source: G. Code-Aware Beam-Search Strategy and… · p. 6

Code-aware search

• CURE combines valid-identifier checking and length control to make beam search code-aware.The two strategies incorporate identifier validity and patch-length information into candidate selection.Source: G. Code-Aware Beam-Search Strategy and… · p. 6

Valid-identifier check

• Static analysis extracts valid identifiers, tokenizes them into subwords, and maps prefixes to valid succeeding tokens during decoding.For example, the mapping lets the search complete “max_ending_” with “here” when that identifier is valid.Source: G. Code-Aware Beam-Search Strategy and… · p. 7
• Vanilla beam search can discard the correct-fix subtree when a lower average log-likelihood path loses to higher-scoring alternatives.In the example, the correct “... Math .” path is dropped, whereas invalid “max CaMeL” is penalized and the correct subtree remains searchable.Source: G. Code-Aware Beam-Search Strategy and… · p. 7

Length control

• 75% of bugs in the patch training data have fix-length differences of at most 5 tokens, motivating penalties for patches much shorter or longer than buggy lines.Length control adjusts the log-likelihood of <EOS>; a tolerance threshold of 5 provides flexibility.Source: G. Code-Aware Beam-Search Strategy and… · p. 7
• Length control prevents very short candidates from consuming beam slots, allowing search to explore deeper paths when the buggy sequence is longer.In the example, the short “{ }” patch receives a large <EOS> penalty and is excluded from the top 1,000 nodes.Source: G. Code-Aware Beam-Search Strategy and… · pp. 7–8
• CURE focuses on fixes similar in length to buggy lines, while its length-control strategy can be adapted to longer patches by changing penalty weights.The paper suggests separating similar-length and other bugs as a possible decomposition of APR.Source: G. Code-Aware Beam-Search Strategy and… · p. 8

Patch reconstruction

• CURE reconstructs subwords into words and replaces abstracted numbers and strings using donor code from the buggy file.The resulting token sequences become code statements before patch insertion.Source: H. Patch Validation · p. 8

Patch validation

• CURE ranks reconstructed statements by average token log-probability, inserts them at buggy lines, and compiles every patched project.Compilation filters out uncompilable patches before test-based validation.Source: H. Patch Validation · p. 8
• A patch is accepted when it preserves passing tests and passes at least one test that failed on the buggy project.These are the validation criteria used in previous work.Source: H. Patch Validation · p. 8

Data and protocol

• CURE avoids future-data leakage by using commits made before the first benchmark bug in 2006 for pre-training, fine-tuning, and validation.This design makes the evaluation realistic with respect to available historical data.Source: §IV · p. 8
• The programming-language training set contains 4.04 million methods extracted from 1,700 open-source Java projects rolled back before 2006.Methods longer than 1,024 tokens and abstract methods were excluded; 30,000 instances were used for validation.Source: §IV · p. 8
• Patch training data comes from 45,180 Java projects and excludes instances whose context or fixes exceed 1,024 tokens after subword tokenization.The source data is CoCoNuT’s shared Java patch-training corpus.Source: §IV · p. 8

Model configuration

• CURE uses BPE with a target vocabulary of 50,000 and trains a GPT model with eight transformer layers and six attention heads.The GPT model uses embedding size 384 and is trained for five epochs.Source: §IV · p. 8
• Hyperparameters are tuned with random search, training 100 APR models on a smaller patch-data subset before retaining the top five models for each GPT-based combination.The search covers convolution dimensions, kernel sizes, convolutional layers, dropout, and λ=0.3.Source: §IV · p. 8

Evaluation benchmarks

• Evaluation uses Defects4J v1.4.0 and QuixBugs, excluding duplicate Closure 63 and Closure 93 bugs.Plausible patches pass relevant tests, while two co-authors independently assess correctness as identical or semantically equivalent to developer patches.Source: §V · p. 8

Baselines

• CURE is compared with 25 APR techniques, including pattern-based and NMT-based approaches, none of which combines BPE, a pre-trained PL model, and code-aware search.Table I reports correctly fixed and plausible bugs as x/y.Source: §V · p. 8, Table I · p. 9

Overall performance

• 57 Defects4J bugs and 26 QuixBugs bugs are correctly fixed by CURE, the highest totals reported for both benchmarks.CURE outperforms the best existing approach on each benchmark.Source: A. RQ1: How does CURE perform against… · p. 9
• CURE fixes five more Defects4J bugs than TBar and 13 more than the best NMT-based approach, CoCoNuT.It also fixes twice as many QuixBugs bugs as CoCoNuT and fixes 12 bugs no evaluated QuixBugs tool fixes.Source: A. RQ1: How does CURE perform against… · p. 9

Unique fixes

• Chart 17 is a Defects4J bug that only CURE fixes among the 25 compared approaches.Its correct patch ensures the second parameter is non-negative, a transformation captured by the PL model despite being rare in patch-training data.Source: A. RQ1: How does CURE perform against… · p. 9
• CURE fixes the QuixBugs KTH bug because its code-aware search avoids the uncompilable-patch overload that defeats CoCoNuT.Nopol, RSRepair, and Astor lack the required fix pattern for this bug.Source: A. RQ1: How does CURE perform against… · p. 9

NMT comparison

• CURE fixes Closure 10 by extracting the valid identifier “anyResultsMatch” even though it is declared outside the buggy context.CoCoNuT prioritizes tokens in the buggy line and context and therefore fails to generate that identifier.Source: A. RQ1: How does CURE perform against… · p. 9

Pattern-based comparison

• CURE fixes Math 41 while TBar does not because the repair requires coordinated changes to loop initialization and its condition.The paper attributes CURE’s success to learning similar transformations from patch-training data.Source: A. RQ1: How does CURE perform against… · p. 9

Compilability

• CURE produces more compilable top-ranked patches than SequenceR and CoCoNuT across the reported top-k ranges.The ablation comparison indicates contributions from the model components and code-aware search.Source: A. RQ1: How does CURE perform against… · p. 9, Table II · p. 10, Table III · p. 10

Scope

• CURE is designed for single-hunk bugs, where buggy lines and patches are single code segments with separate test cases.This is the stated applicability scope shared with many state-of-the-art generate-and-validate APR techniques.Source: A. RQ1: How does CURE perform against… · p. 10

1) Impact of the GPT PL model:

• GPT pre-training helps fix six more bugs in each benchmark when comparing BPE+GPT+CoNuT+vanilla with BPE+CoNuT+vanilla.Source: 1) Impact of the GPT PL model · p. 10
• GPT increases the average compilable rate by 2%–4% and improves correct-patch rank from 414 to 131.The reported average rank is 68% higher for BPE+GPT+CoNuT+vanilla than for BPE+CoNuT+vanilla.Source: 1) Impact of the GPT PL model · p. 10

2) Impact of the code-aware beam-search strategy:

• Code-aware beam search helps fix six more Defects4J bugs and four more QuixBugs bugs than BPE+GPT+CoNuT+vanilla.It also increases average compilable rate by 3%–7%.Source: 2) Impact of Code-Aware Beam-Search… · p. 10
• CURE’s code-aware beam search improves correct-patch rank from 131 to 101 before validation.The reported improvement is 21% higher than BPE+GPT+CoNuT+vanilla.Source: 2) Impact of Code-Aware Beam-Search… · p. 10
• Length control reduces the average candidate-to-correct-fix length difference from seven tokens to five tokens.This produces candidate patches more similar in length to correct fixes.Source: 2) Impact of Code-Aware Beam-Search… · p. 10

3) Impact of subword tokenization:

• Subword tokenization reduces vocabulary size from 139,423 to 50,057 tokens and helps cover more correct fixes.It helps fix four more bugs in one comparison and 10 more bugs in another.Source: 3) Impact of subword tokenization · p. 10
• Subword tokenization separates out-of-vocabulary tokens in 14 bugs into more common tokens.The reported examples include “binsearch” and “charno”.Source: 3) Impact of subword tokenization · pp. 10–11

Execution cost

• CURE takes 2.5 minutes to generate 10,000 candidate patches and 16.5 minutes to validate one bug on average.Inference uses four GPUs.Source: C. Execution Time · p. 11
• Pre-training the GPT PL model takes ten days on four GPUs for five epochs, while fine-tuning the top-10 APR models takes 10.7 hours per model on average.The paper characterizes these training expenses as one-time costs.Source: C. Execution Time · p. 11
• CURE’s reported per-bug process uses fewer models and validates fewer patches than the state-of-the-art NMT-based approach.The authors report that CURE is faster and fixes more bugs in this comparison.Source: C. Execution Time · p. 11

Scope

• The evaluation uses two Java benchmarks, although the authors state that CURE is not tied to a specific programming language or benchmark.Generalization to other languages requires updating the programming-language parser.Source: §VI · p. 11