PaperlayerView exact revision on arXiv

Source-linked AI summary

NARRepair: Non-Autoregressive Code Generation Model for Automatic Program Repair

Zhenyu Yang, Zhen Yang, Zhongxing Yu

arXiv:2406.16526v1cs.SEcs.AI

TL;DR

Long word-by-word APR inference delays hinder repair of complex real-life bugs. NARRepair is a NAR model designed for APR; it is reported 5.4 to 18.6 times faster than other APR models while maintaining accuracy.

  • Problem

    Word-by-word APR inference creates long delays for complex real-life bugs, which can involve modifying long code.

    Source: §1 · p. 2, §2 · p. 5

  • Method

    NARRepair is a NAR model designed for APR to increase inference speed while maintaining repair accuracy.

    Source: §1 · p. 4, §7 · p. 21

  • Results

    5.4 to 18.6 times faster than other automatic program repair models, while maintaining repair accuracy.

    Source: §7 · p. 21

  • Takeaways & Limitations

    The reported speedup addresses the long repair delays identified for complex real-life bugs.

    Source: §1 · p. 2, §7 · p. 21

    The evaluated bugs are only Java bugs; further studies are needed to investigate effectiveness on other programming languages.

    Source: §6.1 · p. 21

Abstract

from arXiv · show

With the advancement of deep learning techniques, the performance of Automatic Program Repair(APR) techniques has reached a new level. Previous deep learning-based APR techniques essentially modified program sentences in the Autoregressive(AR) manner, which predicts future values based on past values. Due to the manner of word-by-word generation, the AR-based APR technique has a huge time delay. This negative consequence overshadows the widespread adoption of APR techniques in real-life software development. To address the issue, we aim to apply the Non-Autoregressive(NAR) method to the APR task, which can output target code in a parallel manner to avoid huge inference delays. To effectively adapt the NAR manner for the APR task, we in this paper propose NARRepair, the first customized NAR code generation model for the APR task. The NARRepair features three major novelties, including 1) using repair actions to alleviate the over-correction issue, 2) extracting dependency information from AST to alleviate the issue of lacking inter-word dependency information, 3) employing two-stage decoding to alleviate the issue of lacking contextual information. We evaluated NARRepair on three widely used datasets in the APR community, and the results show that our technique can significantly improve the inference speed while maintaining high repair accuracy.

1 INTRODUCTION

The introduction motivates adapting parallel non-autoregressive generation to automatic program repair, addressing the delays of sequential generation and the challenges of naïve non-autoregressive methods. It presents NARRepair and its techniques for repair accuracy, alongside evaluation reporting high accuracy and reduced inference latency.

1 INTRODUCTION

  • NARRepair uses parallel non-autoregressive generation to reduce inference delays, while addressing the over-correction, lost inter-word dependency, and missing contextual information that limit naïve NAR for repair.The paper identifies sequential autoregressive generation as a source of delays, especially for long code sequences.Source: §1 · pp. 2–3
  • Action-guided repair predicts “keep”, “replace”, “insert”, or “delete” for each buggy-code word, reducing the learned output dimension from dictionary words to actions and helping prevent over-correction.Predicted actions guide the decoder’s word generation.Source: §1 · p. 3
  • NARRepair uses nearest common parent nodes in the abstract syntax tree to represent inter-word dependencies, supplying dependency information during parallel generation to improve grammaticality.The method is intended to capture structural and semantic dependencies between words.Source: §1 · pp. 3–4
  • Two-stage decoding keeps high-confidence words, masks the remaining words, then regenerates them using contextual information to improve repair accuracy.High-confidence words include adjacent words with “keep” actions and words whose prediction probabilities exceed a threshold.Source: §1 · p. 4
  • NARRepair is 5.4–15.1 times faster on CPU and 6.2–18.6 times faster on GPU than autoregressive APR models, fixing 69 Defect4J v1.2 bugs (90% of the optimal model, Tare) and 45 Defect4J v2.0 bugs (95% of the optimal model, Rewardrepair).The introduction reports these results across the two Defect4J datasets.Source: §1 · p. 4
  • NARRepair fixes 23 QuixBugs bugs (82% of the optimal model, AlphaRepair), while its accuracy exceeds advanced NAR models across three datasets and inference latency is greatly reduced.The paper reports that NARRepair maintains high accuracy while increasing inference speed.Source: §1 · p. 4
  • NARRepair is introduced as the first non-autoregressive model designed specifically for the automatic program repair task.The authors qualify the priority claim with “to the best of our knowledge.”Source: §1 · p. 4
  • Three techniques address naive non-autoregressive repair’s limitations: repair-action prediction for over-correction, AST dependency extraction for missing inter-word dependencies, and two-stage decoding for missing context.The paper identifies the inter-word dependency extractor as extracting dependency information from ASTs.Source: §1 · p. 4

2 RELATED WORK

Prior APR work spans search-, constraint-, and template-based methods, while many deep-learning approaches use autoregressive sequence-to-sequence generation that can slow repair. Non-autoregressive models generate in parallel, but the paper says naive transfer to APR performs poorly and introduces NARRepair for the task.

APR Approaches

  • APR also includes search-based methods, constraint-based methods, and template-based methods.These respectively search a predefined space, solve constraints, or apply targeted repair templates.Source: §2 · p. 5

Deep Learning-based APR

  • DeepFix predicts defect locations and corrected code, while DeepRepair ranks similar code and maps out-of-scope identifiers to similar in-scope identifiers.SequenceR uses a sequence-to-sequence model with abstract context to simulate bug analysis and repair.Source: §2 · p. 5
  • Other neural APR methods include CoCoNuT’s convolutional translation model, Recoder’s syntax constraints, and RewardRepair’s test information for compilable candidates.Related work also treats repair as contextual fill-in-the-blanks or uses type awareness to reduce unusable patches.Source: §2 · p. 5
  • Most deep-learning APR techniques use sequence-to-sequence models that generate autoregressively, whereas non-autoregressive models aim to generate target sentences in parallel.The paper presents parallel generation as a way to reduce inference time.Source: §2 · p. 5
  • Autoregressive generation waits for earlier output positions, leading to slow reasoning and delays that hinder real-time repair of complex bugs.The section links these delays to modifications involving long code snippets and obstacles to adoption in software development and maintenance.Source: §2 · p. 5

Non-Autoregressive Models

  • Prior non-autoregressive models reduce inference time by generating target sentences in parallel; Gu et al. assume target words are independent and generate them in one step, while other cited approaches use latent variables [2].PNAR uses latent variables to capture target-word arrangement [2]. Shu et al. use a spherical Gaussian to generate latent variables for each input word.Source: §2 · p. 5
  • NARRepair is proposed to address APR’s distinct needs because directly applying existing non-autoregressive models does not yield satisfactory results.The paper identifies over-correction, weak inter-word dependency information, and limited contextual information as challenges for NAR in APR.Source: §2 · p. 5

3 NARREPAIR

This section describes NARRepair’s architecture and inference pipeline for generating repaired code in parallel, including repair actions, AST-derived dependencies, and two-stage decoding. It also covers inference choices such as output lengths and generating 200 patches per buggy program.

3.1 Code Encoder

  • The code encoder converts buggy-code text into word embeddings for subsequent prediction and tagging, using the encoder part of a transformer.Source: §3.1 · p. 7

3.2 Repair Action Predictor

  • The repair-action predictor assigns each buggy-code word a repair action, with four types (“keep”, “insert”, “delete”, and “replace”), and a repair length representing generated repair words per fixed word.Typically, “replace” and “keep” have length 1, “delete” has length 0, and “insert” has length equal to the number of words inserted.Source: §3.2 · p. 7

3.3 Inter-word Dependency Extractor

  • The extractor derives word-pair dependencies from nearest common AST parents, predicts this dependency matrix, and fuses the information into word features for decoding.The matrix is symmetric because the relationship between a word pair is equivalent in either order.Source: §3.3 · pp. 8–9

3.4 Two-stage NAR Decoder

  • The two-stage decoder keeps high-confidence words, then masks and regenerates low-confidence words using the retained context; the evaluation threshold is 0.7.High-confidence words include adjacent words labeled “keep” and words whose prediction probabilities exceed the threshold.Source: §3.4 · p. 10

3.5 Training and Inference

  • The three NARRepair sub-modules are jointly learned during training.Source: §3.5 · p. 11
  • The training-loss hyperparameters α and λ adjust the importance of each loss, and α is set to 0.1.The supplied passage truncates the value assigned to λ.Source: §3.5 · p. 11
  • At inference, NARRepair predicts repair actions and output lengths per buggy-code word, uses attention and AST-derived dependencies to build fixed-code features, then decodes the repair in two stages.The decoder conditions on the repair actions and dependency-aware fixed-code features.Source: §3.5 · p. 11
  • For each buggy program, NARRepair generates 200 patches and tests whether they include the correct patch.Source: §3.5 · p. 11

4 EVALUATION SETUP

The evaluation tests NARRepair's performance, component contributions, and pretrained-model use, using collected repair data, comparison baselines, patch metrics, and specified implementation settings.

4.1 Research Questions

  • The research questions compare NARRepair with deep-learning APR and NAR models, assess module contributions, and examine pretrained-model use, over-correction, and AST parent-child relationships.For over-correction, the study compares correct words changed into incorrect ones before and after removing two modules; it also tests whether nearest AST parents relate closely to their children.Source: §4.1 · pp. 11–12

4.2 Data Collection

  • The model dataset uses selfAPR-published code-text pairs supplemented with real-project GitHub data, totaling 1.52 million instances, with 90% used for training and 10% for validation.Source: §4.2 · p. 12

4.3 Data Preprocessing

  • Repair actions come from edit-distance calculation and backtracking, while AST paths and nearest common parents supply the inter-word dependency matrix.Tree-sitter generates ASTs for correct code; the resulting actions and matrix are prepared for each training example.Source: §4.3 · p. 12

4.4 Knowledge Distillation

  • To address multimodal targets that can yield incompatible combinations of individually valid tokens, the study distills CodeT5-large outputs and trains mainly on distilled data alongside original data.The text generated by the trained CodeT5-large model on the original training set forms the distilled dataset.Source: §4.4 · p. 13

4.5 Baselines

  • Baselines include six APR models and five NAR models; CodeT5 weights are loaded into NARRepair and NAR baselines for comparison.The APR set includes SequenceR, CoConut, Rewardrepair, Recoder, AlphaRepair, and Tare; NAR baselines include DePA, OAXE-NAR, Fully-NAR, SNAR, and CTC.Source: §4.5 · p. 13

4.6 Metrics

  • Patch validity is assessed with tests and manual analysis: passing all tests makes a patch plausible, while manual confirmation makes it correct.Correct patches are a subset of plausible patches; results report both counts.Source: §4.6 · pp. 13–14
  • Table 1 reports corrected-bug counts and inference-speed differences, including comparisons with the fastest autoregressive model, SequenceR, and NARRepair.Positive speed differences indicate speeding up and negative differences indicate slowing down; a dash marks unpublished values.Source: Table 1 · p. 14, §4.6 · p. 14

4.7 Implementation Details

  • Implementation uses PyTorch and Adam, with batch size 50, 200 epochs, maximum sequence length 512, and maximum repair length 20.The learning rate ranges from 0 to 0.00005.Source: §4.7 · p. 14

5 EVALUATION RESULTS

This section evaluates NARRepair’s repair performance and inference speed against baseline models, assesses its modules, and examines loading pretrained-model weights. It reports faster inference with pretrained weights while retaining much of their AR performance, reduced changes to correct words from repair components, and declining AST-node similarity with distance in support of modeling dependencies through nearest common parent nodes.

5.1 (RQ1) Results of Comparison with the Baselines

  • With patch evaluation time included, NARRepair remains 4.2 to 11.3 times faster than baseline models.The evaluation times for patches generated by different models show no significant differences, reducing the inference-speed gap but leaving NARRepair faster.Source: Table 2 · p. 15, §5.1 · p. 16
  • NARRepair fixes 69 bugs on Defect4J v1.2 (90% of Tare), 45 on Defect4J v2.0 (95% of Rewardrepair), and 23 on QuixBugs (82% of AlphaRepair).It also exceeds the optimal NAR model by 197%, 186%, and 209% in repaired programs across the three datasets, respectively.Source: §5.1 · p. 15
  • NARRepair correctly repairs 51, 26, and 20 bugs on Defect4J v1.2, Defect4J v2.0, and QuixBugs, respectively, improving over other NAR models by more than 150% across all three datasets.Source: §5.1 · p. 15
  • Against AlphaRepair, NARRepair is 18.6 times faster on GPU and 15.2 times faster on CPU; against SequenceR, it is 5.4 times faster on CPU and 6.2 times on GPU.The paper also reports that NARRepair’s repair performance is better than that of some autoregressive APR models, such as CoCoNut.Source: §5.1 · pp. 15–16

5.2 (RQ2) Results of Ablation Study

  • Removing the two-stage decoder reduces repairs by 15, removing the inter-word dependency extractor reduces them by 9, and the repair-action predictor has the greatest impact.The decoder’s [Mask]-tag contextual information and extracted dependency information improve accuracy; predicting repair actions helps avoid changing correct words into incorrect ones, while removing attention-based feature fusion causes a slight performance decrease.Source: §5.2 · p. 16

5.3 (RQ3) Results of Loading Pre-trained Model Weights

  • The study loads CodeBERT[11], GraphCodeBERT[16], CodeT5[63], and CodeGPT[39] weights into NARRepair and compares generated results with those of the pretrained models.The supplied passages describe the experiment setup but do not report its results.Source: §5.3 · p. 17
  • Compared with four pretrained models, NARRepair runs 6.2–6.4 times faster; after loading their weights, it achieves 85.7%–94.5% of their performance obtained using the AR manner.Source: §5.3 · p. 18

5.4 (RQ4) Results of Alleviating the Over-Correction Problem

  • Removing the repair action predictor increased correct words modified by 0.8 on average; removing the two-stage decoder as well increased them by a further 2.3, indicating the decoder was more effective at preventing over-correction.The study also concludes that the repair action predictor remains important to the full model.Source: §5.4 · p. 18, Table 6 · p. 19

5.5 (RQ5) Results of the Similarity Between AST Nodes

  • AST parent-child cosine similarity decreased with distance, while its fluctuation range increased at greater distances.The authors present this trend as evidence that parent-child relationships are close and become less close as distance grows.Source: Figure 8 · p. 19, §5.5 · p. 19
  • In code examples, parent-child similarities were high and declined with distance, supporting nearest-common-parent nodes as word dependencies.Examples include the binary expression and parameter nodes in the AST of “int add (int a, int b) {return a+b;}”.Source: Figure 9 · p. 20, §5.5 · p. 20

6 DISCUSSION

The discussion identifies benchmark and implementation-related threats to validity, then proposes broader evaluations and further work on speed and accuracy.

6.1 Threats to Validity

  • Internal-validity threats include potential faults in NARRepair or its evaluation; the implementation uses mature libraries and the authors report thorough testing.Baseline accuracy is taken from published papers when available, with author-provided models rerun when needed.Source: §6.1 · p. 20
  • The evaluation uses Defect4J v1.2, Defect4J v2.0, and QuixBugs, but all three contain Java bugs, so effectiveness on other programming languages remains to be investigated.The authors also say evaluation on additional benchmarks could further confirm effectiveness and generalizability.Source: §6.1 · p. 21

6.2 Future Work

  • Future work targets higher accuracy alongside faster inference, broader benchmark and language evaluations, and applying the ideas to other speed-sensitive software-engineering tasks.The authors also aim to evaluate NARRepair on more benchmarks and programming languages to examine effectiveness and generalizability.Source: §6.2 · p. 21

7 CONCLUSIONS

NARRepair adapts non-autoregressive generation to automatic program repair with modules addressing over-correction, missing dependencies, and missing context. On three APR datasets, it maintains high repair accuracy while outperforming other APR models in inference speed.

7 CONCLUSIONS

  • NARRepair pairs a repair-action predictor, inter-word dependency extractor, and two-stage decoder with the over-correction, dependency-information, and contextual-information problems, respectively.The model is proposed as a non-autoregressive approach for generating fixed code sequences.Source: §7 · p. 21
  • 5.4 to 18.6 times faster than other automatic program repair models, NARRepair maintains high repair accuracy and performs better than other non-autoregressive models.The evaluation covers Defect4J v1.2, Defect4J v2.0, and QuixBugs.Source: §7 · p. 21
  • The authors report that the method is effective for pretrained models and may generalize to large language models for real-time code repair.The stated aim is to increase inference speed while maintaining repair accuracy.Source: §7 · p. 21

Related papers

Loading 2406.16526v1…