PaperlayerView exact revision on arXiv

Source-linked AI summary

CIRCLE: Continual Repair across Programming Languages

Wei Yuan, Quanjun Zhang, Tieke He, Chunrong Fang, Nguyen Quoc Viet Hung, Xiaodong Hao, Hongzhi Yin

arXiv:2205.10956v4cs.SE

TL;DR

Most deep-learning APR systems are limited to one programming language and offline learning, despite growing repair requirements. CIRCLE uses a T5-based model with prompting, replay, regularization, and re-repairing for continual cross-language repair, and the paper reports state-of-the-art performance from one model across languages. Its evaluation assumes languages arrive in popularity order.

  • Problem

    Most deep-learning APR systems repair only one language and are developed offline, limiting support for multiple languages and newly arriving requirements.

    Source: §1 · p. 2

  • Method

    CIRCLE uses a prompted T5-based repair model with difficulty-based example replay, EWC regularization, and re-repairing for continual repair across languages.

    Source: §1 · p. 2

  • Results

    A single CIRCLE model achieves state-of-the-art performance across languages and benchmarks compared with selected APR baselines, while continually learning without severely forgetting previous knowledge.

    Source: §1 · p. 3, §7 · p. 11

  • Takeaways & Limitations

    The reported results show that one model can continually learn bug repair across multiple programming languages in the evaluated setting.

    Source: §1 · p. 3

    The continual-learning evaluation orders languages by popularity: JavaScript, Python, Java, then C.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Automatic Program Repair (APR) aims at fixing buggy source code with less manual debugging efforts, which plays a vital role in improving software reliability and development productivity. Recent APR works have achieved remarkable progress via applying deep learning (DL), particularly neural machine translation (NMT) techniques. However, we observe that existing DL-based APR models suffer from at least two severe drawbacks: (1) Most of them can only generate patches for a single programming language, as a result, to repair multiple languages, we have to build and train many repairing models. (2) Most of them are developed in an offline manner. Therefore, they won't function when there are new-coming requirements. To address the above problems, a T5-based APR framework equipped with continual learning ability across multiple programming languages is proposed, namely \emph{C}ont\emph{I}nual \emph{R}epair a\emph{C}ross Programming \emph{L}anguag\emph{E}s (\emph{CIRCLE}). Specifically, (1) CIRCLE utilizes a prompting function to narrow the gap between natural language processing (NLP) pre-trained tasks and APR. (2) CIRCLE adopts a difficulty-based rehearsal strategy to achieve lifelong learning for APR without access to the full historical data. (3) An elastic regularization method is employed to strengthen CIRCLE's continual learning ability further, preventing it from catastrophic forgetting. (4) CIRCLE applies a simple but effective re-repairing method to revise generated errors caused by crossing multiple programming languages. We train CIRCLE for four languages (i.e., C, JAVA, JavaScript, and Python) and evaluate it on five commonly used benchmarks. The experimental results demonstrate that CIRCLE not only effectively and efficiently repairs multiple programming languages in continual learning settings, but also achieves state-of-the-art performance with a single repair model.

1 INTRODUCTION

Deep-learning APR can learn bug-fixing patterns, but most systems are limited to one language and offline training. CIRCLE combines a T5-based repair model with continual-learning strategies to repair across languages, and the reported experiments show state-of-the-art performance against dedicated APR systems.

1 INTRODUCTION

  • Deep-learning APR learns bug-fixing patterns from code repositories, commonly treating repair as translation from buggy code to fixed code.Existing approaches use encoder-decoder models to encode buggy code and context, then generate repaired statements.Source: §1 · pp. 1–2
  • Most deep-learning APR systems repair only one language, requiring separate models and limiting transfer of learned knowledge across languages.The paper argues that code understanding and bug-identification abilities may be shared across languages.Source: §1 · p. 2
  • Offline APR systems cannot continually improve as requirements or bug-fixing corpora grow, and retraining on all corpus is time-consuming.The paper describes new language services and newly collected repair data as examples of changing requirements.Source: §1 · p. 2
  • Cross-language repair is harder than single-language repair, while learning new tasks can cause catastrophic forgetting of knowledge from earlier tasks.The paper identifies effective use of large pre-trained models and prevention of forgetting as challenges.Source: §1 · p. 2
  • A single CIRCLE model continually learns repair across languages without severely forgetting previous knowledge and outperforms conventional state-of-the-art APRs trained for particular languages.Experiments cover four languages and five benchmarks.Source: §1 · pp. 2–3
  • CIRCLE combines a prompt-based T5 repair model and re-repairing with difficulty-based replay and EWC regularization for continual bug fixing across languages.The prompt turns repair into a fill-in-the-blank task; replay and EWC address learning new tasks while limiting forgetting.Source: §1 · pp. 2–3

2 BACKGROUND

Deep-learning APR commonly treats repair as translation, while repair across multiple languages with one model remains underexplored. The background also distinguishes continual learning from approaches that retain and retrain on all historical data, and describes prompts as a way to align pre-trained and downstream tasks.

2.1 DL-based APR

  • Existing deep-learning APR commonly learns repair from bug-fix pairs, while using one model to repair multiple programming languages remains underexplored.The paper proposes T5 as the skeleton for a cross-language repair model.Source: §2.1 · p. 3

2.2 Continual Learning

  • Continual learning updates a model for new tasks while seeking good performance on all tasks seen so far, unlike multitask learning, which retrains on historical datasets.The paper says multitask learning is more costly because it must retain and reuse prior data.Source: §2.2 · p. 3
  • Catastrophic forgetting is the abrupt loss of previously learned task knowledge; mitigation methods include rehearsal, parameter regularization, and architectural changes.This work focuses on combining rehearsal with regularization.Source: §2.2 · p. 4

2.3 Prompt for Pre-trained Model

  • A prompt inserts tokens into an input to formulate a task as language modeling and bridge pre-trained tasks with downstream tasks.CIRCLE manually designs prefixes to concatenate input components.Source: §2.3 · p. 4

3 APPROACH

CIRCLE combines prompt-based T5 repair with rehearsal and elastic regularization to learn bug fixing across languages over time, then re-repairs generated patches to address cross-language errors.

3.1 Overview

  • CIRCLE processes arriving language-repair tasks sequentially, training on current data plus selected past examples and retaining only those examples rather than all historical data.Its training also applies EWC regularization to limit changes to parameters important for previous tasks.Source: §3.1 · p. 4

3.2 Prompt based Data Representation

  • A prompt template labels buggy code and context, then asks for fixed code, casting repair as a fill-in-the-blank task aligned with T5 pretraining.CIRCLE keeps T5’s original tokenization vocabulary rather than building a new one.Source: §3.2 · p. 4

3.3 T5 as APR Model Skeleton

  • CIRCLE uses pretrained T5 as its APR model skeleton; the authors describe this as the first use of T5 for continual, simultaneous multi-language APR.The model is an encoder-decoder transformer pretrained on over 750 GB of datasets.Source: §3.3 · p. 5

3.4 Difficulty-based Example Replay

  • Difficulty-based replay retains a small set of representative, diverse examples from past tasks, motivated by the possibility that patterns for difficult examples are forgotten first.The selection score is normalized by the inverse length of the target, and the N highest-scoring examples form each task’s set E_t, much smaller than D_t.Source: §3.4 · p. 5

3.5 Sampling-based EWC Regularization

  • EWC constrains updates according to parameter importance, measured by the Fisher matrix, to help preserve knowledge from earlier tasks.To avoid accessing all historical data, CIRCLE estimates the Fisher matrix using difficulty-selected examples and samples M items from them.Source: §3.5 · p. 6

3.6 Cross Language Re-repairing

  • Re-repairing addresses cross-language keyword mismatches with a mapping table and format mismatches by removing unnecessary spaces between tokens.It also replaces unknown tokens generated for rare programming-language symbols with those symbols.Source: §3.6 · p. 6

4 EXPERIMENTAL SETUP

The experiments ask whether CIRCLE learns effectively as tasks arrive, how one model compares with APR methods, and what its components contribute. They evaluate a four-language training setup on five benchmarks.

4.1 Research Questions

  • The study evaluates continual learning, comparison with state-of-the-art APR methods, and the contributions of CIRCLE’s components.For continual learning, languages arrive in the order JavaScript, Python, Java, then C; the final model is used for the APR-method comparison.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Datasets

  • Training data comes from CoCoNut’s released corpus, with Java data committed after 2006 removed and 0.4 million randomly selected pairs used for training.Inputs longer than 512 subword tokens are truncated.Source: §4.2 · p. 6

4.3 Implementation Details

  • The implementation uses HuggingFace t5-base with AdamW, a 3e-4 learning rate, up to 20 epochs per task, and an EWC λ of 110000.The replay set is capped at 20000 examples; inference uses beam search with beam size 250, top-k and top-p sampling, and up to 1000 candidate patches.Source: §4.3 · p. 7

4.4 Benchmarks and Baselines

  • Evaluation spans five benchmarks across Java, JavaScript, C, and Python, and compares CIRCLE with 30 APR techniques.Finetuned-APR serves as the continual-learning baseline, while the state-of-the-art comparison includes prior APR tools and three recent NMT-based tools.Source: §4.4 · p. 7

5 EVALUATION AND RESULTS

Across five benchmarks and four programming languages, the evaluation reports that one CIRCLE model achieves state-of-the-art repair performance. Experiments also examine continual learning, prompting, re-repairing, and cross-language keyword ambiguity.

5.2 RQ2: What is the performance of a single CIRCLE model compared to the dedicatedly trained state-of-the-art APR methods?

  • 120 bugs were fixed by one CIRCLE model across the benchmarks, achieving state-of-the-art performance in different languages.On Defects4J v1.2, CIRCLE repaired 23.1% (12 bugs) more bugs than TBar.Source: §5.2 · pp. 7–8, Table 2 · p. 8

5.1 RQ1: Can CIRCLE effectively learn bug fixing in “task requirements increase constantly” scenario?

  • After Task 3, CIRCLE fixed 26 QuixBugs-Py bugs versus Finetune-APR’s 18; on Task 2, it fixed 28 versus 23.The comparison tracks fixes on current and previously learned tasks; CIRCLE still fell slightly below an estimated upper bound.Source: §5.1 · p. 7, Figure 4 · p. 9

5.2 RQ2: What is the performance of a single CIRCLE model compared to the dedicatedly trained state-of-the-art APR methods?

  • On Defects4J, CIRCLE fixed 29 unique bugs versus traditional approaches and 14 versus DL-based approaches, showing complementary repairs.The analysis compares CIRCLE with TBar, FixMiner, CURE, and Recoder.Source: §5.2 · p. 9, Figure 5 · p. 9

5.3.1 Impact of prompt-based data representation.

  • With the same Java model and training epochs, prompt-based inputs reached lower validation loss than inputs without prompts.The authors describe the prompt words as marking the input with natural language to help exploit T5’s pre-trained knowledge.Source: §5.3.1 · p. 9, Figure 6 · p. 9

5.3.2 Impact of re-repairing.

  • Re-repairing increased fixed-bug counts on all benchmarks by addressing keyword, format, and rare-symbol mismatches.Examples include correcting “NULL” to “None,” removing incorrect blank space, and filling rare symbols in unknown-token positions.Source: §5.3.2 · pp. 9–10, Figure 10 · p. 10

5.3.3 Impact of the continual learning module.

  • After new C/C++ training, Finetune-APR omitted “endIndex < 0” from a previously correct Java patch, while CIRCLE retained it.Both models had generated the correct patch when first given the Java training set.Source: §5.3.3 · p. 10, Figure 11 · p. 10

5.4 Case Study

  • In two BITCOUNT examples, CIRCLE generated correct patches despite Java and Python variable names being replaced with the other language’s keywords.The Java variable was changed to “def” and the Python variable to “public.”Source: §5.4 · p. 10, Figure 12 · p. 11

6 RELATED WORK

Related work situates CIRCLE among established APR techniques and continual-learning methods, identifying continual-learning APR as previously unexplored.

6.1 APR

  • APR techniques include heuristic-based, constraint-based, template-based, and DL-based repair.CIRCLE is distinguished as the first work targeting multilingual repair in a lifelong-learning scenario.Source: §6.1 · p. 11

6.2 Continual Learning

  • Continual learning has been applied across NLP and computer vision tasks, but this paper identifies APR with continual learning as previously unstudied.The authors describe their work as the first to explore continually learning APR tasks.Source: §6.2 · p. 11
  • Continual-learning methods address catastrophic forgetting through rehearsal, regularization, or architectural changes; CIRCLE combines rehearsal with regularization.Architectural approaches can increase model parameters as task counts grow.Source: §6.2 · p. 11

7 CONCLUSION

CIRCLE combines prompting, a T5-based repair model, continual-learning strategies, and re-repairing for multilingual program repair. Across four languages and five benchmarks, it continually learns bug fixing and achieves state-of-the-art performance with one model.

7 CONCLUSION

  • Across four languages and five benchmarks, CIRCLE continually learns bug fixing across languages and achieves state-of-the-art performance on all benchmarks with one model.The experiments evaluate CIRCLE on four programming languages and five benchmarks.Source: §7 · p. 11
  • CIRCLE uses a prompt-based representation to recast program repair as a fill-in-the-blank task suited to T5's pre-trained task.The T5-based model forms the APR model's skeleton.Source: §7 · p. 11
  • Difficulty-based example replay and EWC-based regularization let CIRCLE update parameters as task requirements arrive incrementally.These are the framework's two lifelong-learning strategies.Source: §7 · p. 11
  • CIRCLE applies re-repairing to eliminate form errors arising in multilingual repair.Source: §7 · p. 11

Related papers

Loading 2205.10956v4…