PaperlayerView exact revision on arXiv

Source-linked AI summary

Break-It-Fix-It: Unsupervised Learning for Program Repair

Michihiro Yasunaga, Percy Liang

arXiv:2106.06600v2cs.LGcs.CLcs.SE

TL;DR

The paper addresses the difficulty of learning code repair from unlabeled data when synthetic errors do not match real bad inputs. BIFI uses a critic and a learned breaker to iteratively create paired training data, improving repair accuracy over initial fixers on GitHub-Python and DeepFix.

  • Problem

    Paired broken-and-fixed code is costly to label, while fixers trained on synthetic corruptions face a mismatch with real bad inputs.

    Source: §1 · p. 1

  • Method

    BIFI uses a critic to retain successful fixes of real bad inputs and trains a breaker to generate realistic errors, iteratively updating the breaker and fixer with the resulting paired data.

    Source: §1 · p. 2

  • Results

    90.5% repair accuracy on GitHub-Python (+28.5% absolute) and 71.7% repair accuracy on DeepFix (+5.6% absolute), outperforming the initial fixers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    BIFI turns raw unlabeled data into usable paired data with a critic, and the paper presents it as a potentially general framework for repair tasks.

    Source: §6 · p. 9

    Ensuring that fixes preserve semantics is nontrivial, so the approach relies on edit distance as a constraint.

    Source: §3 · p. 2

Abstract

from arXiv · show

We consider repair tasks: given a critic (e.g., compiler) that assesses the quality of an input, the goal is to train a fixer that converts a bad example (e.g., code with syntax errors) into a good one (e.g., code with no syntax errors). Existing works create training data consisting of (bad, good) pairs by corrupting good examples using heuristics (e.g., dropping tokens). However, fixers trained on this synthetically-generated data do not extrapolate well to the real distribution of bad inputs. To bridge this gap, we propose a new training approach, Break-It-Fix-It (BIFI), which has two key ideas: (i) we use the critic to check a fixer's output on real bad inputs and add good (fixed) outputs to the training data, and (ii) we train a breaker to generate realistic bad code from good code. Based on these ideas, we iteratively update the breaker and the fixer while using them in conjunction to generate more paired data. We evaluate BIFI on two code repair datasets: GitHub-Python, a new dataset we introduce where the goal is to repair Python code with AST parse errors; and DeepFix, where the goal is to repair C code with compiler errors. BIFI outperforms existing methods, obtaining 90.5% repair accuracy on GitHub-Python (+28.5%) and 71.7% on DeepFix (+5.6%). Notably, BIFI does not require any labeled data; we hope it will be a strong starting point for unsupervised learning of various repair tasks.

1. Introduction

BIFI addresses the mismatch between synthetic and real coding errors by using a critic and a learned breaker to iteratively train a fixer on more realistic paired data. It improves repair accuracy on GitHub-Python and DeepFix over their initial fixers.

1. Introduction

  • Learning a fixer from unlabeled data is motivated by the cost of collecting paired broken and fixed code, while code analyzers and compilers can assess whether inputs contain errors.The paper also frames critic-to-fixer learning as an important machine-learning problem and mentions molecular design and essay editing as related settings.Source: §1 · p. 1
  • Synthetic perturbations can miss the structure of real errors, such as missing parentheses in nested contexts or coordinated indentation changes.Random perturbations may generate indentation mistakes requiring paired indent and dedent tokens only with very small probability.Source: §1 · p. 2, Figure 2 · p. 2
  • BIFI uses the critic to retain successful fixes of real bad examples, trains a breaker to generate realistic errors, and iteratively trains both models on the resulting paired data.The cycle combines real bad inputs with breaker-generated examples to move training toward realistic error distributions.Source: §1 · p. 2
  • Unlike backtranslation, BIFI uses the critic to verify generated examples and trains the fixer on real bad examples as well as breaker-generated ones.The paper states that these differences improve the correctness and distributional match of generated paired data.Source: §1 · p. 2
  • Evaluation covers GitHub-Python, which repairs Python AST parse errors from a 3M-snippet dataset, and DeepFix, which repairs compiler errors in student C code.The initial GitHub-Python fixer uses random perturbations; DeepFix starts from DrRepair.Source: §1 · p. 2
  • 90.5% repair accuracy on GitHub-Python (+28.5% absolute) and 71.7% repair accuracy on DeepFix (+5.6% absolute), outperforming the initial fixers.The paper reports a new state-of-the-art on both datasets.Source: §1 · p. 2

2. Problem statement

The problem is to learn a fixer from unlabeled code and a critic that distinguishes erroneous from error-free inputs. The fixer should turn bad inputs into nearby good ones, and is evaluated by repair accuracy on held-out bad examples.

2. Problem statement

  • The critic c classifies code snippets as bad when they have errors and good when they have none.The corresponding sets are Dbad and Dgood.Source: §2 · p. 2
  • The fixer f maps a bad example to a nearby output that the critic judges good.The target output must satisfy c(f(x))=1 while remaining close to the input.Source: §2 · p. 2
  • Repair accuracy is measured on a held-out set of bad examples.Source: §2 · p. 2

3. Approach

BIFI addresses the mismatch between synthetic and real code errors by iteratively training a fixer and breaker on critic-verified examples. It starts from synthetic pairs, incorporates real bad inputs, and adds breaker-generated errors to fixer training.

3. Approach

  • The authors note that automatically ensuring semantics preservation is nontrivial, so they rely on an edit-distance constraint.The constraint is described in §4.1.Source: §3 · p. 2
  • Synthetic perturbations can differ from real errors, including how parentheses are omitted and how multiple indentation tokens are changed.The authors report that this distribution mismatch results in low performance.Source: §3 · p. 3

3.1. Initialization

  • Initialization creates paired data by corrupting good examples with a predefined procedure, then trains the initial fixer and breaker on paired examples.Corruption can use random noising or heuristic noising designed to mimic common programming errors.Source: §3.1 · pp. 3–4

3.2. Break-It-Fix-It (BIFI)

  • BIFI iteratively uses critic-verified fixer outputs on real bad code to train a breaker, then uses verified breaker outputs and real examples to retrain the fixer.The cycle is intended to adapt both models toward the distribution of real examples.Source: §3.2 · p. 4

3.3. Comparison with Backtranslation

  • Unlike backtranslation, BIFI uses a critic to verify generated examples and trains the fixer on real bad examples as well as breaker-generated examples.The authors state that these additions improve the correctness and distributional match of training data.Source: §3.3 · p. 4

3.4. Version of BIFI without breaker: FixerOnly

  • FixerOnly trains on critic-verified fixer outputs from real bad examples but omits breaker-generated examples.The authors describe it as especially useful when many real bad examples are available, with smaller gains than BIFI when they are scarce.Source: §3.4 · p. 4

4. Experiments

The paper evaluates BIFI on DeepFix and a larger GitHub-Python dataset collected by the authors.

  • BIFI is evaluated on DeepFix and a larger GitHub-Python dataset collected in this paper.Source: §4 · p. 4

4.1. Dataset and setup

The experiments use GitHub-Python for Python AST parse errors and DeepFix for C compiler errors. They describe each dataset, its evaluation setup, and the corresponding initial fixer.

4.1. Dataset and setup

  • The GitHub-Python and DeepFix tables report test-set repair accuracy, and their captions state that BIFI outperforms the respective initial fixer.DeepFix uses DrRepair as its initial fixer; GitHub-Python uses a synthetic-data-trained initial fixer.Source: Table 1 · p. 5, Table 2 · p. 5

4.1.1. Github-Python

  • GitHub-Python contains 3M collected Python snippets; its AST-parser task evaluates repair accuracy on 15K held-out bad snippets, requiring no parse errors and edit distance under five tokens.The dataset has 38K bad snippets, of which 23K are available to BIFI after the test holdout.Source: §4.1.1 · p. 5

4.1.2. DeepFix

  • DeepFix contains 37K good and 7K bad student C snippets; repair is successful when output has no compiler errors, with 20% of bad examples held out for testing.DrRepair is the initial fixer, and the fixer is applied repeatedly at test time, up to five times.Source: §4.1.2 · p. 5

4.2. Main results

BIFI improves repair accuracy on both GitHub-Python and DeepFix. Examples show the breaker generating errors seen in real bad code and the fixer learning to correct an indentation error after one round.

4.2. Main results

  • BIFI reports 90.5% accuracy on real bad code across all error categories; on DeepFix, using DrRepair as the initial fixer, it reaches 71.7%, 5.6% absolute above DrRepair.Source: §4.2 · p. 6
  • Continuing training on synthetic perturbations alone did not improve the initial fixer, suggesting a performance ceiling for synthetic-only training.Source: §4.2 · p. 6
  • The round-1 breaker assigns high probability to real-code errors, including obsolete raise usage and unbalanced parentheses in nested contexts.Source: Figure 4 · p. 6
  • After one BIFI round, the fixer inserts and deletes the correct pair of indent tokens, correcting an error the initial fixer mishandled.The initial fixer's failure is attributed to the mismatch between real errors and synthetic perturbations used in its training.Source: Figure 5 · p. 6
  • BIFI improves repair accuracy without additional manual effort.Source: §4.2 · p. 6

4.3. Analysis

The analysis examines the contributions of real bad examples and breaker-generated examples, compares BIFI with backtranslation, and studies how the fixer adapts to real error distributions.

4.3.1. Effect of real bad examples

  • Even with Bad 10%, FixerOnly outperforms synthetic-only training, with repair accuracy of 78.5% versus 62.7%.FixerOnly trains on real bad examples but not breaker-generated examples; its improvement over the initial fixer is reported as +27% on GitHub-Python.Source: §4.3.1 · pp. 6–7

4.3.2. Effect of bad examples generated by breaker

  • BIFI consistently improves over FixerOnly, especially when real bad examples are scarce, and its breaker generates errors seen in real code.Examples include obsolete Python 3 raise usage and unbalanced parentheses in nested contexts; the learned breaker improves training-data coverage and efficiency compared with random perturbations.Source: §4.3.2 · p. 7

4.3.3. Comparison with backtranslation

  • +10% absolute on GitHub-Python: BIFI outperforms backtranslation, and removing either critic verification or real-bad training hurts performance.The comparison identifies both components as improving the quality of training data.Source: §4.3.3 · p. 7

4.3.4. How does the fixer adapt?

  • BIFI improves repair of error patterns that synthetic perturbations poorly represent, including indentation errors, which rise from 39% to 85%.The initial fixer also has lower accuracy on nested than non-nested unbalanced parentheses and on multi-line than single-line missing commas; performance catches up in rounds 1 and 2.Source: §4.3.4 · p. 7, Table 5 · p. 7

5. Related work and discussion

BIFI connects code repair to prior work on synthetic-data repair, denoising autoencoding, domain adaptation, data augmentation, self-training, and backtranslation, while using a critic to verify outputs and adapt training to real errors.

  • Unlike prior repair methods that train on randomly or heuristically perturbed code, BIFI adapts a synthetic-data fixer to real errors without manual, domain-specific effort.Source: §5 · p. 8
  • BIFI turns a denoising autoencoder—its initial fixer trained on random perturbations—into a fixer for real-world bad examples.Source: §5 · p. 8
  • Unlike typical domain adaptation, BIFI uses a critic to verify outputs on unlabeled real bad code and trains a breaker to simulate that test-domain data.Source: §5 · p. 8
  • Unlike data augmentation focused on adding examples or self-training that accepts predicted targets, BIFI augments both sides and keeps predictions verified by the critic.Source: §5 · p. 8
  • Like backtranslation, BIFI uses models to generate training examples, but it verifies examples with a critic and trains on real bad inputs; the authors suggest applying it to unsupervised MT and style transfer with a critic.Source: §5 · pp. 8–9

6. Conclusion

The paper presents BIFI as a critic-guided way to build realistic, correct paired data from unlabeled data and improve repair performance on two code-repair datasets.

  • BIFI uses a breaker and critic to build realistic, correct paired training data, improving baseline fixers on GitHub-Python and DeepFix.Source: §6 · p. 9
Loading 2106.06600v2…