Source-linked AI summary
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
Michihiro Yasunaga, Percy Liang
TL;DR
Program repair from diagnostic feedback must reason across source code and compiler messages despite limited labeled data. DrRepair uses program-feedback graphs and self-supervised pre-training on corrupted unlabeled programs, outperforming prior work on DeepFix and SPoC.
Problem
Program repair must connect broken source code with diagnostic feedback, while manually curated labeled repair pairs do not scale.
Method
DrRepair uses program-feedback graphs for cross-source-feedback reasoning and self-supervised pre-training on repair examples generated by corrupting unlabeled programs.
Results
DrRepair outperforms prior work on both tasks, achieving 68.2% full repair on DeepFix and 48.4% synthesis success on SPoC.
Takeaways & Limitations
The paper finds program-feedback graphs useful for multi-line repairs and self-supervised data effective for overcoming scarce labeled repair data.
Takeaways & Limitations
The self-supervised programs share the SPoC source but not exactly the DeepFix source, limiting direct source alignment across tasks.
Abstract
from arXiv · showhide
We consider the problem of learning to repair programs from diagnostic feedback (e.g., compiler error messages). Program repair is challenging for two reasons: First, it requires reasoning and tracking symbols across source code and diagnostic feedback. Second, labeled datasets available for program repair are relatively small. In this work, we propose novel solutions to these two challenges. First, we introduce a program-feedback graph, which connects symbols relevant to program repair in source code and diagnostic feedback, and then apply a graph neural network on top to model the reasoning process. Second, we present a self-supervised learning paradigm for program repair that leverages unlabeled programs available online to create a large amount of extra program repair examples, which we use to pre-train our models. We evaluate our proposed approach on two applications: correcting introductory programming assignments (DeepFix dataset) and correcting the outputs of program synthesis (SPoC dataset). Our final system, DrRepair, significantly outperforms prior work, achieving 68.2% full repair rate on DeepFix (+22.9% over the prior best), and 48.4% synthesis success rate on SPoC (+3.7% over the prior best).
1. Introduction
DrRepair addresses program repair from compiler diagnostics with program-feedback graphs and self-supervised pre-training on corrupted unlabeled programs. Evaluations on DeepFix and SPoC show strong repair and synthesis results.
- Motivation and approach: DrRepair connects broken source code with compiler diagnostics to localize an erroneous line and generate its repair.The approach targets compiler errors such as unresolved symbols, missing delimiters, and type errors.
- Program-feedback graph: Program-feedback graphs model dependencies among relevant symbols across source code and diagnostic feedback.The graph is designed to capture long-range dependencies that support program-repair reasoning.
- Self-supervised learning: Self-supervised pre-training corrupts working online programs to generate broken/fixed training pairs, then fine-tunes models on target tasks.The extra data is approximately 10 times the size of the original datasets and covers diverse errors.
- Evaluation: DrRepair is evaluated on correcting introductory C assignments from DeepFix and synthesized C++ programs from SPoC.SPoC involves translating pseudocode into C++ implementations, whose candidate programs often fail to compile.
- Results: 68.2% full repair on DeepFix improves 22.9% absolute over the prior best, while SPoC reaches 48.4% synthesis success.These are reported as significant improvements over prior work.
2. Problem statement
The task takes a broken program and compiler feedback as input, then localizes the line requiring repair and generates a fixed version of that line. The reported compiler line may differ from the line that must be repaired.
- Inputs: The system receives a broken program x with L lines and compiler feedback f containing a reported line number and error message.The feedback is represented as f = (ierr, merr), where merr is a token sequence.
- Outputs: The task identifies an erroneous line k and generates its repaired version yk while leaving other program lines unchanged.The fixed program satisfies yi = xi for i ≠ k.
- Error localization: The compiler-reported line ierr does not necessarily match the line k that requires repair.The example reports line 9 while the fifth line is repaired from a char declaration to a string declaration.
3. Approach
DrRepair combines a program-feedback graph with graph attention to connect source-code and diagnostic symbols, then uses self-supervised pre-training from corrupted working programs. Its architecture encodes, propagates, recontextualizes, localizes, and repairs code lines.
- Program-feedback graph: DrRepair models program repair with a program-feedback graph that connects identical symbols across diagnostic feedback and source code.The graph includes diagnostic arguments, their source-code occurrences, and remaining identifiers; undirected edges form symbol-specific cliques.
- Model architecture: The encoder takes program and compiler feedback as input, and the decoder predicts the erroneous line index and its repaired code.The training example includes a broken program, feedback, erroneous line index, and repaired line; localization and repair are learned jointly.
- Model architecture: Initial encoding represents code and compiler-message tokens with bidirectional LSTMs and injects the reported-line offset into code-token representations.The line offset is Δi=ierr−i and is incorporated through positional encoding and a feedforward network.
- Model architecture: Graph attention propagates information across long-range symbol connections, while sequence networks handle local context before and after graph propagation.The model uses initial encoding, graph attention, and recontextualization stages, with LSTMs supporting sequence-level representations.
- Self-supervised learning: Self-supervised pre-training converts unlabeled working programs into repair examples by corrupting compilable code, then fine-tunes the model on labeled in-domain data.The corruption procedure uses perturbation modules designed around common compiler-error categories, including syntax, identifier-type, identifier-typo, and keyword changes.
- Self-supervised learning: DrPerturb is reported as significantly more effective than baseline corruption procedures such as randomly deleting tokens.The proposed perturbation modules are intended to generate diverse compiler-error examples for self-supervised data.
4. Experiments
The experiments evaluate DrRepair for program repair and synthesis on DeepFix and SPoC, including diagnostic-feedback, graph, pre-training, and search-method comparisons. Results show improvements across full repair, single-step repair, and synthesis success, with graph structure and self-supervision contributing complementary gains.
- Experimental setup: The evaluation covers two applications: repairing student C programs in DeepFix and repairing synthesized C++ programs in SPoC.DeepFix contains introductory-programming submissions, while SPoC evaluates correction of candidate programs generated from pseudocode.
- Experimental setup: SPoC training data substitutes sampled program lines with candidate code pieces that produce compiler errors, creating broken-program repair examples.The resulting examples are used as original train/dev data, alongside separate extra pre-training data.
- DeepFix results: 68.2% full repair rate: DrRepair improves over the prior best on the DeepFix raw test set by 22.9% absolute.The full system combines compiler messages, the program-feedback graph, and self-supervised pre-training; full repair is evaluated after iterative repair on programs that may contain multiple errors.
- DeepFix results: The graph adds 3% full-repair improvement over the base model, while self-supervised pre-training adds a further 2%.The reported gains indicate complementary improvements from the two components.
- SPoC results: On SPoC, graph and pre-training improve single-step repair by 4.4% and 3.2%, while adding pseudocode reaches 68% single repair accuracy on the dev set.The synthesis method equipped with DrRepair also improves over best-first search by 6% on TestP/TestW at budget 100.
- Ablations and interpretation: Graph representations are especially helpful for compiler errors requiring multi-line analysis, including undeclared identifiers and member-access errors.The ablation reports that code-only and code-feedback edges are equally important, with the final graph most effective.
5. Related work and discussion
Prior program-repair work spans graph-based source-code modeling, self-supervised pre-training, sequence-based repair, and specialized bug repair. This paper differs by combining a program-feedback graph with diagnostic-feedback-conditioned self-supervision.
- Graph neural networks: Graph neural networks have been applied to program graphs and source-code structure for tasks such as variable-name prediction.Prior work augments abstract syntax trees with data-flow edges and uses GNNs to model code structure.
- Self-supervised pre-training: Unlike typical self-supervised pre-training, the pre-training task directly matches program repair and is conditioned on diagnostic feedback.Unlabeled programs are corrupted and paired with evaluator feedback to synthesize repair examples, which the paper reports improves test-time generalization.
- Learning program repair: Existing introductory-assignment repair systems primarily use sequence models, reinforcement learning, or improved sampling methods.DeepFix uses sequence-to-sequence translation, while RLAssist and SampleFix extend the approach with reinforcement learning or sampling.
- Learning program repair: This paper uses a graph representation spanning source code and diagnostic feedback to capture long-range symbol dependencies.The approach targets dependencies across code and compiler feedback that sequence-only methods do not explicitly model.
- Learning program repair: Related work also learns repair patterns from labeled edit or error-resolution records and targets specific bug classes or programming languages.Examples include AST-based repair, hierarchical clustering of repair patterns, variable misuse, name-based bugs, and JavaScript bugs.
6. Conclusion
The paper contributes a program-feedback graph and a self-supervised procedure for generating repair examples from unlabeled programs and compiler feedback. It reports that these contributions help model multi-line repairs and address scarce labeled data, while suggesting broader feedback-based editing applications.
- Conclusion: The program-feedback graph models reasoning across symbols in program repair, particularly when repairs require analyzing multiple code lines.It connects source-code and diagnostic-feedback information for repair reasoning.
- Conclusion: Self-supervision creates extra repair examples by corrupting unlabeled programs and obtaining compiler feedback, helping overcome labeled-data scarcity.The procedure uses an evaluator to produce feedback for synthetic broken-program examples.
- Conclusion: The framework may extend feedback-based editing beyond programs to essays and interactive dialogue.The paper contrasts high-bandwidth diagnostic feedback with single-number rewards such as compile success.
Reproducibility
The paper provides code and data through a GitHub repository and lists experiment materials on CodaLab.
- Reproducibility: Code and data are available in the DrRepair GitHub repository.The paper supplies the repository URL for implementation and data access.
- Reproducibility: Experiments are available through a CodaLab worksheet.The supplied link identifies the experiment-access location.