Source-linked AI summary
Repair Is Nearly Generation: Multilingual Program Repair with LLMs
Harshit Joshi, José Cambronero, Sumit Gulwani, Vu Le, Ivan Radicek, Gust Verbruggen
TL;DR
Small last-mile programming mistakes are difficult and existing repair systems are hard to transfer across languages. RING uses a code-trained large language model with developer-inspired prompting and evaluation across six languages. It outperforms language-specific engines in three languages, though performance is substantially lower for PowerShell.
Problem
Existing automated repair systems require substantial engineering or additional data and training to adapt to new languages, limiting multilingual repair.
Method
RING uses Codex with prompts that organize repair into fault localization, code transformation through few-shot examples, and candidate ranking.
Results
Across six languages, RING significantly outperforms language-specific engines in three, achieves competitive results in two others, and performs substantially worse in PowerShell.
Takeaways & Limitations
RING provides evidence that a single LLMC-based engine can perform multilingual repair with little engineering effort and can outperform language-specific engines in some languages.
Takeaways & Limitations
RING performs substantially worse on PowerShell, possibly because Codex contains fewer PowerShell commands and because its repairs often make fewer edits than required.
Abstract
from arXiv · showhide
Most programmers make mistakes when writing code. Some of these mistakes are small and require few edits to the original program -- a class of errors recently termed last mile mistakes. These errors break the flow for experienced developers and can stump novice programmers. Existing automated repair techniques targeting this class of errors are language-specific and do not easily carry over to new languages. Transferring symbolic approaches requires substantial engineering and neural approaches require data and retraining. We introduce RING, a multilingual repair engine powered by a large language model trained on code (LLMC) such as Codex. Such a multilingual engine enables a flipped model for programming assistance, one where the programmer writes code and the AI assistance suggests fixes, compared to traditional code suggestion technology. Taking inspiration from the way programmers manually fix bugs, we show that a prompt-based strategy that conceptualizes repair as localization, transformation, and candidate ranking, can successfully repair programs in multiple languages with minimal effort. We present the first results for such a multilingual repair engine by evaluating on 6 different languages and comparing performance to language-specific repair engines. We show that RING can outperform language-specific repair engines for three of these languages.
Introduction
RING addresses last-mile programming mistakes with a multilingual LLMC-based repair engine that combines developer-inspired repair stages. Evaluated across six languages, it competes with or outperforms language-specific systems while enabling a flipped interaction model.
- Motivation: Last-mile repairs are small edits to faulty programs whose correctness can often be checked with a parser or type checker.These mistakes can frustrate novices and slow experienced programmers.
- Research gap: Existing symbolic and neural repair systems require substantial engineering or additional data and training to adapt to new languages.RING instead uses a single LLMC-based engine for multilingual repair.
- Approach: RING divides repair into fault localization, code transformation, and candidate ranking, addressing each stage with LLMC prompts that emulate developer behavior.The system uses compiler or diagnostic messages, few-shot examples, and candidate ranking.
- Evaluation: RING was evaluated across Excel, Power Fx, Python, JavaScript, C, and PowerShell, repairing significantly more programs than a language-specific engine in three languages.It also achieved competitive results in two additional languages.
- Findings: The results provide evidence that an LLMC can support multilingual repair with performance matching or exceeding single-language methods.This supports a flipped model in which users write code and AI assistance suggests fixes.
Related Work
Prior automated repair research spans symbolic and neural techniques, but both approaches face important portability or training constraints. Code-trained large language models offer few-shot adaptation that motivates RING’s broader multilingual study.
- Automated Program Repair: Automated program repair aims to reduce the time and costs associated with finding and fixing bugs across languages and use cases.The field includes symbolic, neural, and program-synthesis approaches.
- Symbolic methods: Symbolic systems enumerate candidates using templates, heuristics, or synthesis, but are strongly tied to their domain language and restrictive in scope.They can provide strong guarantees for generated code.
- Neural methods: Neural repair models have addressed settings including student C code, Java, C, and JavaScript diagnostics, but generally require substantial training data and do not generalize across languages.They may also generate plausible but incorrect or unparseable code.
- Large Language Models: Code-trained LLMs such as Codex support zero-shot and few-shot task adaptation, although training them remains technically challenging and expensive.Earlier Codex repair work used manually provided buggy lines and fixed few-shot examples for a small set of bugs.
- Positioning: RING differs by studying prompt-building strategies for multilingual repair on larger datasets spanning more languages.Its design targets adaptation without language-specific retraining.
Approach
RING treats repair as a three-stage pipeline: localize faults, transform code using selected examples, and rank generated candidates. Its prompts and selection methods adapt these stages across languages.
- Architecture: RING’s pipeline separates fault localization, program transformation, and candidate ranking according to how developers might manually repair bugs.Each stage is implemented using an LLMC.
- Fault Localization through Language Tooling: Fault localization uses compiler or analyzer messages, normalized across languages to provide consistent diagnostic information.Messages may include line and column spans; when locations are imprecise, RING can preserve error text while removing the reported location.
- Code Transformation through Few-shot Learning: RING uses few-shot learning by selecting relevant buggy-fixed examples from an example bank according to similarity between error diagnostics.The selected examples are inserted into the prompt to guide code transformation.
- Code Transformation through Few-shot Learning: Error vector selection represents diagnostic-category frequencies and selects examples by minimizing L2 distance between error vectors.This variant assumes fine-grained error reporting is available.
- Code Transformation through Few-shot Learning: Message embedding selection compares compiler-message embeddings using a pretrained CodeBert model when tools provide detailed natural-language descriptions.This supports cases where different mistakes share the same high-level error label.
- Code Transformation through Few-shot Learning: In the running example, message embedding retrieves an example with the same error and required fix, making RING’s top candidate the correct repair.The example removes parentheses from invalid Python tuple parameter unpacking.
- Candidate Ranking: RING generates multiple candidates by sampling with temperature, then ranks them by descending average token logprobability.Development results favored diverse candidates generated at higher temperatures over generation at temperature zero.
Language-Specific Datasets
The evaluation spans six languages and uses language-specific datasets, baselines, and success criteria tailored to each repair setting. It also introduces a PowerShell benchmark and adjusts JavaScript inputs to reduce artifacts from arbitrary snippet windows.
- RING is evaluated across six languages, with each language paired with its dataset, language-specific baseline or baselines, and evaluation metric.
- Excel: Excel evaluation uses 273 repair tasks: 200 syntax or semantic-error formulas and 73 formulas containing at least one type error.
- Power Fx: Power Fx evaluation uses 200 syntactic and basic semantic-error tasks collected from help forums and anonymized product telemetry, compared with LaMirage.
- Python: Python evaluation uses 200 syntactically invalid snippets without ground-truth repairs, requiring successful parsing and fewer than 5 token edits.
- PowerShell: The PowerShell benchmark contains 208 manually validated pairs of invalid question code and valid accepted-answer code.Pairs were drawn from StackOverflow threads and validity was judged with Get-Command -syntax.
Results and Analysis
RING’s Codex-powered multilingual repair is viable across six languages, outperforming language-specific engines in several settings while revealing localization, length, selection, ranking, and language-coverage effects.
- RQ1. Viability of Multilingual Repair: RING outperforms state-of-the-art repair engines at pass@1 for Excel, Python, and C.
- RQ1. Viability of Multilingual Repair: In Power Fx, RING’s pass@3 rate is comparable to LaMirage’s pass@1 rate, with a substantial improvement over RING’s pass@1.
- RQ1. Viability of Multilingual Repair: JavaScript exposes a ranking weakness: TFix reaches pass@1 0.59 on original snippets, approximately 7 points above RING, but falls to 0.09 on extended snippets.The extended snippets average 208 T5 tokens versus 74 for original snippets.
- RQ1. Viability of Multilingual Repair: PowerShell performance is substantially lower than in other languages, possibly reflecting scarce PowerShell commands in Codex’s training data and insufficient edits.
- RQ2. Error Localization: RING correctly localizes a larger fraction of unrepaired programs than language-specific baselines across four languages with ground-truth repairs.Localization counts edits within ±k tokens of the ground-truth locations; for Power Fx, it exceeds one quarter at tolerance one token.
- RQ3. Code Transformation: Successful RING repairs tend to involve shorter buggy programs in JavaScript and Python, whereas the relationship is weaker for Excel.The paper attributes the Excel behavior to shorter program lengths and the restrictive Excel grammar.
- RQ3. Code Transformation: Smart few-shot selection improves performance in all languages by choosing examples with errors and expected edits similar to the target program.Power Fx improves least because imprecise compiler messages can add noise to example selection.
- RQ4. Candidate Ranking: RING ranks candidates by Codex’s average per-token log probabilities, whose distribution separation is clearer in better-performing languages such as Excel and C.PowerShell reverses the distribution-peak relationship, while less popular languages show lower probability peaks than JavaScript.
Discussion
RING’s adaptation to new languages centers on curated few-shot examples and language-specific diagnostic handling, while evaluation choices and scaling introduce practical constraints.
- Designing the Example Bank: Example banks should cover diverse error types, starting with public-forum scraping and expert curation of corner cases.Telemetry can add examples later, but anonymization may affect retrieval.
- Evaluation Scope: The evaluation uses strict leave-one-out example-bank construction, which can restrict available examples and potentially reduce successful repairs.
- Scaling Retrieval: Growing production example banks may make retrieval time more significant, motivating fast indexing systems such as FAISS or ANNOY.
- Adapting to New Languages: Applying RING to a new language requires building an example bank and assessing available diagnostic tooling, including error selection and message abstraction.
- Error-Based Selection: Languages with granular diagnostic categories can use error-vector selection, which retrieves examples by minimizing L2 distance between diagnostic-frequency vectors.
- Error Message Abstraction: Abstracting error messages can help localization and few-shot selection when exact location information is extra or imprecise, as observed for Excel.
- Error Message Abstraction: For C missing-semicolon diagnostics, RING removes line-location text to retain the abstract message “expected ’;’ before ’printf’.”
Conclusion
RING is a Codex-powered multilingual repair engine that uses prompt strategies for localization, transformation, and ranking across six languages. It performs well across languages, sometimes outperforming language-specific engines, with little engineering effort.
- RING uses Codex-driven prompts for error localization, code transformation, and candidate ranking across six languages, including a new PowerShell benchmark.
Fewer edits in PowerShell
For incorrect PowerShell Pass@1 repairs, Codex often makes fewer edits than the ground-truth repair requires.
- Codex makes fewer edits than required in incorrect PowerShell Pass@1 repair attempts.Figure 9 compares edit-distance differences between predicted and ground-truth programs relative to the original buggy program.
Combining RING with language-specific repair engines
Combining RING with language-specific repair engines could provide complementary repairs, especially for Power Fx and JavaScript.
- At pass@1, language-specific engines repair 34 Power Fx programs (17%) and 52 JavaScript programs (26%) that RING does not repair.
Program Normalization
The evaluation combines candidate normalization, configuration comparisons, and analyses of repair outcomes across languages. RING generally repairs shorter programs more successfully and remains competitive without a token edit-distance threshold.
- Candidate normalization: All systems normalize candidates for deduplication and ground-truth matching, while filtering generated candidates identical to the buggy code.PowerShell and PowerFx are excluded from normalization; Excel-specific processing capitalizes cell references and identifiers and removes whitespace tokens.
- Repair outcomes: RING successfully repairs shorter programs than it fails to repair for most languages with groundtruth repair available.JavaScript shows a narrowing gap as token counts increase, while Excel has no clear distinction, likely because its dataset contains shorter programs.
- Repair outcomes: Python shows a similar length trend without groundtruth repair, whereas C exhibits overlap between successfully and unsuccessfully repaired programs.These observations come from the cumulative pass@1 analysis by original buggy-program token count.
- Repair confidence: RING is more confident about correct repairs than incorrect ones in every evaluated language except PowerShell.The comparison is reported using token log-probability analyses in the paper’s figures.
- Configuration robustness: Without a token edit-distance threshold, RING still outperforms BIFI in Python and Dr Repair in C.The threshold used in the main results is four tokens; Table 7 reports results without that threshold.