Source-linked AI summary
Automated Repair of Programs from Large Language Models
Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, Shin Hwei Tan
TL;DR
The paper studies whether automated repair can improve incorrect programs generated by language models. It evaluates a workflow combining Codex-generated solutions, test-based repair tools, and held-out tests, documenting recurring generation mistakes and repair outcomes.
Problem
Codex has low passing rates on programming tasks, while limited understanding of task descriptions and program semantics contributes to incorrect generated programs.
Method
The study builds LMDefects from 113 LeetCode tasks and applies test-based repair tools to incorrect Codex solutions, validating patches on public and held-out tests.
Results
Codex-generated solutions repeatedly contain similar code blocks and irrelevant helper functions, while Recoder produces two uniquely fixed solutions in the reported comparison.
Takeaways & Limitations
The study provides a dataset and a systematic evaluation setting for examining automated repair of buggy programs produced by language models.
Takeaways & Limitations
The experimental findings may not generalize beyond the studied configurations or beyond Java programming languages.
Abstract
from arXiv · showhide
Large language models such as Codex, have shown the capability to produce code for many programming tasks. However, the success rate of existing models is low, especially for complex programming tasks. One of the reasons is that language models lack awareness of program semantics, resulting in incorrect programs, or even programs which do not compile. In this paper, we systematically study whether automated program repair (APR) techniques can fix the incorrect solutions produced by language models in LeetCode contests. The goal is to study whether APR techniques can enhance reliability in the code produced by large language models. Our study revealed that: (1) automatically generated code shares common programming mistakes with human-crafted solutions, indicating APR techniques may have potential to fix auto-generated code; (2) given bug location information provided by a statistical fault localization approach, the newly released Codex edit mode, which supports editing code, is similar to or better than existing Java repair tools TBar and Recoder in fixing incorrect solutions. By analyzing the experimental results generated by these tools, we provide several suggestions: (1) enhancing APR tools to surpass limitations in patch space (e.g., introducing more flexible fault localization) is desirable; (2) as large language models can derive more fix patterns by training on more data, future APR tools could shift focus from adding more fix patterns to synthesis/semantics based approaches, (3) combination of language models with APR to curate patch ingredients, is worth studying.
I. INTRODUCTION
Large language models generate code for many programming tasks, but their success remains low on complex problems because they lack deep understanding of task descriptions and program semantics. This paper studies whether automated program repair can improve such code, evaluates Codex edit mode alongside existing repair tools, and proposes directions for combining language models with APR.
- Motivation: Language models’ low success rate is attributed to limited understanding of task descriptions and program semantics.Sequence-to-sequence token transformation does not capture deep semantic features needed for complex programming tasks.
- Automated Program Repair: APR repairs buggy programs by changing them to satisfy a correctness specification, but existing tools typically generate only small patches.Semantic-based tools use symbolic execution, while search-based tools search predefined patch spaces; multi-line fixes face semantic complexity and search-space explosion.
- Research Questions: The study asks whether APR can improve language-model-generated code, which mistakes are common, and whether Codex edit mode can fix program bugs.It evaluates existing tools TBar and Recoder on Codex-generated code and studies Codex edit mode as a repair tool.
- Implications: The study identifies limitations in existing APR patch space, fault locations, and patch size, and suggests combining APR with Codex-e to curate patch ingredients.It specifically motivates more flexible fault localization and collaboration between APR tools and Codex-e for complex patches.
- Contributions: The authors introduce LMDefects, a dataset of 113 Java programming tasks, including 46 solved and 67 unsolved by Codex.The paper presents a systematic study and the first evaluation of Codex edit mode as an automated repair tool.
II. STUDY SETTING
The study evaluates whether automated repair can fix Codex-generated LeetCode solutions using public tests for repair and private tests for validation. It combines Codex generation, test-based APR tools, and the LMDefects dataset.
- Methodology and Dataset: The workflow generates Codex solutions, tests them publicly, repairs unsolved solutions, and validates patches on both public and private tests.The study uses public tests to guide repair and held-out LeetCode tests to assess correctness.
- Methodology and Dataset: LMDefects contains 113 LeetCode programming tasks used to study defects in language-model-generated solutions.LeetCode tasks range from easy to hard and include natural-language descriptions with public input-output examples.
- Prompt and Parameters: Prompts combine task descriptions with code snippets, and the evaluation uses zero-shot prompting while reserving public tests for APR guidance.For each task, 50 candidate solutions are generated and the five highest-probability candidates are selected.
- APR Tools: TBar and Recoder are evaluated as representative Java APR tools, using public tests to guide repairs and private tests to validate patched solutions.The tools run with default settings, a 15-minute timeout, and stop when a patch passes all public tests.
III. RQ1: WHAT MISTAKES ARE COMMON IN AUTO-GENERATED CODE?
The analysis examines defects in Codex-generated solutions and finds overlap with human programming mistakes, alongside syntax, algorithm, and repetition-related failures. These patterns motivate applying APR techniques to auto-generated code.
- Study of Generated Defects: Codex solved 46 programming tasks, while 335 incorrect solutions from 67 unsolved tasks were analyzed for defect patterns.A solution was incorrect if it failed public or private tests; a task was unsolved when all five generated solutions were incorrect.
- Defect Classification: The defect classification follows Codeflaws categories and compares auto-generated mistakes with defects in human competition submissions.Two annotators constructed and cross-validated minimal patches accepted by LeetCode to identify solution differences.
- Comparison with Human Defects: Auto-generated and human-written solutions share single-hunk and multi-hunk defects, including operator and variable mutations.The overlap suggests Codex makes programming mistakes similar to those made by human participants.
- Syntax Errors: Syntax errors commonly involve incomplete code or references to undefined variables, functions, or classes.Even with the maximum 2048-token generation limit, incomplete programs remained common, with average output length of 628 tokens.
- Misaligned Algorithm: Codex can select algorithms or data structures based on misleading names, producing misaligned solutions such as dynamic programming for an unsuitable task.Variable names such as “dp,” “pq,” and “q” can indicate the wrongly chosen underlying algorithm or data structure.
- Similar Code Blocks: When Codex struggles, it tends to repeatedly generate similar code blocks with minor variations in names, statements, or control structures.Figure 3 illustrates duplicated blocks differing only in variable names.
3 Irrelevant Helper Functions: Although
Codex-generated solutions can contain irrelevant helper functions, a recurring negative symptom in the studied code.
- Codex-generated solutions are likely to include redundant helper functions irrelevant to the prompt.
IV. RQ2: HOW EFFECTIVE ARE APR TOOLS IN FIXING THE CODE PRODUCED BY CODEX?
APR tools can repair some incorrect Codex solutions, but their effectiveness is constrained by patch space, multi-line edits, and fault localization that ignores dependencies.
- Plausible patches pass public tests, whereas correct patches pass both public and private tests and are accepted by LeetCode.
- 16 and 22 plausible patches from TBar on easy- and medium-level tasks yielded only 6 and 3 correct patches, respectively.
- Recoder generated 6 and 5 correct patches on easy- and medium-level tasks, compared with TBar's 6 and 3.
- APR tools together helped Codex solve 4 additional easy-level and 5 additional medium-level tasks.
- Existing APR tools remain limited in generating complex patches requiring edits to multiple lines.
- TBar fails when its patch space lacks the needed literal or when it lacks an insert-statement pattern.
- For multi-line fixes, TBar and Recoder separately localize suspicious lines without considering dependencies among them, preventing correct patches in such tasks.
- Pattern-based and learning-based APR face limited search space, inability to generate multi-edit patches, and lack of program-dependency awareness.
V. RQ3: CAN CODEX EDIT MODE FIX PROGRAM BUGS?
Codex edit mode uses natural-language instructions to revise existing programs, and its effectiveness depends on how fault locations are specified.
- Codex edit mode takes a program and a natural-language instruction as input and outputs an edited program.
- Codex-ebug asks Codex-e to fix a bug without providing a fault location.
- Codex-eline instructs Codex-e to fix a specified line, while Codex-estm identifies the suspicious statement by its program text.
- For each incorrect solution, Codex-e examines the ten most suspicious statements and generates five possible edits for each.
- Codex-ebug produced 15 correct patches, including four multi-hunk patches, while Codex-eline fixed nine single-hunk and two multi-hunk solutions.
- Codex-e can generate patches at flexible locations, including moving a clause out of a loop and changing its condition.
- Codex-estm's effectiveness with a specific fault location was nearly comparable to Codex-ebug's effectiveness without location guidance.
A. Comparison between TBar, Recoder and Codex-estm.
Codex-estm and Recoder produce overlapping but distinct patches, while TBar's patches are largely subsumed by the union of the other tools' patches.
- A. Comparison between TBar, Recoder and Codex-estm.: Codex-estm was selected for comparison because experiments identified it as the best overall Codex-e strategy.
- A. Comparison between TBar, Recoder and Codex-estm.: TBar's patches form a proper subset of Codex-estm ∪ Recoder and are almost subsumed by Recoder.
- A. Comparison between TBar, Recoder and Codex-estm.: Codex-estm and Recoder share 8 patches, while Codex-estm has 6 unique patches and Recoder has 2.
- A. Comparison between TBar, Recoder and Codex-estm.: Codex-estm can produce complex patches at flexible locations and learn more fix patterns from a much larger training dataset.
- A. Comparison between TBar, Recoder and Codex-estm.: Recoder's syntax-guided decoder can copy and insert an AST subtree, enabling two patches unique to Recoder despite its smaller training dataset.
B. Combine Patch Space of Different Tools.
Combining patch ingredients from APR tools, Codex-e, and multiple Codex solutions can expand the search space for correct repairs beyond any single tool.
- Combine Patch Space of Different Tools: TBar and Codex-e can separately produce complementary patch ingredients even when neither tool alone generates the complete correct fix.In one example, TBar changed n>0 to n!=0, while Codex-e added a nums.size()>0 bound check after receiving the exception in its instruction.
- Combine Patch Space of Different Tools: “TBar+Codex” produced all required patch ingredients for 12 incorrect solutions, making it the most effective evaluated combination.The combination borrows ingredients from TBar patches and multiple candidate solutions generated by Codex.
- Combine Patch Space of Different Tools: APR tools may lack large, unseen code snippets, while Codex candidates can enrich the patch space with code borrowed from alternative solutions.Figure 11 describes a required for-loop that existing APR tools and Codex-e did not generate, but another Codex candidate supplied relevant code.
- Combine Patch Space of Different Tools: Codex-e and Codex’s multiple solutions provide patch ingredients that are absent when incorrect solutions are repaired using only APR techniques.These ingredients can be extracted and combined to identify the components needed for more correct fixes.
VI. IMPLICATIONS AND DISCUSSIONS
The study identifies implications and suggestions for both language-model and automated program-repair research.
- VI. IMPLICATIONS AND DISCUSSIONS: The paper presents implications and suggestions for language models and program repair research.The supplied passage introduces the discussion without specifying individual recommendations.
A. Open dataset for language model defects.
The paper introduces LMDefects as a community-oriented dataset for studying defects in language-model-generated programs and supporting future improvements.
- A. Open dataset for language model defects: Auto-generated Codex programs exhibited wrong-algorithm names, repeated code blocks, and irrelevant function helpers.These symptoms were identified through manual analysis of generated solutions.
- A. Open dataset for language model defects: Even after bracket-mismatch syntax errors were manually fixed, the generated programs still failed held-out LeetCode tests.The observation indicates that syntax correction did not resolve the remaining incorrectness in these solutions.
- A. Open dataset for language model defects: Codex appears to rely heavily on function names, which can fail when a customized algorithm is required.The paper relates this behavior to the relative concision and searchability of function names compared with full prompts.
D. Pattern-based APR versus learning-based APR.
Pattern-based APR tools have complementary strengths, while Codex-e’s repair behavior depends strongly on the supplied edit instruction and fault-localization context.
- D. Pattern-based APR versus learning-based APR: Recoder generated a few more correct fixes than TBar because pattern-based APR depends on available fix patterns and the search space for ingredients.Recoder uniquely fixed a case requiring insertion of the statement steps++; TBar uniquely fixed other cases.
- D. Pattern-based APR versus learning-based APR: TBar succeeded on a relational-operator change from < to != that Recoder failed to apply, despite Recoder using operator mutation elsewhere.The result indicates that a learned repair pattern is not guaranteed to be correctly applied to every program.
- D. Pattern-based APR versus learning-based APR: The study found that existing APR techniques fixed only a small number of bugs in Codex-generated programs.The paper nevertheless reports that public LeetCode tests can guide repair and fault-localization information can guide Codex-e.
- D. Pattern-based APR versus learning-based APR: Unspecified edit targets let Codex-e search across the program, encouraging larger patches but potentially reducing precision for single-line fixes.Statement-focused instructions instead provide code context derived from fault localization.
- D. Pattern-based APR versus learning-based APR: The findings may not generalize beyond the studied configurations, Java, or the Codex model and Codex edit mode.The authors note that repair-tool and Codex performance may vary across settings and programming languages.
- D. Pattern-based APR versus learning-based APR: Automated scripts may contain bugs that affect the reported results.The authors plan to make the scripts available upon acceptance as mitigation.
VIII. RELATED WORK
This section situates the study at the intersection of automated program repair and language-model code generation, then presents its findings and research directions. It highlights Codex edit mode, fault localization, and language-model-generated patch ingredients as links between the two areas.
- Automated Program Repair: APR tools take buggy programs and correctness criteria as inputs, then generate patches through predefined transformations and search.The related-work passage distinguishes search-based, semantic-based, and learning-based APR approaches.
- Large Language Model for Code Generation: Language-model code-generation research includes Codex, AlphaCode, CodeWhisperer, and other platforms evaluated on datasets such as APPS.These systems extend transformer-based language modeling into automatic code generation.
- This Study: The study examines mistakes in language-model-generated programs and whether APR tools can repair the resulting buggy code.Its findings compare auto-generated defects with human defect categories and evaluate repair tools on generated programs.
- This Study: Codex-generated programs share defect categories with human programs, while TBar and Recoder perform poorly on auto-generated bugs.With fault-localization information, Codex edit mode shows promising code-editing results and outperforms TBar and Recoder.
- Research Directions: The authors suggest using software-engineering artifacts such as fault locations to improve language-model-generated code.The recommendation targets higher-quality code generation through additional repair-relevant information.
- Research Directions: Future APR research could make fault localization more flexible and extract patch ingredients from Codex-generated solution sets.These directions are inspired by the complementary strengths of language models and program-repair systems.