PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Self-Repair a Silver Bullet for Code Generation?

Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, Armando Solar-Lezama

arXiv:2306.09896v5cs.CLcs.AIcs.PLcs.SE

TL;DR

Self-repair aims to improve LLM code generation, but its effectiveness is difficult to assess when additional model calls increase computational cost. This paper evaluates staged self-repair for Code Llama, GPT-3.5, and GPT-4 on HumanEval and APPS, finding modest, variable cost-adjusted gains while stronger or human feedback substantially improves repair.

  • Problem

    Existing self-repair studies provide limited evidence about efficacy across settings, especially regarding the model’s ability to diagnose its own generated code under a fixed compute budget.

  • Method

    The paper evaluates self-repair for Code Llama, GPT-3.5, and GPT-4 on self-contained Python tasks from HumanEval and APPS, comparing repair trees with equivalent-cost i.i.d. sampling and replacing feedback sources.

  • Results

    Cost-adjusted self-repair gains are often modest, variable, or absent, whereas stronger feedback consistently improves repair and human feedback raises GPT-4 success from 33.3% to 52.6%.

  • Takeaways & Limitations

    Self-repair benefits depend strongly on feedback quality and sufficient diversity among initial programs rather than extensive repair alone.

  • Takeaways & Limitations

    The experiments use self-contained Python tasks with executable unit tests, unlike real-world development with incomplete specifications, long dependencies, and limited tests.

Abstract

from arXiv · show

Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance in these settings. However, despite its increasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair is taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, a small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.

1 INTRODUCTION

Self-repair augments code generation with model-produced debugging feedback, but its benefits must be evaluated against additional sampling cost. The paper finds that gains are often modest or absent, while stronger feedback substantially improves repair.

  • Motivation and workflow: Self-repair samples code, tests it, generates textual feedback about failures, and uses that feedback to produce repairs.The workflow can conceptually separate code generation, execution, feedback generation, and repair, although feedback and repair may occur in one interaction.
  • Motivation and workflow: Additional model invocations increase computational cost, making self-repair worthwhile only when it outperforms equivalent-cost i.i.d. sampling.The comparison asks whether repair offers a greater chance of success than drawing and testing more independent code samples.
  • Findings: Self-repair is not a silver bullet: after repair cost is counted, i.i.d. sampling often matches or exceeds its pass rates, especially under small budgets.The authors report that outcomes vary across tasks and conjecture that generation and repair rates jointly influence which strategy wins.
  • Findings: For GPT-4 on APPS, allocating more budget to diverse initial programs yields 1.05× pass@20, whereas extensive repair yields 0.97× baseline pass@22.The first configuration draws 10 initial samples and one repair candidate each; the second draws 2 initial samples and 10 repair candidates each.
  • Findings: Artificially stronger feedback improves self-repair across tested model substitutions, and human feedback raises GPT-4 repaired-program success from 33.3% to 52.6%.The human-feedback improvement is a 1.58× increase in the fraction of repaired programs passing tests.

2 RELATED WORK

Prior work has studied program synthesis, code repair, and LLM self-repair, but these literatures emphasize different settings and evaluation choices. This paper situates its analysis at their intersection.

  • Program synthesis with large language models: Program-synthesis research has extensively evaluated LLM-generated programs using raw accuracy or pass@k.The cited literature includes work on program synthesis with large language models and commonly uses pass@k as an evaluation metric.
  • Program synthesis with large language models: Unlike some prior work, this paper does not distinguish public filtering tests from private correctness tests because its method does not filter outputs.The distinction is discussed in relation to earlier synthesis approaches.
  • Code repair: Statistical and learning-based code repair has mainly targeted human-written software, with newer work applying repair after automatically synthesized code.The related literature spans programming languages and machine learning communities, including domain-specific-language synthesis.
  • Contemporary work on LLM self-repair: Contemporary LLM self-repair research spans code generation and other domains, including approaches with and without natural-language feedback.The paper identifies recent work on APPS and Codex as especially close while referring to a broader survey for coverage.

3 METHODOLOGY

The methodology models self-repair as a staged process that expands sampled programs into feedback and repair candidates. Performance is compared with equivalent-cost independent sampling using repair-tree estimates.

  • Self-repair overview: Self-repair comprises code generation, code execution, feedback generation, and code repair.The framework formally defines these four stages.
  • Code generation: Given a specification ψ, the programming model MP first generates np independent program samples.These samples form the initial program branches of the repair process.
  • Code execution: The programs are executed against tests; passing any test suite stops the process, while failures produce error messages for unsuccessful programs.Errors contain compile or runtime information or an example input with an incorrect output.
  • Feedback generation: A feedback model generates nf explanations for each wrong program because execution errors usually provide little repair signal.The explicit feedback stage can be ablated to study its contribution separately.
  • Code repair and repair trees: For each initial program and feedback string, the programming model samples nr repaired-program candidates, forming a repair tree rooted at ψ.The tree branches from the specification to initial programs, feedback strings, and repairs.
  • Model configuration: The framework permits separate programming and feedback models, while joint feedback-and-repair sampling is used when MP = MF.The same model is used for initial generation and repair because the authors view them as fundamentally similar tasks.
  • Evaluation: The evaluation compares repair trees containing np + npnfr programs with a baseline of the same number of i.i.d. samples.This pass@k adaptation accounts for programs sampled during both initial generation and repair.
  • Evaluation: Large repair trees are bootstrapped by subsampling a single pre-generated tree, reducing experimental cost but risking statistical artefacts.The analysis bounds sampled counts below the large-tree counts and reports small standard deviations across settings.

4 EXPERIMENTS

The experiments compare self-repair with i.i.d. sampling under equivalent budgets across HumanEval and APPS, varying models, datasets, and repair settings. Self-repair is inconsistent overall, but benefits from diverse initial programs and stronger feedback, with human feedback improving GPT-4 repair further.

  • Experimental setup: The study evaluates self-repair against equivalent-budget i.i.d. sampling for CodeLlama-13b-instruct, GPT-3.5, and GPT-4 on HumanEval and APPS.The experiments vary initial programs, feedback, and repair candidates to test when self-repair is effective.
  • Self-repair is not a silver bullet: Up to 8% improvement over the baseline occurs for GPT-4 on APPS, while GPT-3.5 reaches up to 34% on competition-level problems and CodeLlama up to 10% on HumanEval.GPT-3.5’s HumanEval gains are limited to up to 3% as performance approaches the ceiling.
  • Self-repair is not a silver bullet: Increasing initial-program diversity consistently improves relative performance, whereas increasing repair candidates often yields marginal gains or reduces performance at lower budgets.The results suggest that diverse initial samples are more important because they increase the chance of obtaining a program close enough to the ideal for successful repair.
  • Self-repair is not a silver bullet: Self-repair is not consistently superior: gains vary across datasets and tasks, and are often marginal or absent at smaller budgets.On APPS, only GPT-4 significantly benefits in the nfr = 1 setting, while CodeLlama and GPT-3.5 mostly match or lag their baselines.
  • Human feedback significantly improves GPT-4 repair: Human feedback is more effective partly because it is more accurate, suggests higher-level changes when needed, and expresses uncertainty instead of confidently giving inaccurate feedback.GPT-4 feedback is inaccurate in 32/80 cases versus 7/80 for human feedback, while participants express uncertainty in 7/80 cases and GPT-4 never does.

5 LIMITATIONS

The study’s limitations concern computational shortcuts, task scope, and the unmeasured time cost of human debugging.

  • Bootstrapping repair trees from one large tree reduces computational cost but risks introducing statistical artefacts.The authors bounded key hyperparameters and note small standard deviations to reduce this risk.
  • The experiments cover self-contained Python tasks with executable unit tests, unlike real-world development with incomplete specifications and long dependencies.The paper leaves the role of self-repair in those broader settings for future work.
  • The human-feedback study did not track participants’ debugging time, so it evaluates feedback quality and repair impact but not time cost.

6 CONCLUSION

Across HumanEval and APPS, self-repair often yields modest, variable gains after accounting for cost, while stronger feedback substantially improves repair.

  • Self-repair gains are often modest, vary within and between datasets, and depend on sufficient diversity among initial programs.
  • Replacing weaker models’ feedback with stronger models’ feedback significantly improves self-repair performance.
  • The results suggest that current models are limited by their inability to produce reliably accurate and useful feedback about why code is wrong.

A ALTERNATIVE EVALUATION STRATEGIES FOR SELF-REPAIR

The appendix introduces evaluation strategies that measure self-repair against both success and token cost, addressing limitations of sample-count comparisons.

  • The main evaluation treats one repair tree as equivalent to k = np + np*nfr baseline samples.
  • This adapted pass@k comparison omits feedback tokens, which may overstate self-repair’s benefits.
  • Pass@t compares pass rate with the number of sampled program and feedback tokens required to achieve it.It therefore captures both success and model work under non-uniform sampling costs.
  • The metric defines performance as expected pass rate at the token count expected for a chosen set of hyperparameters.

A.1 BATCHED PASS@T

Batched pass@t evaluates self-repair while accounting for token usage, and broadly reproduces the paper’s main trends across APPS and HumanEval.

  • BATCHED PASS@T: Batched pass@t samples all initial programs in parallel, then all repairs if no initial program passes, recording tokens at each point.
  • BATCHED PASS@T: The plots jointly vary pass rate and sampled tokens, with better models approaching the top-left point (0.0, 1.0).
  • BATCHED PASS@T: The same broad trends appear: gains for GPT-4 on APPS and for Code Llama and GPT-3.5 on HumanEval.
  • BATCHED PASS@T: Performance is typically better when np > nfr, except for GPT-3.5 on HumanEval, where it remains relatively stable near ceiling.
  • BATCHED PASS@T: In sequential evaluation, self-repair appears less beneficial, especially when the baseline pass rate is already high.

B ADDITIONAL RESULTS: GPT-4 ON HUMANEVAL, CODE LLAMA ON APPS

This appendix section presents the full Code Llama APPS and GPT-4 HumanEval results, which were omitted from Section 4.1 for brevity.

  • The section reports complete Code Llama results on APPS.
  • The section reports complete GPT-4 results on HumanEval.
  • These results were omitted from Section 4.1 for brevity.

C ADDITIONAL RESULTS: SELF-REPAIR VS. PROBLEM DIFFICULTY

APPS difficulty affects self-repair differently across models: GPT-3.5 and GPT-4 appear to benefit more on harder problems, whereas Code Llama benefits less. Repair success rates are reported separately, with caution advised when interpreting their specific values.

  • Self-repair versus problem difficulty: GPT-3.5 and GPT-4 appear to benefit more from self-repair as APPS problems become harder.APPS is divided into introductory, interview, and competition categories.
  • Self-repair versus problem difficulty: Code Llama benefits less from self-repair across increasing APPS difficulty.
  • Repair success rates: Repair success rates measure the fraction of repaired programs that pass the tests on APPS.Table 2 defines repair success rate as passing repairs divided by total sampled repairs.
  • Repair success rates: The specific repair-success numbers should not receive too much weight because initial programs may differ in repair difficulty across models.
  • Human-study setup: The human study recruited 16 participants and assigned them 40 failing GPT-4 programs across 20 APPS tasks.The task distribution was skewed toward introductory problems: 14 introductory, 3 interview, and 3 competition.
  • Human-study setup: Participants received instructions through a slide deck that included an example task.

E HUMAN EXPERIMENT (QUANTITATIVE ANALYSIS): RESULTS PER TASK

The quantitative human experiment reports repair outcomes per task, tracking how many repair candidates passed all unit tests under GPT-4 and human feedback.

  • Per-task results: Each program is associated with four feedback pieces: two sampled from GPT-4 and two provided by human participants.
  • Per-task results: Each table cell counts repair candidates, out of 25, that passed all unit tests.
  • Per-task results: The appendix provides a complete breakdown of the quantitative results presented in Section 4.3.

F HUMAN EXPERIMENT (QUALITATIVE ANALYSIS): EXAMPLES

The qualitative examples compare human and GPT-4 feedback on incorrect competitive-programming solutions. They illustrate feedback ranging from concrete minor edits to recommendations requiring major algorithmic changes, alongside a case where GPT-4 never expresses uncertainty in the study.

  • Example categories: The qualitative appendix pairs human and GPT-4 examples when possible, including specifications, incorrect programs, errors, feedback, and classification comments.
  • Election example: In the election example, the feedback incorrectly proposes changing a strict majority test from > to >=, while the comment identifies a different issue.
  • Card-partition example: In the card-partition example, feedback notices the constraint that both participants must take at least one card but incorrectly recommends negative infinity initialization.The comment explains that this would prevent the loop from updating the result.
  • Square example: For the square-vertex example, swapping the signs in the coordinate calculations is classified as a minor change because the program structure remains unchanged.
  • Day-of-week example: For the day-of-week example, removing the modulo operator is classified as a relatively small but explicit change.
  • Uncertainty: Across the 80 examples studied, GPT-4 never expresses uncertainty, according to the qualitative analysis.

G PROMPTS

The appendix describes prompts for generating, critiquing, and repairing Python programs on APPS call-based and standard-input tasks. It also illustrates these prompts with numerical-palindrome examples and expected repairs.

  • G PROMPTS: CodeLlama uses the same GPT prompts with [PYTHON] delimiters instead of Python code fences because this improves code extraction reliability.The prompt appendix covers initial generation, feedback-only, and repair formats.
  • G.1 APPS: The experiments distinguish call-based APPS tasks, which return function outputs, from stdio-based tasks, which read stdin and write stdout.Separate prompts are used for initial code generation in the two formats.
  • G.1 APPS: Feedback prompts supply the specification, incorrect program, and error message, while repair prompts additionally supply feedback and request only fixed code.Joint feedback-repair sampling combines these stages in the experimental setup.
  • G.1 APPS: The numerical-palindrome prompt asks for sorted, duplicate-free palindromic substrings, while excluding single digits and numbers beginning or ending with zero.Invalid inputs return “Not valid,” and inputs without valid palindromes return “No palindromes found.”
  • G.1 APPS: The corrected palindrome implementation validates that num is a nonnegative integer, scans substrings, tests palindromicity, removes duplicates through a set, and sorts numeric results.The appendix also lists related numerical-palindrome kata and a standard-input listing caption.
  • G.1 APPS: The stdio prompt requires an expert Python programmer to return only a code-delimited program that matches the specification and passes all tests.The prompt example uses a binary-words problem with input, output, and reversal requirements.
  • G.1 APPS: The binary-words task asks for the minimum number of reversals needed to order all unique words according to adjacent-character game rules.Its input contains multiple test cases and binary words, with stated bounds on counts and total lengths; the output reports the minimum k and indices to reverse, or -1.

H APPS TASKS USED FOR OUR EVALUATIONS

The evaluation tasks were randomly sampled from the APPS test set according to the full dataset’s relative difficulty frequencies. The resulting task list is reported for reproducibility.

  • H APPS TASKS USED FOR OUR EVALUATIONS: APPS evaluation tasks were randomly sampled from the test set using the full dataset’s relative difficulty frequencies.This sampling was intended to avoid distribution shift.

I MORE EXAMPLES OF FEEDBACK FROM GPT-4 AND OUR HUMAN PARTICIPANTS

The appendix provides additional GPT-4 and human feedback examples across travel, train, citation, and geometry tasks. Their repair outcomes vary substantially across feedback sources and errors.

  • I MORE EXAMPLES OF FEEDBACK FROM GPT-4 AND OUR HUMAN PARTICIPANTS: The appendix listings pair each task with its specification, incorrect program, error message, GPT-4 feedback, participant feedback, and resulting repair counts.Five additional examples are provided so readers can compare feedback quality and repair outcomes.
  • I MORE EXAMPLES OF FEEDBACK FROM GPT-4 AND OUR HUMAN PARTICIPANTS: The Old Peykan specification requires minimum travel time across sequential roads, with fuel consumed during travel and supplies transferred immediately at cities.The vehicle starts at c_1 with s_1 liters, has unlimited tank capacity, and cannot continue if fuel runs out between cities.
  • I MORE EXAMPLES OF FEEDBACK FROM GPT-4 AND OUR HUMAN PARTICIPANTS: In the Old Peykan task, GPT-4 feedback identifies incorrect handling of travel time and fuel waiting, producing 7/25 and 10/25 passing repairs.The two feedback variants focus on different aspects of the minimum-time calculation.
  • I MORE EXAMPLES OF FEEDBACK FROM GPT-4 AND OUR HUMAN PARTICIPANTS: The train task concerns booking the earliest departure from station 1 that qualifies for compensation under a delay of at least 30 minutes.Inputs describe scheduled trains, stations, planned times, and delays.
  • I MORE EXAMPLES OF FEEDBACK FROM GPT-4 AND OUR HUMAN PARTICIPANTS: For the train-delay task, GPT-4 feedback receives 3/25 passing repairs, while another correction receives 0/25 and addresses tracking the earliest valid journey.A further feedback example recommends comparing delayed and undelayed arrival paths and correcting numeric sorting and departure-time handling.
  • I MORE EXAMPLES OF FEEDBACK FROM GPT-4 AND OUR HUMAN PARTICIPANTS: Other examples report 25/25 passing repairs for correcting geometry sign or ordering errors and for replacing an incorrect citation formula.The cited examples include both successful and unsuccessful repair feedback outcomes.
Loading 2306.09896v5…