Self-Edit: Fault-Aware Code Editor for Code Generation

1 Introduction

• GPT3 achieves only 7% pass@1 on APPS-test, motivating execution-guided editing because example outputs and errors can expose bugs in generated code.The motivation follows human programmers’ practice of testing an initial program and refining it based on results.Source: §1 · p. 1
• Self-Edit generates code with an LLM, executes it on example tests, then supplies execution results as comments to a fault-aware editor for refinement.The approach is presented as a generate-and-edit method for improving LLM code quality and accuracy.Source: §1 · pp. 1–2
• Average pass@1 improves by 89% on APPS-dev, 31% on APPS-test, and 48% on HumanEval; GPT3-175B rises from 26.6% to 32.4% on APPS-dev.The experiments cover two benchmarks and nine LLMs with parameter sizes from 110M to 175B.Source: §1 · p. 2
• Self-Edit maintains a constant sample budget and reduces LLM computational overhead, while outperforming reranking-based post-processing, especially at pass@1.The paper describes Self-Edit as the first editing-based post-processing method for competitive programming tasks.Source: §1 · p. 2

2.1 Code Generation

• Code generation maps requirements such as natural-language descriptions, pseudocode, old code, or tool responses into source code, and competitive programming is a particularly challenging task.The paper cites AlphaCode and GPT-3 as demonstrating strong capabilities and says it uses text-davinci-002 and competitive code-generation models.Source: §2.1 · p. 2

2.2 Post-processing of LLMs for code generation

• Existing post-processing reranks or filters generated solutions using trained rankers, execution outputs, self-generated tests, or back-translation probabilities.Examples include ranking math solutions (Cobbe et al., 2021; Shen et al., 2021), execution-based clustering and filtering (Li et al., 2022c; Shi et al., 2022), fault-aware ranking (Inala et al., 2022), test-driven agreement (Chen et al., 2022), and back-translation probability (Zhang et al., 2022).Source: §2.2 · pp. 2–3
• Prior methods require large-scale sampling; AlphaCode uses 1 million samples per problem at a cost of 105 TPU-seconds, while Self-Edit maintains a constant sample budget.The paper presents this budget constraint as a practical limitation and says Self-Edit improves computational efficiency.Source: §2.2 · p. 3

3 Methodology

• Self-Edit first generates code, executes the example test case, and uses the resulting comment to guide a fault-aware editor in refining the code.The editor uses the problem description, generated code, and supplementary comment as its inputs.Source: §3 · p. 3

3.1 LLMs as Black-box Generator

• Nine black-box LLM generators, kept at fixed parameters, span sizes from 110M to 175B, making the approach usable when training or model access is restricted.The paper motivates this design by the cost of training LLMs and restrictions such as paid API access.Source: §3.1 · p. 3

3.2 Executor and Supplementary Comments

• PyCodeGPT-110M more often produced SyntaxErrors, while GPT3-175B had fewer syntax and runtime errors but more normally executed cases on APPS-train.The executor classifies results as passed, wrong answer, or error; comments report a pass, include input and expected and actual outputs for wrong answers, or give error location and message.Source: §3.2 · pp. 3–4, Figure 3 · p. 3, Figure 4 · p. 4

3.3 Fault-aware Code Editor

• The editor maps the problem description, LLM-generated program, and supplementary comment to refined code intended to implement the description and pass test cases.The task is formulated as sequence-to-sequence generation.Source: §3.3 · p. 4, §3.3.1 · p. 4

3.3.1 Code Editor Models

• At inference, each generated program is tested and edited; the decoder-only editor is fine-tuned from PyCodeGPT-110M using segmented inputs, preserving a small sample budget.The edited program becomes the final version for evaluation.Source: §3.3.1 · p. 4

3.3.2 Dataset Construction for Code Editor

• For each LLM, the editor dataset uses 10 sampled programs per APPS-train problem, execution-derived comments, and up to 15 ground-truth targets per program.Each model-specific dataset contains nearly 4.5k programs with comments because generated models produce different comment distributions.Source: §3.3.2 · p. 4

3.3.3 Training Objective of Code Editor

• The editor uses a GOLD variation with off-policy importance weighting to prioritize high-likelihood correct solutions, matching the one-of-many editing task and limited attempts.The weighting also enables copying parts of existing generated programs into better outputs.Source: §3.3.3 · pp. 4–5

4.1 Experiment Setup

• Experiments use APPS and HumanEval, fine-tune models on APPS-train, and evaluate functional correctness with hidden tests using pass@k for k = {1, 5, 10}.The study evaluates nine code-generation LLMs spanning four orders of magnitude in parameter count; its editor is fine-tuned PyCodeGPT-110M, and inference uses a sample budget of 10 with one edited final code per LLM output.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Comparison with Base LLMs

• APPS-dev average pass@1 rises from 6.17% to 11.67% across nine models, an 89% improvement with fault-aware editing.The editor also improves all evaluated models; for GPT-3 (175B), the reported gains are 5.9%, 5.0%, and 8.4% on pass@{1,5,10}, respectively.Source: §4.2 · p. 6, Table 2 · p. 7
• On APPS-test, editing improves pass@1 by 0.12%–0.7%, solving 6–35 more problems, and can correct hundreds of codes at sol@10.The test set is more challenging than APPS-dev, but the editor maintains improvements across model sizes; the passage reports the sol@10 corrections without specifying an exact count.Source: §4.2 · p. 6, Table 3 · p. 7
• On HumanEval, the editor improves all evaluated models on all metrics and can correct out-of-distribution code outputs.Only outputs that fail the example test case are edited; at larger sample budgets, CodeGen-2B's pass@10 does not increase, while its sol@10 increases significantly.Source: §4.2 · pp. 6–7, Table 4 · p. 8
• Editing GPT-J-6B-finetuned improves pass rates at all APPS-test difficulty levels, with relatively larger gains on Introductory problems and slight Competition gains as the sample budget grows from 1 to 10.The authors attribute limited correction of very difficult problems to the poor quality of the base LLM outputs.Source: §4.2 · p. 6, Table 5 · p. 8

4.3 Comparison with Post-processing Baseline

• On APPS-test, GPT-Neo-1.3B-finetuned pass@1 rises from 0.14% to 0.68% with the editor, versus 0.14% to 0.3% with CodeRanker.The editor solves 19 more problems; unlike CodeRanker, which requires 100 outputs per problem, Self-Edit uses k = {1, 5} outputs and edits them into final solutions.Source: §4.3 · pp. 7–8, Table 6 · p. 8

4.4 Ablation on Supplementary Comments

• Removing supplementary comments causes a significant pass-rate decline on APPS-dev and eliminates improvement over the base model on APPS-test.The ablation indicates that error-message guidance in the supplementary comments is crucial to the editor's performance.Source: Table 7 · p. 8, §4.4 · p. 8

4.5 Ablation on the Number of Edit Rounds

• A second editing round slightly increases APPS-dev pass@1 but hurts APPS-test performance.In the second round, the editor edits its own output despite being trained to edit LLM outputs, creating a train–test gap.Source: Table 7 · p. 8, §4.5 · p. 9

5.1 Time Cost compared with Post-processing Baseline

• In the reported APPS-test demo, Self-Edit averages 8.4s per sample versus CodeRanker’s >110s for candidate lists plus 0.53s for ranking, with better performance and fewer LLM calls.The setup uses text-davinci-002 as the base model on Google Colab with a Tesla T4 GPU; the executor takes <0.01s and the editor 3.7s.Source: §5.1 · p. 9

5.2 Performances of In-Context Learning Self-Edit

• The in-context variant uses text-davinci-002 as both base and editor, prompting zero-shot edits with supplementary execution comments and requiring no additional training.The comments are constructed as described in Section 3.2.Source: §5.2 · p. 9
• The in-context Self-Edit achieves significantly better performance than smaller editors across APPS-test and HumanEval, but still incurs many LLM calls.The authors identify efficient use of in-context learning capabilities as future work.Source: Table 8 · p. 9, §5.2 · p. 9

• Self-Edit uses execution results from LLM-generated code to improve code quality on competitive programming tasks.Source: §6 · p. 9
• Its central component, the fault-aware code editor, edits and optimizes generated code.Source: §6 · p. 9
• In-depth evaluations demonstrate that Self-Edit significantly improves the quality of LLM output code.Source: §6 · p. 9

• The editor uses relatively small pretrained models, leaving performance across editor models and architectures for future study.The authors describe their evaluations as preliminary evidence for the generate-and-edit approach.Source: Limitations · p. 10
• The editor dataset is small: they sampled only 10 programs per problem, and leave dataset-size effects and alternative construction methods for future work.They note that their dataset is quite small compared with those used by existing post-editing methods.Source: Limitations · p. 10
• The authors lack a strict computing-resource comparison with other post-editing methods, despite reporting better performance than a re-ranking baseline using few LLM samples.Both approaches use an additional model with a similar parameter count; the authors cite the cost of accessing LLMs in describing their approach's efficiency.Source: Limitations · p. 10
• The implementation is not designed for multiple editing rounds and exhibits a train–test gap in the second round, motivating a specially designed editor model.The authors also identify effective strategies for interacting with advancing LLMs as an important future research area.Source: Limitations · p. 10

A Compared with CodeRanker

• With a very small number of LLM-generated samples, Self-Edit significantly exceeds CodeRanker (Inala et al., 2022).The passage reports that full results are in Table 9, but supplies no numerical values.Source: A Compared with CodeRanker · p. 13
• For a fair comparison, the study matches CodeRanker’s base models, training dataset, and test benchmark, using three fine-tuned GPT models.The models are GPT-Neo-125M-finetuned, GPT-Neo-1.3B-finetuned, and GPT-J-6B-finetuned, trained on APPS-train; their base-model performance is reported as similar to CodeRanker’s.Source: A Compared with CodeRanker · p. 13

B Qualitative analysis of Code Editor

• 19.10% of edited output on APPS-dev and APPS-test is a common prefix with GPT3 output, demonstrating the LLM output’s impact on edited code.The authors note that this figure does not account for similarities in intermediate content.Source: B Qualitative analysis of Code Editor · p. 13
• Edited code is similar to GPT3 output: its first few lines are exactly the same, and subsequent code is partially based on the original output.Figures 5 and 6 show GPT3 case studies on APPS-dev and APPS-test, respectively.Source: B Qualitative analysis of Code Editor · p. 13, Figure 5 · p. 15, Figure 6 · p. 16
• A case study in Figure 7 shows code generated with CodeGen-2B on HumanEval.The authors also provide HumanEval case studies.Source: B Qualitative analysis of Code Editor · p. 13, Figure 7 · p. 17