Source-linked AI summary
Self-Edit: Fault-Aware Code Editor for Code Generation
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, Zhi Jin
TL;DR
Limited sample budgets leave LLMs with poor accuracy on competitive-programming code generation. Self-Edit generates programs, executes them on example tests, and uses the resulting comments to guide a fault-aware editor. Across benchmarks and nine LLMs, it improves pass@1 and is reported to outperform reranking methods with greater efficiency.
Problem
LLMs often have low accuracy on competitive-programming tasks, especially when limited to one generated program per task.
Method
Self-Edit executes LLM-generated programs on example tests, wraps execution results as comments, and uses a fault-aware editor to revise the code.
Results
Self-Edit improves average pass@1 by 89% on APPS-dev, 31% on APPS-test, and 48% on HumanEval across nine LLMs.
Takeaways & Limitations
Self-Edit provides a constant-budget editing-based alternative that outperforms reranking methods, especially when sample budgets are limited.
Takeaways & Limitations
The method is evaluated with relatively small editor datasets, including only 10 sampled programs per problem, and lacks strict computing-resource comparisons with other post-editing methods.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated an impressive ability to generate codes on competitive programming tasks. However, with limited sample numbers, LLMs still suffer from poor accuracy. Inspired by the process of human programming, we propose a generate-and-edit approach named Self-Edit that utilizes execution results of the generated code from LLMs to improve the code quality on the competitive programming task. We execute the generated code on the example test case provided in the question and wrap execution results into a supplementary comment. Utilizing this comment as guidance, our fault-aware code editor is employed to correct errors in the generated code. We perform extensive evaluations across two competitive programming datasets with nine different LLMs. Compared to directly generating from LLMs, our approach can improve the average of pass@1 by 89\% on APPS-dev, 31\% on APPS-test, and 48\% on HumanEval over nine popular code generation LLMs with parameter sizes ranging from 110M to 175B. Compared to other post-processing methods, our method demonstrates superior accuracy and efficiency.
1 Introduction
Self-Edit addresses low single-sample accuracy in competitive-programming code generation by using execution feedback to edit LLM-generated programs. Evaluations across datasets and models report improved pass@1 and advantages over reranking-based post-processing.
- Motivation: 7% accuracy is achieved by GPT3 on APPS-test when it submits only one program per task.This illustrates the low pass@1 accuracy motivating execution-guided refinement.
- Approach: Self-Edit follows generation, execution on example tests, and editing with execution results wrapped as supplementary comments.The approach is inspired by programmers who execute initial code and refine it using outputs or compile/runtime errors.
- Results: 89% and 31% average pass@1 improvements are reported on APPS-dev and APPS-test, respectively, across nine LLMs.The evaluation covers two public benchmarks and models ranging from 110M to 175B parameters.
- Results: 48% average pass@1 improvement is reported on HumanEval, indicating transfer to a different dataset style.The result is reported as evidence of transfer ability on an out-of-distribution benchmark.
- Comparison: Self-Edit maintains a constant sample budget and outperforms reranking-based methods, especially under limited budgets such as pass@1.The comparison emphasizes lower computational overhead alongside direct program modification.
2 Related Work
Related work includes code-generation and post-processing methods, while Self-Edit targets the computational cost of large-scale sampling through a constant-budget editing pipeline.
- Code Generation: Competitive programming requires models to generate complete solutions from problem descriptions and example input-output pairs.The task is presented as a challenging form of code generation.
- Post-processing: Existing code-generation post-processing methods filter, cluster, or rerank generated programs using execution outputs or learned rankers.These approaches include sampling-based filtering and fault-aware reranking.
- Limitations of Prior Methods: 1 million samples per problem and 105 TPU-seconds are required by AlphaCode, illustrating the cost of large-scale sampling.The passage characterizes these resource demands as impractical when computing resources are limited.
- Self-Edit: Self-Edit addresses large-scale sampling by maintaining a constant sample budget and improving computational efficiency.Its pipeline generates programs, obtains execution-based comments from example tests, and edits programs before evaluation.
3 Methodology
Self-Edit builds a generate-execute-edit pipeline that uses example-test execution results as supplementary comments for fault-aware code refinement. The method trains an editor to produce higher-quality programs while emphasizing likely correct outputs and maintaining a small inference sample budget.
- Self-Edit Pipeline: Self-Edit generates code, executes it on example tests, wraps the results into supplementary comments, and uses a fault-aware editor to refine the program.The pipeline uses the problem description, generated code, and supplementary comment as editor inputs.
- Executor and Supplementary Comments: Execution results are classified as Passed, Wrong Answer, or Error, enabling templates to provide situation-specific feedback to the editor.Wrong-answer comments include the input, expected output, and actual output, while error comments wrap relevant error information.
- Inference: At inference, the editor refines generated programs after example-test execution, retaining a small sample budget compared with large-scale sampling and reranking.The generator is treated as a fixed-parameter black box, supporting use across models ranging from 110M to 175B parameters.
- Training Data: The supplementary-comment datasets pair problem descriptions, generated programs, and comments, with ground-truth programs collected for editor training.For each problem, ten programs are sampled and executed on the example test case before constructing the training data.
- Training Objective: The editor addresses a one-of-many task by using a GOLD variation whose likelihood weighting favors easier-to-learn correct solutions and copying from existing code.This objective prioritizes obtaining at least one correct output within a limited attempt budget rather than modeling all solutions equally.
4 Experiment
Experiments evaluate Self-Edit across two benchmarks, nine LLMs, multiple difficulty levels, and post-processing comparisons. The editor consistently improves code-generation performance, with gains in accuracy, sample efficiency, and computational efficiency.
- Experiment Setup: Experiments span APPS and HumanEval, using nine code-generation LLMs ranging from 110M to 175B parameters.APPS-train is used for finetuning, while APPS-dev, APPS-test, and HumanEval provide evaluation settings.
- Comparison with Base LLMs: 89%: average APPS-dev pass@1 increases from 6.17% to 11.67% across nine models with the fault-aware editor.The editor improves all evaluated code-generation models despite differences in size and training settings.
- Comparison with Base LLMs: 0.12% to 0.7%: APPS-test pass@1 absolute improvements correspond to solving 6 to 35 additional problems.The editors maintain significant improvements across models on the more challenging APPS-test benchmark.
- Comparison with Base LLMs: Editing can be more sample-efficient than generation: edit-pass@1 sometimes outperforms pass@5.A smaller LLM with an editor can also match or exceed larger models; GPT-Neo-125M with a 110M editor can outperform GPT-Neo-1.3B and sometimes GPT-J-6B.
- Difficulty-Level Analysis: The editor improves GPT-J-6B-finetuned performance across all APPS-test difficulty levels, with relatively larger gains on Introductory problems.On APPS-test, it can additionally correct hundreds of generated codes at sol@10.
- Post-processing Baselines: Self-Edit outperforms CodeRanker on APPS-test pass@1, improving 0.14% to 0.68% versus CodeRanker’s 0.14% to 0.3%.The comparison uses GPT-Neo-1.3B-finetuned under matched experimental settings; Self-Edit solves 19 more problems.
- Supplementary-Comment Ablation: The supplementary comments are crucial: removing them significantly lowers performance, and produces no improvement on APPS-test.The ablation compares the original editor with a modified editor without comments.
- Edit-Round Ablation: Two-round editing slightly increases APPS-dev pass@1 but hurts APPS-test performance.The authors attribute this to a training–test mismatch in which the editor edits its own output during the second round.
5 Discussion
The discussion examines resource costs and an in-context-learning variant of Self-Edit. The variant improves performance across benchmarks but still requires many LLM calls, while the trained editor is substantially faster than reranking.
- Efficiency: Self-Edit requires nearly 8.4s to obtain one sample, less than 0.01s for execution, and 3.7s for editing on the evaluated setup.The measurements use text-davinci-002 on APPS-test with a Tesla T4 GPU.
- Efficiency: CodeRanker requires more than 110s to obtain candidate lists plus 0.53s for reranking, whereas Self-Edit uses fewer LLM calls and less total time.The comparison is made on the same APPS-test demonstration setup.
- In-Context Learning: The in-context-learning variant uses text-davinci-002 as both base model and editor, prompting self-editing with supplementary comments without additional training.The constructed comments are supplied as input prompts for zero-shot inference.
- In-Context Learning: The in-context-learning variant achieves significantly better performance than smaller editors across APPS-test and HumanEval, but still incurs relatively many LLM calls.Resource optimization remains an open concern for this variant.
6 Conclusion
Self-Edit is a generate-and-edit framework that uses execution results to guide a fault-aware editor. Evaluations show that it significantly improves the quality of LLM-generated code for competitive programming.
- Conclusion: Self-Edit uses generated-code execution results to improve code quality through a fault-aware code editor.The approach is presented as a generate-and-edit method for competitive programming.
- Conclusion: In-depth evaluations demonstrate significant improvements in the quality of LLM-generated output code.The conclusion summarizes the reported effectiveness of the generate-and-edit approach.
Limitations
The paper identifies limitations involving the editor models, editor datasets, resource comparisons, and multi-round editing. These constraints motivate further evaluation and development.
- The editor uses relatively small pretrained models, leaving performance across different editor architectures for future study.The authors describe their evaluations as preliminary for this design space.
- The constructed editor datasets are relatively small because each problem uses only 10 sampled programs from the LLM.The authors propose analyzing dataset-size effects and alternative construction methods.
- The paper does not provide a strict computing-resource comparison with other post-editing methods.It compares against a reranking baseline using a similarly sized additional model, but not with strict resource accounting.
- The second editing round shows a training–test gap because the existing implementation was not designed for multi-round editing.The authors plan to try a specially designed editor model.
A Compared with CodeRanker
The comparison with CodeRanker uses matched base models, training data, and benchmarks to make the post-processing comparison fair.
- The study compares Self-Edit with CodeRanker as a post-processing baseline.
- For fair comparison, both methods use the same base model, training dataset, and test benchmark.The reproduced models are GPT-Neo-125M-finetuned, GPT-Neo-1.3B-finetuned, and GPT-J-6B-finetuned.
- The authors reproduce the finetuned base models to align their base-model results with CodeRanker’s reported results.
B Qualitative analysis of Code Editor
The qualitative case studies show edited programs retaining substantial structure from the original LLM outputs while incorporating execution-guided editing across APPS and HumanEval examples.
- Case studies: The editor concatenates the GPT3 output, problem description, and supplementary comment before generating the final edited code.The case studies cover APPS-dev and APPS-test examples using GPT3.
- Case studies: The edited code is similar to the GPT3 output, with its first few lines exactly matching the original output in the reported cases.The authors present this as evidence of the LLM output’s impact on the edited code.
- APPS-test example: The APPS-test case study concerns sorting numbers by frequency, with earlier input appearance breaking ties.The problem description identifies this as question APPS-test-4854.
- HumanEval example: A separate HumanEval case study is presented using the CodeGen-2B model.