PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfEvolve: A Code Evolution Framework via Large Language Models

Shuyang Jiang, Yuhao Wang, Yu Wang

arXiv:2306.02907v1cs.CLcs.SE

TL;DR

Retrieval-based code-generation augmentation is limited by retriever performance, while generating correct code in one turn remains challenging. SELFEVOLVE has an LLM generate task knowledge and refine code using interpreter feedback. The authors report improvements over strong baselines across diverse code-generation tasks.

  • Problem

    Retrieval-based augmentation depends on retrievers that can suffer from domain mismatch and return noisy or unsuitable knowledge, including semi-structured documentation.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    SELFEVOLVE prompts an LLM to generate task-specific knowledge and then refine preliminary code using interpreter feedback from executable examples.

    Source: §1 · p. 2

  • Results

    SELFEVOLVE improves execution-based measurements over strong baselines on DS-1000, outperforms baselines on HumanEval, and brings improvement on TransCoder.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The fully LLM-driven framework can be adapted to other datasets without an extra retriever or previously set up database.

    Source: §6 · p. 9

    Hand-written prompts may limit automatic adaptation across tasks, and generated knowledge may need fine-grained selection for each task.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have already revolutionized code generation, after being pretrained on publicly available code data. However, while various methods have been proposed to augment LLMs with retrieved knowledge and enhance the quality of code generation, the performance of these retrieval-based methods is limited by the strength of the retrievers used. In addition, while LLMs show great emergent ability, they still struggle to produce the correct code in one turn. To address these challenges, we propose a novel two-step pipeline, called \autoknow, that leverages LLMs as both knowledge providers and self-reflective programmers. Unlike retrieval-based methods, \autoknow~obtains the knowledge from input prompts and generates intermediate code based on the generated knowledge. After that, \autoknow~asks LLM to act as an expert programmer to perform debugging for the generated code. This is achieved by receiving the error message from the interpreter, without requiring special test cases for correctness verification. We evaluate \autoknow~on three code generation datasets, including DS-1000 for data science code, HumanEval for software engineering code, and TransCoder for C++-to-Python translation. Our empirical experiments show that \autoknow~outperforms strong baselines by a significant margin on all datasets. We also conduct exhaustive analytical experiments to validate the effectiveness of the two stages of \autoknow, and find that both are superior to other prompting-based methods. Further scalability analysis demonstrates that \autoknow~can be adapted to other more advanced models, such as GPT-4, and bring consistent efficacy improvement.

1 Introduction

Retrieval-based code-generation methods depend on retrievers that can struggle with domain mismatch and semi-structured knowledge. SELFEVOLVE uses LLM-generated task knowledge and interpreter feedback to refine code, and the authors report improvements across three code-generation tasks.

1 Introduction

  • Retrieval-based augmentation can suffer from domain mismatch, poor results for semi-structured knowledge, and limited generality across tasks.These issues may require retriever fine-tuning or training from scratch for a target domain.Source: §1 · pp. 1–2
  • SELFEVOLVE generates task-specific knowledge with an LLM, then refines preliminary code using interpreter feedback from executable examples.The examples are extracted from problem descriptions, avoiding generated test cases and evaluation-set samples.Source: §1 · p. 2
  • SELFEVOLVE improves execution-based measurements over strong baselines on DS-1000, outperforms baselines on HumanEval, and improves results on TransCoder.The authors also report more accurate generated knowledge than retrieval-based methods and adaptation to GPT-4.Source: §1 · p. 2

2 Related Work

Prior code-generation work augments prompts with context, synthetic information, or retrieved code and documentation, while refinement work uses execution feedback or generated tests. Figure 1 depicts SELFEVOLVE’s sequence of knowledge generation, trial-code generation, and interpreter-feedback refinement.

2 Related Work

  • Prompt augmentation has used notebook context, synthetic puzzles and solutions, similar code snippets, and retrieved API documentation, with retrieval limited by retriever quality and domain adaptation.The cited approaches include JuPyT5 [7], Haluptzok et al. [17], Parvez et al. [35], and DocPrompting [56].Source: §2 · p. 2
  • Figure 1 shows LLM-generated knowledge conditioning trial code, followed by iterative refinement using executable examples and interpreter feedback.The examples and generated code are combined into executable programs before the LLM receives feedback.Source: Figure 1 · p. 3
  • Code refinement work includes dedicated repair models, execution-derived traces or compiler errors, synthetic test selection, and iterative model-generated feedback.CodeT [8] uses synthetic tests, while Self-Debugging [10] uses a ground-truth test case, limiting their fit to ordinary coding scenarios.Source: §2 · p. 3

3 SELFEVOLVE: Code Evolution via Large Language Models

SELFEVOLVE generates task-relevant knowledge with an LLM, then iteratively refines code using execution feedback. Its knowledge-generation strategy adapts to whether problem intents are explicit or implicit.

3 SELFEVOLVE: Code Evolution via Large Language Models

  • SELFEVOLVE uses LLM-generated knowledge before executor-guided iterative revision, framing the stages as ordered optimizations whose second step does not deteriorate the intermediate output.The pipeline is designed to operate without external models or knowledge bases.Source: §3 · p. 3, §3.2 · p. 4, §3.1 · p. 4

3.2 SELFEVOLVE: the Proposed Two-step Pipeline

  • For implicit intents, a trial solution helps extract more precise knowledge; for explicit intents, SELFEVOLVE uses direct extraction because the LLM can do so with high accuracy.The generated knowledge can take any problem-specific structure suited to the task instruction.Source: §3.2 · p. 4
  • A Python interpreter supplies error information and standard output for revising executable programs, and refinement repeats until no exception occurs or an iteration threshold is reached.The implementation focuses on API errors and incorrect assertion statements.Source: §3.2 · p. 5
  • The authors argue that self-generated knowledge can reduce repair iterations, while interpreter-guided refinement can remove irrelevant or noisy generated knowledge.They describe the two components as reinforcing each other almost orthogonally.Source: §3.2 · p. 5

4 Experiments

SELFEVOLVE is evaluated on DS-1000, HumanEval, and TransCoder, with comparisons to strong baselines and analyses of refinement, generated knowledge, and GPT-4 scalability.

4.1 Baselines

  • The experiments use ChatGPT as SELFEVOLVE’s knowledge generator and code refiner, comparing against DocPrompting and zero-shot Self-Debugging where applicable.Evaluation covers DS-1000, HumanEval, and C++-to-Python translation on TransCoder.Source: §4.1 · p. 5, §4 · p. 5

4.2 Main Results of SELFEVOLVE

  • On DS-1000, SELFEVOLVE gains 7.8 (relatively 15.8%) pass@1 over ChatGPT on average; on HumanEval, it gains 11.59 pass@1 and 6.71 pass@10.On DS-1000 it also surpasses Self-Debugging by 4.1; on HumanEval it remains 3.95 pass@1 behind GPT-4.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 7
  • On TransCoder, SELFEVOLVE improves over ChatGPT by 2.1 computational accuracy and 2.4 pass@1.Without self-refinement, it still gains 0.7 accuracy and 0.5 pass@1 over ChatGPT.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3 Discussion

  • The first refinement step brings most gains on HumanEval and TransCoder, while DS-1000 improves almost uniformly through the third step.The authors attribute the difference to DS-1000’s greater problem difficulty and suggest more refinement for harder problems.Source: §4.3 · pp. 7–8, Figure 2 · p. 7
  • Generated knowledge is more accurate than retrieved knowledge in precision and recall across the evaluated libraries, especially where DS-1000 queries contain implicit API demands.Two data science experts assessed API-document matches to oracle solutions on 200 randomly selected problems; retrieval matched requirements slightly in Matplotlib but lagged generated knowledge.Source: Figure 2 · p. 7, §4.3 · p. 8
  • With GPT-4, SELFEVOLVE improves pass@1 over the ChatGPT-based version on Scipy (+4.72), Pytorch (+19.12), Sklearn (+6.09), and HumanEval (+2.76).Adding self-refinement further improves GPT-4 results on all evaluated datasets; DS-1000 coverage was limited to four libraries and 444 problems.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Case Study

  • The case study shows documentation conditioning yields more deterministic, general code, while traceback-guided refinement repairs an omitted AVG computation without specific test cases.The examples illustrate how generated documentation and self-refinement complement one another.Source: §4.4 · pp. 8–9, Figure 3 · p. 9

5 Limitation & Future Work

SELFEVOLVE may require task-specific prompting and fine-grained selection of generated knowledge, limiting automatic adaptation across use cases.

5 Limitation & Future Work

  • Hand-written prompting words may prevent SELFEVOLVE from operating automatically across different tasks and use cases.The authors suggest developing a more comprehensive set of prompting words to ease adaptation.Source: §5 · p. 9
  • Generated knowledge may not suit every task and may require fine-grained selection to be effective.The authors propose developing an algorithm to select appropriate knowledge automatically.Source: §5 · p. 9

6 Conclusion

SELFEVOLVE uses a single LLM as both knowledge provider and self-reflective programmer in a two-step code-generation framework, and experiments report gains across diverse tasks.

6 Conclusion

  • SELFEVOLVE uses one LLM in two steps—as a knowledge provider and self-reflective programmer—to generate code.The framework requires neither an extra retriever nor a previously set-up database.Source: §6 · p. 9
  • Experiments across diverse code-generation tasks and datasets report substantial performance gains over two prompting-based methods.The conclusion also reports that SELFEVOLVE can scale to more intelligent language models for further improvements.Source: §6 · p. 9
  • Analysis experiments indicate that SELFEVOLVE provides problem-related knowledge more suitably than traditional dense retrievers.The framework is described as flexible and extendable to datasets such as Spider and APPS.Source: §6 · p. 9

A Comparison between Generated Knowledge and Retrieved Knowledge

The appendix describes a human evaluation setup and reports the problem counts by library in the selected sample.

A Comparison between Generated Knowledge and Retrieved Knowledge

  • For the human evaluation, researchers randomly selected one-fifth of the problems in each library to form a 200-problem set; retrieval used problem descriptions as queries and retrieved k = 5 knowledge items.Source: A Comparison between Generated… · p. 14
  • Table 5 reports the number of selected problems for each library.The table is described as giving problem counts by library.Source: Table 5 · p. 14

B Prompt for the First-Stage of SELFEVOLVE in Each task

The appendix points to task-specific prompts for SELFEVOLVE’s first stage.

B Prompt for the First-Stage of SELFEVOLVE in Each task

  • The first-stage prompts for DS-1000, HumanEval, and TransCoder are shown in Figures 4, 5, and 6, respectively.The passage identifies where to find each task’s prompt.Source: B Prompt for the First-Stage of… · p. 14

C Prompt for Self-Refinement in Each task

The appendix provides task-specific self-refinement prompts for DS-1000, HumanEval, and TransCoder, shown in Figures 7, 8, and 9, respectively.

Loading 2306.02907v1…