PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering

Tal Ridnik, Dedy Kredo, Itamar Friedman

arXiv:2401.08500v1cs.LGcs.CLcs.SE

TL;DR

Code generation requires exact syntax and attention to specification details, making techniques optimized for natural-language generation potentially inadequate. AlphaCodium uses a staged, test-based flow that preprocesses problems and iteratively runs and fixes code. On CodeContests, it improves results across open- and closed-source models; GPT-4 validation pass@5 increased from 19% to 44% compared with a single well-designed direct prompt.

  • Problem

    Code generation requires searching a huge structured program space with sparse feedback, and natural-language prompting approaches may not suit its exacting requirements.

    Source: §1 · p. 1

  • Method

    AlphaCodium preprocesses problems and iteratively generates, runs, and fixes code against public and AI-generated tests.

    Source: §1 · p. 1

  • Results

    On CodeContests, AlphaCodium consistently improved results for open- and closed-source models; GPT-4 validation pass@5 rose from 19% with a single well-designed direct prompt to 44%.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors believe many of AlphaCodium’s principles and practices may apply broadly to general code generation tasks.

    Source: §6 · p. 8

    Some additional tricks and methods the authors tried did not lead to improved results.

    Source: §4 · p. 6

Abstract

from arXiv · show

Code generation problems differ from common natural language problems - they require matching the exact syntax of the target language, identifying happy paths and edge cases, paying attention to numerous small details in the problem spec, and addressing other code-specific issues and requirements. Hence, many of the optimizations and tricks that have been successful in natural language generation may not be effective for code tasks. In this work, we propose a new approach to code generation by LLMs, which we call AlphaCodium - a test-based, multi-stage, code-oriented iterative flow, that improves the performances of LLMs on code problems. We tested AlphaCodium on a challenging code generation dataset called CodeContests, which includes competitive programming problems from platforms such as Codeforces. The proposed flow consistently and significantly improves results. On the validation set, for example, GPT-4 accuracy (pass@5) increased from 19% with a single well-designed direct prompt to 44% with the AlphaCodium flow. Many of the principles and best practices acquired in this work, we believe, are broadly applicable to general code generation tasks. Full implementation is available at: https://github.com/Codium-ai/AlphaCodium

1. Introduction

Code generation requires searching a vast space of programs where small specification details matter, so natural-language prompting techniques may not transfer well. AlphaCodium combines problem preprocessing with iterative code execution and testing; on CodeContests it improves results, including GPT-4 validation pass@5 from 19% to 44%.

1. Introduction

  • Code generation involves a huge structured program space, sparse reward, and solutions whose behavior can change with a single-character edit.The authors caution that techniques optimized for natural-language generation may not work as well for code [4] [13] [10].Source: §1 · p. 1
  • AlphaCodium preprocesses the problem in natural language, then iteratively generates, runs, and fixes code against public and AI-generated tests.Its flow also generates additional data, including problem reflection and test reasoning, to support iteration.Source: §1 · p. 1
  • Generating useful additional tests is easier than generating a correct solution because test generation does not require fully solving the problem.The tests are intended to enrich the public tests and support iterative code improvement.Source: §1 · pp. 1–2
  • AlphaCodium uses code-oriented practices including YAML output, modular code, bullet-point analysis, soft decisions with double validation, exploration, and test anchors.Source: §1 · p. 2
  • GPT-4 validation pass@5 rose from 19% with a single well-designed direct prompt to 44% with AlphaCodium.The flow also outperformed prior work while using four orders of magnitude fewer LLM calls than AlphaCode.Source: §1 · p. 2
  • The authors argue that their practices may apply broadly to code generation and that harder benchmarks such as CodeContests can improve evaluation over simpler benchmarks such as HumanEval [2].Source: §1 · pp. 2–3

2. CodeContests Dataset

CodeContests is a competitive-programming dataset used to evaluate LLMs on challenging code problems with detailed specifications. The paper focuses on its validation and test sets, whose private tests assess solutions beyond the public examples.

2. CodeContests Dataset

  • CodeContests contains 10K problems for training, plus validation and test sets for assessing LLMs on challenging code problems.The problems are curated from competitive-programming platforms such as Codeforces.Source: §2 · p. 3
  • The study uses pretrained coding models on 107 validation problems and 165 test problems, evaluating solutions with private tests unavailable to the models.The authors ignore the training set because their focus is a flow applicable to pretrained LLMs rather than training a dedicated model.Source: §2 · p. 3
  • Each problem has about 200 private input-output tests, providing a check against solutions that pass public examples but fail on unseen cases.Source: §2 · p. 3
  • CodeContests descriptions are deliberately lengthy and nuanced, testing attention to small details unlike concise, easier problems such as HumanEval [2].The paper frames this complexity as closer to real-world problems involving multiple factors and considerations.Source: §2 · p. 3
  • The paper illustrates that problem reflection can make a description clearer and more coherent, which it connects to the prospect of generating correct code.Source: §2 · p. 3

3. The Proposed Flow

AlphaCodium replaces single-prompt code generation with a staged flow that reasons about the problem, generates candidate solutions and tests, then iteratively runs and fixes code. Its stages accumulate useful information, including AI-generated tests aimed at cases public tests miss.

3.1. Overview

  • Single prompts and chain-of-thought prompts did not meaningfully improve CodeContests accuracy, while generated code could pass public tests and fail on unseen private tests.The authors identify repeated execution and validation against known examples as an underused opportunity for code generation.Source: §3.1 · p. 3
  • The flow has a natural-language preprocessing phase and a code-iteration phase that generates, runs, and fixes solutions against tests.Preprocessing includes self-reflection and public-test reasoning; the public tests are augmented with AI-generated tests.Source: §3.1 · p. 3

3.2. Flow stages

  • Problem reflection summarizes the goal, inputs, outputs, rules, constraints, and other relevant details, while public-test reasoning explains each input-output example.Source: §3.2 · pp. 3–4
  • The flow generates two or three possible solutions in natural language and ranks them for correctness, simplicity, and robustness, not necessarily efficiency.Source: §3.2 · p. 4
  • It generates six to eight diverse AI tests aimed at cases not covered by public tests, including large inputs and edge cases.The flow then uses these tests during code iterations, including test anchors.Source: §3.2 · p. 4
  • The initial code is run on selected public and AI tests, then run-fix iterations continue on failing tests until they pass or a try limit is reached.The first code that passes, or has the closest output, becomes the base for later steps.Source: §3.2 · p. 4

3.3. Additional insights

  • The flow accumulates knowledge from easier steps, using outputs such as problem reflection to inform harder steps including code generation.The pre-processing outputs are intended to aid the critical code-iteration phase.Source: §3.3 · p. 5
  • Generating useful tests requires problem understanding and basic reasoning, but not a complete solution; the flow uses this to target gaps in public tests.Some conceptual stages can be combined into one LLM call through structured output, to save resources or improve concurrent task performance.Source: §3.3 · p. 5

4. Code-Oriented Design Concepts

AlphaCodium combines structured, staged reasoning with modular code generation and test-based iteration. Its design favors gradual exploration, double-checking, and test anchors to guard fixes.

Structured and semantic reasoning

  • YAML structured output and bullet-point analysis organize model responses into explicit, logical components.The YAML format is specified as equivalent to a Pydantic class; bullet points divide the problem into semantic sections.Source: §4 · p. 5

Modular code

  • Modular code with small, meaningfully named sub-functions is associated with fewer bugs and better iterative fixes than a lengthy monolithic function.The authors report that monolithic code can make it harder for the model to locate and fix problems, even when given an error message.Source: §4 · p. 5

Double validation

  • Double validation asks the model to regenerate and correct its output, which the authors found more effective than a direct yes-or-no correctness question.The example applies this second pass to AI-generated tests.Source: §4 · p. 5

Postponing decisions

  • Gradual accumulation from easier to harder tasks and ranking multiple solutions preserve room to explore instead of forcing an early choice.The highest-ranked solution gets priority, but not exclusivity, when generating initial code.Source: §4 · pp. 5–6

Staged reasoning and test generation

  • Useful-test generation requires understanding and basic reasoning, not a complete solution, and tests support the later iterative process.The flow moves from easier tasks—problem reflection and public-test reasoning—to AI-generated tests and possible solutions, then code generation and run-fix iterations.Source: §4 · p. 6

Test anchors

  • Test anchors protect fixes: accepted public tests become anchors first, then passing AI-generated tests are added and must continue to pass.When an AI-generated test fails, the flow assumes the code is wrong and requires the fix to preserve all acquired anchors.Source: §4 · p. 6

Methods that did not work

  • The authors report that additional tricks and methods described in Appendix C did not improve results.This statement concerns the tried additional methods, not the proposed flow as a whole.Source: §4 · p. 6

5. Results

On CodeContests, AlphaCodium improves on a direct prompt across models and sets, and the reported comparisons favor it over prior works. Comparisons to other systems have reproducibility constraints.

5.1. Direct prompt vs. AlphaCodium flow

  • Pass@k is the percentage of problems solved using k generated solutions per problem.The provided passage gives this metric definition while introducing Table 1.Source: §5.1 · p. 6
  • AlphaCodium flow consistently and significantly improves LLM performance on CodeContests across open-source DeepSeek and closed-source GPT models, on validation and test sets; GPT-4 validation pass@5 rises from 19% to 44%.Source: §5.1 · p. 7

5.2. Comparison to previous works

  • AlphaCodium’s top results are reported as better than AlphaCode’s, whose approach fine-tunes a model for code, generates and clusters many solutions, then submits K from the top clusters.For example, pass@10@100K means 100K solutions were generated and clustered, and 10 selected solutions were submitted.Source: §5.2 · p. 7
  • Neither AlphaCode nor CodeChain released reproducible end-to-end CodeContests generation and evaluation scripts.The paper notes evaluation subtleties when comparing results.Source: §5.2 · p. 7
  • For GPT-3.5 at pass@5, AlphaCodium consistently does better than CodeChain.Table 2 compares AlphaCodium results with other works from the literature.Source: Table 2 · p. 7, §5.2 · p. 7

6. Conclusions

AlphaCodium is a two-phase code-oriented flow that iteratively runs and fixes generated code against tests. On CodeContests, it improves model results and outperforms prior works while using a smaller computational budget.

6. Conclusions

  • AlphaCodium iteratively runs and fixes generated code against input-output tests, using natural-language preprocessing followed by iterations on public and AI-generated tests.The conclusion describes these as the flow’s two main phases.Source: §6 · p. 8
  • The flow uses YAML structured output, modular code, bullet-point semantic reasoning, double validation, exploration, and test anchors.These are presented as beneficial design concepts and practices for code generation.Source: §6 · p. 8
  • On CodeContests, AlphaCodium improves results for various open- and closed-source models and outperforms previous works with a smaller computational budget.The conclusion states this as the flow’s evaluation outcome.Source: §6 · p. 8

A. Typical HumanEval code problem

The example problem asks whether any two values in a vector are closer than a specified threshold.

A. Typical HumanEval code problem

  • The example checks whether any two numbers in a vector are closer than a given threshold.The displayed function takes a vector of floats and a float threshold.Source: A. Typical HumanEval code problem · p. 9

B. Why YAML output is better suited for code generation tasks than JSON output

The authors argue that YAML is better suited than JSON for generated code because it avoids JSON’s quoting and escaping burden and uses fewer tokens. The figure compares the same output in both formats.

B. Why YAML output is better suited for code generation tasks than JSON output

  • YAML block-scalar output needs to obey indentation, while JSON requires generated code and special characters to be placed inside quoted strings.The authors argue this makes YAML easier for code generation.Source: B. Why YAML output is better suited… · p. 9
  • YAML output uses fewer tokens than JSON, which the authors say reduces cost and inference time and can increase quality.They attribute the quality effect to the model paying attention to fewer nonessential tokens.Source: B. Why YAML output is better suited… · p. 9
  • Figure 4 compares the same output represented once in JSON and once in YAML.The comparison is from OpenAI Playground.Source: Figure 4 · p. 9

C. What didn’t work for us

The authors report no improvement from several additions to iterative fixing prompts or from optimizing single-stage and non-iterative prompt approaches.

C. What didn’t work for us

  • Adding the last 50 executed lines alongside an error message during iterative fixing did not improve results.The injected execution trace was limited to the last 50 lines.Source: C. What didn’t work for us · p. 10
  • Injecting the last K failed code solutions to steer iterative fixing in different directions produced no improvement.The passage does not specify a value for K.Source: C. What didn’t work for us · p. 10
  • Providing the last applied code patch diff during iterative fixing yielded no improvement.Source: C. What didn’t work for us · p. 10
  • Optimizing single-stage prompts or chains of non-iterative prompts produced no significant improvement; models still missed details in lengthy specifications and generated incorrect code.Source: C. What didn’t work for us · p. 10

D. Estimating distance between tests’ outputs

The authors estimate how far generated test outputs are from expected outputs, supplementing pass/fail checks with output-type-specific distance calculations.

D. Estimating distance between tests’ outputs

Loading 2401.08500v1…