PaperlayerView exact revision on arXiv

Source-linked AI summary

Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation

Yi Cui

arXiv:2505.09027v1cs.SEcs.AI

TL;DR

Many coding benchmarks emphasize well-represented tooling and algorithmic tasks, leaving application-layer test instructions a less explored basis for assessing models. WebApp1K evaluates TDD tasks, and the authors identify instruction following and in-context learning as key improvement areas, with long input contexts a bottleneck.

  • Problem

    Many coding benchmarks focus on well-represented tooling and algorithmic challenges, leaving application-layer tasks as a less explored way to assess whether models understand instructions in tests.

    Source: §1 · p. 2

  • Method

    WebApp1K provides 1,000 challenges in which LLMs implement features in mini web applications, with compact code generation as a design constraint.

    Source: §1 · p. 2

  • Results

    Instruction following and in-context learning are key improvement areas for TDD, while input context length is a main bottleneck across LLMs.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors argue that dedicated TDD benchmarks can evaluate and challenge models on business-logic instructions, which they identify as more critical to task success than pretraining knowledge.

    Source: §5.5 · p. 8

    The authors identify expanding application-scenario coverage and addressing vulnerability to long coded instructions as future directions.

    Source: §6 · p. 8

Abstract

from arXiv · show

We introduce WebApp1K, a novel benchmark for evaluating large language models (LLMs) in test-driven development (TDD) tasks, where test cases serve as both prompt and verification for code generation. Unlike traditional approaches relying on natural language prompts, our benchmark emphasizes the ability of LLMs to interpret and implement functionality directly from test cases, reflecting real-world software development practices. Comprising 1000 diverse challenges across 20 application domains, the benchmark evaluates LLMs on their ability to generate compact, functional code under the constraints of context length and multi-feature complexity. Our findings highlight instruction following and in-context learning as critical capabilities for TDD success, surpassing the importance of general coding proficiency or pretraining knowledge. Through comprehensive evaluation of 19 frontier models, we reveal performance bottlenecks, such as instruction loss in long prompts, and provide a detailed error analysis spanning multiple root causes. This work underscores the practical value of TDD-specific benchmarks and lays the foundation for advancing LLM capabilities in rigorous, application-driven coding scenarios.

1. Introduction

The introduction frames test-driven development (TDD) as coding where tests provide both instructions and verification, and presents WebApp1K as a benchmark for studying what drives success. It highlights instruction following and in-context learning as key abilities, with input context length a major bottleneck.

1. Introduction

  • In TDD, tests serve as both prompt and verification, unlike natural-language-prompted test-last development (TLD).The introduction presents tests as de facto system specifications for mature projects, where noncompliant code can cause serious losses.Source: §1 · p. 1
  • The benchmark targets nuanced application-layer instructions because tooling and algorithmic challenges are already well represented and may not test whether models understand instructions.It also aims to test implementation of many features with few tokens, given heterogeneous context-length constraints.Source: §1 · p. 2
  • WebApp1K introduces a TDD benchmark of 1,000 challenges spanning application scenarios, with tasks asking LLMs to implement one or more features.Human designers create scenarios and use GPT to turn them into tests.Source: §1 · p. 2
  • Instruction following and in-context learning distinguish TDD success, while input context length is identified as the main bottleneck across LLMs.The authors report that programming and algorithmic proficiency are neither critical nor sufficient, and that some low-TDD-success models perform well on sibling TLD tasks.Source: §1 · p. 2
  • Existing coding benchmarks use tests in different ways: algorithmic tasks follow TLD, while problem-solving tasks include tests alongside logs, source code, and issue descriptions.The introduction positions dedicated TDD evaluation as a distinct coding-task category.Source: §1 · p. 2

2. Benchmark

The benchmark uses React-based web applications to create compact, application-layer tasks. Test files provide the task prompts, and test expectations determine whether generated code succeeds.

2.1. Rationales on Technology Stack

  • WebApp1K uses web applications for their broad scenario coverage and ubiquity, and chooses frameworks to fit compact code generation and familiar tooling.Frameworks abstract boilerplate, while a single-page application lets multiple features fit in one code file and supports hooking model output to evaluation.Source: §2.1 · p. 3

2.2. Task Prompt

  • Tasks prompt the model with test files that simulate user actions and check expected outcomes, including successful and failed API responses.The benchmark contains 1,000 tasks across 20 application domains, each described by success and failure cases.Source: §2.2 · pp. 3–4

2.3. Task Verification

  • A task succeeds when generated code meets all expectations in Tables 4 and 5; with n = 10 evaluations maximum, results are measured using pass@1, pass@5, and pass@10.Source: §2.3 · p. 4

3. Evaluation Results

The evaluation compares frontier LLM performance, examines task and error patterns, and tests whether modifying violated tests changes pass@1.

3.1. LLM Performances

  • Top reasoning models lead other LLMs by an obvious pass@k gap, although their results were measured only at pass@1.Table 6 summarizes pass@k results for 19 frontier LLMs; the authors cite inference cost as the reason for measuring only pass@1 for top reasoning models.Source: §3.1 · p. 4, Table 6 · p. 4

3.2. Benchmark Difficulty

  • Most tasks have low failure rates, but a small cluster has very high failure rates, with some unsolved by every LLM.More failures indicate a more difficult task; reasoning models were excluded from this distribution because they were evaluated only once per task.Source: §3.2 · p. 4, Figure 3 · p. 5

3.3. Error Types

  • Error logs were grouped into seven types, and their verbatim forms were projected to narrower root-cause behaviors in this benchmark.The authors further conjecture connections between those root causes and model capabilities.Source: §3.3 · p. 5

3.4. Singular and Twin Errors

  • 93% of error logs contain either a singular error or twin errors.A twin error is two errors of the same type, often pointing to the same line; one bug can fail two unit tests.Source: §3.4 · p. 5, Figure 4 · p. 5

3.5. Error Distribution by Models

  • No model is immune to any of the seven error types, and the authors identify instruction adherence as the key differentiator between stronger and weaker models.The authors say models share the capabilities and vulnerabilities associated with these error types; reasoning models were excluded from the per-model distribution because they had one run per task rather than ten.Source: §3.5 · pp. 5–6, Figure 5 · p. 5

3.6. TLD Experiment

  • Modifying violated tests to accommodate verbatim code outputs raised pass@1 for every LLM, with edits limited to one statement and the test structure unchanged.The experiment follows the finding that syntax errors were rare and that some errors violated tests without making the code dysfunctional.Source: §3.6 · p. 6, Table 8 · p. 6

4. Duo-Feature Upgrade

The duo-feature upgrade lengthens prompts and adds test cases; performance falls, and a worked example links a missed requirement to instruction loss.

4. Duo-Feature Upgrade

  • The upgraded tasks combine two features in one webpage, using four tests—two successes and two failures—and prompts of around 1K tokens.The output remains a single webpage decorated with multiple UI elements.Source: §4 · pp. 6–7

4.1. LLM Performnaces

  • Longer context with more test cases significantly decreases pass@1 for all LLMs, while Claude 3.5 holds the state-of-the-art position.The upgraded tasks combine two features; Table 9 reports duo-feature pass@1 for selected LLMs.Source: §4.1 · p. 7, Table 9 · p. 7
  • The output code remains functional with occasional build failures, while the same errors occur more frequently on duo-feature tasks.The example model solved both component tasks separately in the single-feature benchmark.Source: §4.1 · p. 7, §4.2 · p. 7

4.2. Instruction Loss

  • In the example, o1-preview passes all but the final test, and the authors attribute the miss to instruction loss rather than weak coding ability.Its reasoning chain does not mention catching the internal server error; the authors say it is unknown whether the instruction was missed in the input or lost during early reasoning.Source: §4.2 · p. 7

5. Related Works

The related work contrasts TDD with natural-language-prompted coding and connects the benchmark’s findings to instruction following, reasoning models, and specialized coding LLMs.

5.1. Coding-Related Tasks and Benchmarks

  • Where conventional coding benchmarks pair natural-language prompts with test-suite verification, TDD benchmarks unify the task prompt and verification in tests.The paper argues that this avoids prompt-verification misalignment while introducing other challenges for LLMs.Source: §5.1 · p. 7

5.2. Instruction Following and In-Context Learning

  • The evaluation finds that LLMs can follow coded instructions, but are vulnerable when those instructions grow longer.The authors note that most existing work on instruction following focuses on natural-language instructions.Source: §5.2 · pp. 7–8

5.3. Reinforcement Learning and Reasoning

  • The authors describe reasoning and self-improvement techniques as relevant to benchmark-leading performance, while also noting instruction loss can hurt reasoning-model results.The related work includes reinforcement learning applied to coding, test generation, and error repair.Source: §5.3 · p. 8

5.4. Coding LLMs

  • The authors anticipate that specialized coding LLMs could be fine-tuned for TDD-aligned instructions, including long contexts with multiple code-heavy expectations.They express interest in evaluating such models on the benchmark.Source: §5.4 · p. 8

5.5. TDD in LLM Coding

  • Although prior work has introduced TDD into coding prompts, the authors identify this paper as the first focused on TDD benchmarking.They argue dedicated benchmarks are needed because instruction knowledge is critical to TDD task success, overshadowing pretraining knowledge.Source: §5.5 · p. 8

6. Conclusions

The paper presents a dedicated TDD benchmark and identifies instruction following and in-context learning as key areas for improvement. It points to broader application coverage and robustness to long coded instructions as future directions.

  • The authors evaluate 18 frontier LLMs with a dedicated TDD benchmark and identify instruction following and in-context learning as key improvement areas.Source: §6 · p. 8
  • One future direction is expanding WebApp1K to cover more application scenarios while cross-examining this paper’s findings.Source: §6 · p. 8
  • The authors also propose exploring practical hill-climbing ideas to address vulnerability to long coded instructions.Source: §6 · p. 8

A. Benchmark Construction

WebApp1K is constructed from human-proposed application domains and categories, with GPT-generated tests for scenarios. Its construction yields 1000 benchmark scenarios.

A. Benchmark Construction

B. Experiment Setup

The experiment setup standardizes API parameters and tunes temperature and top p on a task subset. Prompt experiments compare three variants with the original prompt and find no variant that helps all models.

B. Experiment Setup

C. Prompt Experiments

D.1. Single-Feature Task

The task analyses show how models can misread tests or module conventions, while the LOC analysis uses solution length as a proxy signal. In the duo-feature raw format, models often choose an incorrect split-module implementation.

D.1. Single-Feature Task

  • In the ticket-submission task, familiar frontend validation misleads non-reasoning models, although tests require an API request and backend validation.Source: §D.1 · pp. 15, 17, Figure 7 · p. 16
  • Reasoning models can also fall into the ticket-submission trap, while o1 models can reach the correct implementation by sending the API request unconditionally.Source: §D.1 · pp. 15–16

D.2. Duo-Feature Task

  • All models struggle with the raw duo-feature format, and o1 models fail every problem; both reasoning and non-reasoning models often split the implementation into separate modules.Source: §D.2 · pp. 17–18, Table 19 · p. 18
  • The raw format is semantically equivalent to the normalized format: both require one module implementing all expectations with a single default export.Source: §D.2 · pp. 17–18

E. Line-of-Code (LOC) Analysis

E.1. LOC Distribution by Models

Across models and applications, code-length distributions show patterns of variation, but median LOC is only weakly related to correctness. Success distributions are at least as complex as failure distributions, while application-level error patterns show no distinctive signatures.

E.1. LOC Distribution by Models

  • Median LOCs across models range from 35 to 46, with no strong correlation between median code length and pass@1 correctness.Mixtral-8x7b has the shortest median LOC but ranks low on pass@1; stronger models generate longer code.Source: §E.1 · p. 19, Table 22 · p. 19
  • High-performing models tend toward bimodal LOC distributions, while smaller models’ distributions are more tightly centered around their medians.The paper suggests the higher peak in bimodal distributions often corresponds to shorter code.Source: §E.1 · p. 20, Figure 8 · p. 20, Figure 9 · p. 21

E.2. Impact of Success/Failure

  • Successful outputs have more complex LOC distributions than failures; low-performing models’ failures are unimodal, unlike the multimodal success distributions of top models.The paper describes this as reflecting the complexity of writing correct code even when mean LOC is below 50.Source: §E.2 · p. 21

E.3. LOC Distribution by Applications

  • Across applications, median LOCs stay within a narrow range of 37 to 46, and both unimodal and multimodal distributions center on their medians.Fourteen applications are unimodal and six are multimodal.Source: §E.3 · p. 21

E.4. LOC Distribution by Applications: Success vs Failure

  • Across applications, success distributions are equally or more complex than failure distributions, although success and failure sets are about equal in size.Each application combines outputs from models spanning a range of performance levels.Source: §E.4 · p. 22

F. Per-Application Error Analysis

Application-level failures share a common pattern: most problems are easier, while a smaller set is especially challenging. Failure-rate skew varies by domain, but the paper finds no distinctive application-specific error patterns.

F. Per-Application Error Analysis

  • Across applications, failures concentrate among easier problems, with a few harder problems that models struggle with.Figure 17 breaks down the failure pattern by application.Source: F. Per-Application Error Analysis · p. 22
  • Failure rates rise sharply for a few problems in Fitness Tracking and Music Streaming, while Weather and Photo Gallery have fewer failures overall.Customer Support and Pet Care also show a sharper increase among their hardest problems.Source: F. Per-Application Error Analysis · p. 22
  • The paper finds no distinctive application-level error patterns: neither a special error type nor a special application stands out.Raw error counts are on the same scale across applications because each combines outputs from all models.Source: F. Per-Application Error Analysis · p. 23

G. Bias Analysis

A preliminary automated and manual review of WebApp1K found no instances of the searched-for biases. The authors caution that automated analysis may miss more nuanced bias and welcome further guidance to extend the fairness assessment.

G. Bias Analysis

  • The preliminary review searched code for gendered terms, stereotypes, and regional references, and checked APIs and user-facing messages for exclusionary patterns or implicit bias.The authors used an automated analysis script alongside examination of endpoints and messages.Source: G. Bias Analysis · p. 23
  • The investigation found no instances of the examined biases in WebApp1K.Source: G. Bias Analysis · p. 23
  • The authors note that automated analysis may miss nuanced biases and that further investigation may be needed for a comprehensive fairness evaluation.Source: G. Bias Analysis · p. 23
Loading 2505.09027v1…